Exploring Federated Learning Methods for Conventional and Foundation Models

Abstract

Federated Learning (FL) has recently achieved significant progress, enabling collaborative model training on distributed data across edge devices. This dissertation aims to investigate methods for improving the efficiency and effectiveness of FL in various scenarios by proposing (i) a communication-efficient federated learning (FL) paradigm, (ii) a parameter-efficient fine-tuning (PEFT) framework for large language models (LLMs), and (iii) a rank-adaptive adapter method for foundation models. Together, these studies aim to reduce bandwidth and compute costs while preserving or improving utility and stability across diverse clients.

The first study focuses on the data and device heterogeneity of FL. In the standard FL paradigm, iterative gradient or model exchanges between devices and the centralized server suffer from severe efficiency bottlenecks on the server. While enabling collaborative training without a central server, existing decentralized FL approaches either focus on the synchronous mechanism that deteriorates FL convergence, or ignore device staleness with an asynchronous mechanism, resulting in inferior FL accuracy. This study proposes an asynchronous, efficient decentralized framework for heterogeneous settings, AEDFL, that introduces an efficiency-aware aggregation rule to accelerate convergence, uses dynamic, staleness-aware updates to improve accuracy, and employs adaptive sparse training to reduce communication and computation costs.

The second study proposes a Parameter-efficient Prompt Tuning approach with Adaptive Optimization, FedPepTAO, to enable efficient and effective FL of large language models (LLMs). FL provides a collaborative framework for training models across decentralized data sources, but the extensive parameter updates required for LLMs present significant challenges in real-world applications, particularly in terms of communication costs and computational efficiency. In response to these limitations, this study incorporate an efficient partial prompt tuning strategy that reduces the number of trainable parameters, achieving a balance between performance and training efficiency. Furthermore, an adaptive optimization mechanism that dynamically adjusts updates on both the device and server sides is introduced. This dual optimization approach ensures improved performance, scalability, and practical applicability of FL for LLMs.

The third study explores Low-Rank Adaptation (LoRA), a leading PEFT approach for fine-tuning foundation models (FMs). Despite being effective, directly applying LoRA to FL introduces two issues: aggregation noise and rank drift. We propose RAFFT, a Riemannian LoRA algorithm with adaptive rank for federated fine-tuning (FFT-FM) that addresses both while cutting the computation cost. First, using Riemannian Procrustes alignment, we perform parameter matching on the manifold to suppress aggregation noise and reduce SVD cost by operating only on low-dimensional r x r matrices. We prove RAFFT’s rank-adaptive updates are theoretically equivalent (with bounded approximation error) to standard FFT-FM on full parameter matrices under FedAvg. Second, we design a Riemannian gradient descent (RGD) procedure that enforces rank invariance during local updates—clients optimize low-rank factors while the server sets the round’s rank—thereby preventing rank drift and accelerating convergence. RAFFT enables communication- and computation-efficient FL with adaptive ranks.