Abstract
Recently, the advancement in natural language processing has been driven by the concept of Large Language Models (LLMs). Nevertheless, the deployment of these models in distributed systems is associated with an essential trade-off in terms of privacy and model performance. This systematic review examines the various methods of federated learning in fine-tuning LLMs that were developed between 2023 and 2025, considering the trade-offs in terms of privacy and performance. The results indicate that the methods of Parameter-Efficient Fine-Tuning (PEFT) have been successful in reducing the bandwidth requirements, while system heterogeneity is the key challenge. In the near future, the research in the area of fine-tuning LLMs needs to be focused on architectural flexibility to suit the constraints of edge devices.