SARA: Stochastic Adaption of Language Models to In-Domain RAG

In the current landscape of pre-trained large language model (LLM) applications, various methods such as data augmentation, Residual Learning (RL), Curriculum Learning (CL), Low-Rank Adaptation (LoRA) and Retrieval-Augmented Generation (RAG), are commonly employed to integrate more targeted information into pre-trained models. However, these methods often fall short in terms of training costs or practical effectiveness. Therefore, finding a more effective approach to enhance the capabilities of large language models in downstream tasks is imperative. In this paper, we introduce Stochastic Adaption of Retrieval Augmentation (SARA), an integrated fine-tuning strategy that introduces chained adaption stages while incorporating features from Retrieval-Augmented Fine-Tuning (RAFT), in order to enhance the LLM's ability of effectively responding to domain-specific queries. By organizing the various adaptation stages in a stochastic manner, we achieve improved performance, robustness and training efficiency compared to existing fine-tuning methods.