Alpaca is a strong open-source instruction-following language model released by a research team from the Stanford Center for Research on Foundation Models.
The model was released solely for academic research and cannot be utilized for commercial purposes.
The model
Alpaca is based on Meta’s LLaMA 7B (7 billion parameters) model. It has been specifically fine-tuned using over 52,000 instruction-following demonstrations and utilizes OpenAI’s text-davinci-003 (ChatGPT), as shown in the picture below.
Language models like LLaMA are small but high performing and are shared for research proposes. The main drawback of these models is their lack of instruction-tuning.
Instruction-tuning, also known as fine-tuning, is a process to improve the performance of a language model by modifying its pre-trained parameters. It’s an expensive process, requiring a big amount of data and computing resources.
The Stanford team succeeded to fine-tune the LLaMA 7B model with fewer resources and to use it for creating Alpaca model, as shown in picture above. The team used eight 80GB A100s graphics processing units (GPUs) for training, which cost less than $100 on cloud computing platforms.
Performance
Alpaca exhibits numerous characteristics akin to OpenAI’s text-davinci-003, yet it is remarkable for being compact and affordable to duplicate. The team also released the training recipe and data, and intends to release the model weights in the future.
Limitations, future improvements
In addition to its similarities to OpenAI’s text-davinci-003, Alpaca displays various typical shortcomings of language models, such as generating false information, expressing toxic language, and promoting stereotypes.
Potential future directions for research on Alpaca may involve conducting more rigorous evaluations of the model’s performance, exploring the potential risks associated with its use, and investigating alternative approaches to using self-supervision with text-davinci-003.
Conclusion
Alpaca is an open-source model to encourage other researchers to carry out controlled studies on instruction-following language models.
By enabling more scientific research on this topic, it is hoped that new techniques will be developed to address the current limitations of these models.
Learn more:
- Story source: “Alpaca: A Strong, Replicable Instruction-Following Model” (on Stanford University)
- Source code
