OpenELM, open-source efficient language models from Apple

OpenELM represents a suite of open-source language models developed by Apple, designed to work on-device. It is compatible with Apple’s MLX framework, ensuring seamless integration with Apple devices, such as iPhones and iPads.

The model employs layer-wise scaling, a unique resource allocation method that enhances accuracy while minimizing the need for computational resources.

Although OpenELM is optimized for Apple hardware, the models are open source and could potentially be adapted for use on other platforms.

You can access the source code, pre-trained model weights, and training recipes in the CoreNet repository. The models are available on Hugging Face.

Unlike previous models that might only release model weights and inference code, OpenELM provides the entire framework, including data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained checkpoints and training logs. The team released additional code to convert their models into the MLX library format for use on Apple devices.

OpenELM architecture

OpenELM is built on a decoder-only transformer framework, incorporating significant enhancements. Keeping pace with recent advancements in LLMs, OpenELM adopts grouped query attention (GQA) as a substitute for traditional multi-head attention (MHA) and replaces the standard feed forward network (FFN) with the SwiGLU FFN variant, which is a more advanced version for processing information within the network (see the picture below).

Further refinements to improve the model’s architecture:

1. eliminate learnable bias parameters in any linear layers to simplify the model and potentially reduce the training time
2. apply pre-normalization using RMSNorm and rotary positional embedding (ROPE) to accurately encode the position of words in a sentence
3. use flash attention to speed up the model’s computation of the attention scores

Layer-wise scaling strategy

The transformer-based LLMs typically maintain the same number of heads and the same feed forward networks’ dimension. While this uniformity simplifies the model’s architecture, it is not the most efficient way to distribute the parameters throughout the model.

OpenELM uses a layer-wise scaling strategy to optimize the parameters’ allocation within each layer of the transformer. It starts with smaller dimensions in the attention and feed-forward modules of the transformer layers that are closer to the input, and gradually increases the size of these layers that are closer to the output (see the picture below).

Training

The OpenELM models were trained for 350k iterations (training steps) using CoreNet. These models used the same tokenizer as the LLama model, and they filtered and tokenized text data in real-time, rather than using pre-tokenized data.

The pretraining process leveraged publicly available datasets, totaling around 1.8T tokens:

Evaluation

OpenELM outperforms comparable-sized existing LLMs pretrained on publicly available datasets. The average accuracy was calculated across multiple tasks, including reasoning, knowledge understanding, and misinformation & bias.

Model	Public dataset	Code (open-source)	Weights (open-source)	Model size	Pretraining tokens	Average acc. (in %)
OPT	✗	✓	✓	1.3B	0.2T	41.49
PyThia	✓	✓	✓	1.4B	0.3T	41.83
MobiLlama	✓	✓	✓	1.3B	1.3T	43.55
OLMo	✓	✓	✓	1.2B	3.0T	43.57
OpenELM	✓	✓	✓	1.1B	1.5T	45.93

Notably, OpenELM achieves better performance than the existing open-source LLMs trained on public datasets. For instance, OpenELM, with its 1.1B parameters, outperforms OLMo, which has 1.2B parameters, by 2.36% while requiring half as many pretraining tokens.

OpenELM was compared with other widely-used LLMs using different evaluation benchmarks. The results can be seen in the next tables where the highest accuracy scores achieved by each model are marked in bold, and those trained with a smaller dataset are shown in gray.

Conclusion

OpenELM sets a new standard in AI research by offering a comprehensive and fully open framework.

Since the models are trained on publicly available datasets, they lack built-in safety measures, potentially leading to harmful, inaccurate, or biased outputs. Therefore, it’s important for users and developers to conduct extensive safety tests and create appropriate filtering mechanisms for their applications.

Read more:

• Apple press release
• Paper on arXiv: “OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework”
• GitHub repository
• Hugging Face model release