Run Gemma2 on Raspberry Pi 5 using Ollama

DFRobot Dec 11 2024 38

In the realm of artificial intelligence and machine learning, the deployment of models on edge devices like the Raspberry Pi has gained significant traction. The Raspberry Pi, known for its compact size and versatility, has become a popular choice for AI enthusiasts and professionals alike. In this article, we will delve into the process of installing and running the SLM (small language model) Gemma2 on the Single Board Computer (SBC) Raspberry Pi 5 using the Ollama runtime framework.

Overview of Gemma2

Gemma 2 is Google's latest open large language model. Gemma is based on Google DeepMind's Gemini, has a context length of 8K tokens, and gemma-2-9b has a 9 billion base model.

Gemma2-9b-q4, a state-of-the-art AI model, boasts a model size of 1.6 GB. It is released under the Gemma license, an open-source license that fosters innovation and collaboration within the AI community.

Installing Ollama on Raspberry Pi 5

Ollama, a lightweight and efficient runtime framework, is essential for running Gemma2 on the Raspberry Pi 5. To install Ollama, follow these steps:

1. Update the System: Ensure that your Raspberry Pi's operating system is up-to-date by running the following command:

Plain Text
sudo apt-get update && sudo apt-get upgrade

2. Install Ollama: Navigate to the cloned directory and build Ollama by running:

Plain Text
curl -fsSL https://ollama.com/install.sh | sh

Running Gemma2-2b-q4 on Raspberry Pi 5

Once Ollama is successfully installed, you can proceed to run Gemma2 on your Raspberry Pi 5. Follow these steps:

Download Gemma2 Model: Obtain the Gemma2 model file from Ollama. In verbose mode, the program will provide more output information to help developers or users understand detailed information during program operation.

Python
ollama run gemma2:2b
/set verbose

Performance Considerations

When running Gemma2 on a Raspberry Pi 5, it is crucial to consider the device's hardware limitations. The Raspberry Pi 5, while powerful for its size, may not match the performance of dedicated server-grade hardware. Therefore, it is advisable to manage expectations regarding the token speed and overall performance of Gemma2 on this platform.

token speed of gemma2-2b-q4 running on Raspberry Pi 5: 4.48 tokens/s

Conclusion

Running Gemma2 on a Raspberry Pi 5 using the Ollama runtime framework opens up a world of possibilities for AI applications on edge devices. While the process involves several steps, including installing Ollama and obtaining the Gemma2 model, the result is a robust AI setup that can be utilized in various scenarios. As AI continues to advance, the synergy between powerful models like Gemma2 and versatile devices like the Raspberry Pi will only grow stronger, paving the way for innovative applications in the field of edge AI.

Learn More About SLMs on Raspberry Pi 5:

Run Small Language Models (Mathstral, phi 3, Llama 3.1, Gemma2 2b, Qwen) on Raspberry Pi 5

Run phi3.5 on Raspberry Pi 5 using Ollama

Raspberry Pi Products Raspberry Pi Projects