Run phi3.5 on Raspberry Pi 5 using Ollama

The Raspberry Pi, a pint-sized and budget-friendly Single Board Computers (SBC), has emerged as a favorite for diverse applications, ranging from educational endeavors to home entertainment systems. The launch of the Raspberry Pi 5 has markedly elevated the capabilities of this diminutive computing powerhouse, rendering it a perfect choice for handling intricate software, including SLMs(small language models) such as phi3.5 3.8b. In this write-up, we delve into the process of installing and executing phi3.5-3.8b on a Raspberry Pi 5, leveraging the Ollama runtime framework.
 

Overview of phi3.5

• Parameter size: Phi-3.5-mini-instruct has about 3.82 billion parameters.
• Design purpose: The model is designed for following instructions and supports fast reasoning tasks
• Context support: Supports context length of 128k tokens, suitable for processing long text data.
• Applicable scenarios: Suitable for environments with limited memory or computing resources, and can perform tasks such as code generation, mathematical problem solving, and logic-based reasoning.
• Performance: It performs well in multi-language and multi-turn dialogue tasks, and in the RepoQA benchmark, the performance of measuring "long context code understanding" exceeds other models of similar size, such as Lama-3.1-8B-instruct and Mistral-7B-instruct.

phi3.5-3.8b-q4 has a size of 2.2GB. It is released under the MIT open-source license, which allows users to use, modify, and distribute the model freely.
 

Installing Ollama on Raspberry Pi 5

Ollama is a lightweight and efficient runtime framework for running LLMs on devices with limited resources, such as the Raspberry Pi. To install Ollama on a Raspberry Pi 5, follow these steps:
Install the Ollama runtime framework:

Running phi3.5-3.8b-q4 on Raspberry Pi 5

Once Ollama is installed on the Raspberry Pi 5, you can run phi3.5-3.8b-q4 by executing the following command:
In verbose mode, the program will provide more output information to help developers or users understand detailed information during program operation.

This command will start the phi3.5-3.8b model and wait for input. You can then interact with the model by typing text and pressing Enter. The model will generate a response based on the input text, and the output will be displayed on the screen.
 

token speed of phi3.5-3.8b-q4 running on Raspberry Pi 5：3.42 tokens/s

Conclusion

Running phi3.5-3.8b on a Raspberry Pi 5 using the Ollama runtime framework is a straightforward process that can be completed in a few steps. This combination of hardware and software opens up a wide range of possibilities for natural language processing applications on a compact and affordable device. Whether you are a developer, researcher, or simply an enthusiast, the Raspberry Pi 5 and phi3.5-3.8b are powerful tools that can help you explore the world of natural language processing.

Learn More About SLMs on Raspberry Pi 5:

Run Small Language Models (Mathstral, phi 3, Llama 3.1, Gemma2 2b, Qwen) on Raspberry Pi 5