Files

302 lines
9.6 KiB
Plaintext

---
title: "Local LLM on Android"
description: "Run local LLM inference via node-llama-cpp and Ollama on your Android device."
---
## Overview
OpenClaw supports **local LLM inference** via [`node-llama-cpp`](https://github.com/withcatai/node-llama-cpp) and **Ollama integration**. The prebuilt native binary (`@node-llama-cpp/linux-arm64`) is included with the installation and loads successfully under the glibc environment — **local LLM is technically functional on the phone**.
However, there are **practical constraints** to consider before running local models.
<Info>
**☁️ Cloud Models Available**: Ollama now supports cloud-hosted models! Use `ollama launch openclaw --model kimi-k2.5:cloud` for superior performance without local resource usage. See [Cloud Models](#ollama-cloud-models) section below.
</Info>
---
## ⚠️ Practical Constraints
| Constraint | Details |
|------------|---------|
| **RAM** | GGUF models need at least **2-4GB of free memory** (7B model, Q4 quantization). Phone RAM is shared with Android and other apps |
| **Storage** | Model files range from **4GB to 70GB+**. Phone storage fills up fast |
| **Speed** | **CPU-only inference** on ARM is very slow. Android does not support GPU offloading for llama.cpp |
| **Use Case** | OpenClaw primarily routes to **cloud LLM APIs** (OpenAI, Gemini, etc.) which respond at the same speed as on a PC. Local inference is a **supplementary feature** |
<Info>
For **experimentation**, small models like **TinyLlama 1.1B (Q4, ~670MB)** can run on the phone. For **production use**, cloud LLM providers are recommended.
</Info>
---
## ☁️ Ollama Cloud Models
**Best of both worlds**: Run models in the cloud with Ollama's cloud integration — no local RAM/storage constraints!
### Quick Start
```bash
# Pull and launch with cloud model
ollama pull kimi-k2.5:cloud
ollama launch openclaw --model kimi-k2.5:cloud
```
### Recommended Cloud Models
| Model | Use Case | Context |
|-------|----------|---------|
| **`kimi-k2.5:cloud`** | Multimodal reasoning with subagents | 64k+ tokens |
| **`minimax-m2.5:cloud`** | Fast, efficient coding | 64k+ tokens |
| **`glm-5:cloud`** | Reasoning and code generation | 64k+ tokens |
| **`gpt-oss:120b-cloud`** | High-performance tasks | 128k tokens |
| **`gpt-oss:20b`** | Balanced performance | 64k tokens |
### Commands
| Command | Description |
|---------|-------------|
| `ollama launch openclaw` | Launch with model selector |
| `ollama launch openclaw --model <model>` | Launch with specific cloud model |
| `ollama launch openclaw --config` | Configure without launching |
| `ollama pull <model>:cloud` | Pull cloud model to local registry |
### Why Cloud Models?
| Advantage | Details |
|-----------|---------|
| **No Local Resources** | Zero RAM/storage usage on phone |
| **Superior Performance** | Full GPU acceleration on cloud servers |
| **Large Context** | 64k-128k token windows available |
| **Always Updated** | Latest model versions automatically |
| **Privacy Option** | Local models still available for sensitive data |
> 💡 **Recommendation**: Use **cloud models for production** workloads, **local models for testing/experimentation**.
---
## 🚀 Quick Start
### Option 1: node-llama-cpp (Recommended for Android)
**Why `--ignore-scripts`?** The installer uses `npm install -g openclaw@latest --ignore-scripts` because `node-llama-cpp`'s postinstall script attempts to compile `llama.cpp` from source via cmake — a process that takes **30+ minutes on a phone** and fails due to toolchain incompatibilities. The **prebuilt binaries work** without this compilation step, so the postinstall is safely skipped.
**Install**:
```bash
npm install -g node-llama-cpp --ignore-scripts
```
**Download a model** (TinyLlama 1.1B Q4 - good for testing):
```bash
mkdir -p ~/models
cd ~/models
curl -L -o tinyllama-1.1b-q4.gguf "https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
```
**Run inference**:
```bash
node -e "
const { LlamaChatSession } = require('node-llama-cpp');
const session = new LlamaChatSession({
modelPath: '/data/data/com.termux/files/home/models/tinyllama-1.1b-q4.gguf'
});
session.prompt('Hello, how are you?');
"
```
---
### Option 2: Ollama (Full Server)
[Ollama](https://ollama.com) provides a complete local LLM server with model management.
**Install Ollama**:
```bash
curl -fsSL https://ollama.com/install.sh | sh
```
**Start the server**:
```bash
ollama serve &
```
**Pull a model**:
```bash
# Small model for testing
ollama pull tinyllama
# Or larger models if you have RAM
ollama pull llama3.2:1b
ollama pull phi3:mini
```
**Chat with a model**:
```bash
ollama run tinyllama "Hello, how are you?"
```
**API Endpoint**:
```bash
curl http://localhost:11434/api/generate -d '{
"model": "tinyllama",
"prompt": "Hello, how are you?"
}'
```
<Warning>
Ollama needs more RAM and storage than node-llama-cpp. Recommended only for devices with **6GB+ RAM** and **32GB+ free storage**.
</Warning>
---
## 🔗 Official Ollama OpenClaw Integration
OpenClaw officially integrates with Ollama to provide a seamless local AI assistant experience.
### Why it's powerful
1. **Native API Integration**: OpenClaw connects directly to Ollama's native `/api/chat` endpoint. This ensures full support for streaming and **tool calling**.
> ⚠️ **Important**: Do not use the `/v1` OpenAI-compatible URL with OpenClaw. It breaks tool calling and causes models to output raw JSON!
2. **Automatic Model Discovery**: OpenClaw queries `/api/tags` and `/api/show` to automatically find your downloaded Ollama models, detect if they support tool calling, and configure their context windows appropriately.
### Setup Methods
**Method A: Ollama Launcher (Recommended)**
The easiest way to connect OpenClaw to Ollama is using the official launcher command:
```bash
ollama launch openclaw
```
This setups the security profile, configures the provider, and sets your primary model. To launch a specific model directly:
```bash
# Example with cloud model
ollama launch openclaw --model kimi-k2.5:cloud
```
**Method B: OpenClaw Onboarding**
Run the onboarding wizard and select "Ollama" when asked for a provider:
```bash
openclaw onboard
```
It will ask for your Ollama base URL (default is `http://127.0.0.1:11434`).
**Method C: Explicit Configuration**
You can force OpenClaw to use Ollama by exporting the API key environment variable before starting the gateway:
```bash
export OLLAMA_API_KEY="ollama-local"
openclaw gateway
```
---
## 📊 Model Recommendations
| Model | Size (Q4) | RAM Needed | Speed | Use Case |
|-------|-----------|------------|-------|----------|
| **TinyLlama 1.1B** | ~670MB | 2GB | Fast | Testing, experimentation |
| **Phi-3 Mini (3.8B)** | ~2.3GB | 4GB | Medium | Light tasks |
| **Llama 3.2 1B** | ~670MB | 2GB | Fast | Mobile-friendly |
| **Llama 3.2 3B** | ~2GB | 4GB | Medium | Balanced |
| **Mistral 7B** | ~4.1GB | 8GB | Slow | Advanced users only |
| **Llama 3 8B** | ~4.7GB | 8GB+ | Very Slow | Not recommended |
---
## 🔧 Configuration
### node-llama-cpp Context Length
Reduce context length to save RAM:
```javascript
const session = new LlamaChatSession({
modelPath: 'path/to/model.gguf',
contextSize: 2048 // Default is 4096
});
```
### Ollama Configuration
Set environment variables before starting:
```bash
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1
ollama serve
```
---
## 🌐 Cloud vs Local Comparison
| Feature | Local LLM | Cloud LLM (OpenClaw) | Ollama Cloud Models |
|---------|-----------|---------------------|---------------------|
| **Speed** | Slow (CPU-only) | Fast (GPU-accelerated) | ⚡ Fastest (cloud GPU) |
| **Privacy** | ✅ Full privacy | Depends on provider | Depends on provider |
| **Cost** | Free (after hardware) | Pay-per-token | Free via Ollama |
| **Model Size** | Limited by RAM (2-8GB) | Unlimited | Unlimited |
| **Context Window** | 2k-8k tokens | 64k-200k tokens | 64k-128k tokens |
| **Setup** | Manual download | One command | `ollama pull` |
| **Internet** | Not needed | Required | Required |
| **RAM Usage** | 2-8GB | None | None |
| **Storage** | 4-70GB | None | Minimal |
| **Best For** | Testing, offline | Production | Production + testing |
---
## 🛠️ Troubleshooting
### "Cannot find module 'node-llama-cpp'"
Make sure you installed with `--ignore-scripts`:
```bash
npm install -g node-llama-cpp --ignore-scripts
```
### "Out of memory" error
Close other apps and reduce context size:
```bash
export NODE_OPTIONS="--max-old-space-size=1024"
```
### Ollama killed by Android
Disable Phantom Process Killer:
```bash
adb shell settings put global development_settings_enabled 1
adb shell settings put global max_phantom_processes 64
```
### Model download fails
Use a different mirror or download on PC and transfer:
```bash
# On PC
curl -L -o model.gguf "URL"
# Transfer via USB or scp
scp model.gguf phone:~/models/
```
---
## 📚 Resources
- [node-llama-cpp Docs](https://github.com/withcatai/node-llama-cpp)
- [Ollama Docs](https://docs.ollama.com)
- [GGUF Models on HuggingFace](https://huggingface.co/models?library=gguf)
- [TheBloke's Quantized Models](https://huggingface.co/TheBloke)
---
## 💡 Best Practices
1. **Start small**: Begin with TinyLlama 1.1B to test your device
2. **Monitor RAM**: Use `htop` or Termux's `top` to watch memory usage
3. **Use tmux**: Run long inference sessions in tmux to prevent disconnection
4. **Cool your phone**: CPU inference generates heat; consider active cooling
5. **Cloud for production**: Use local LLM for testing, cloud for real work
<Info>
**Pro Tip**: Use OCA's hybrid mode — route simple queries to local LLM, complex tasks to cloud APIs. Best of both worlds!
</Info>