Clawd BotandClaude Opus 4.6
ca9b510922
- Reset master to upstream/main (16,697 commits) - Overlay 2,271 local-only files (skills, tools, workspace, configs, apps) - Restore IDENTITY.md and USER.md templates - Build verified, gateway running, Discord working Co-Authored-By: Claude Opus 4.6 <[email protected]>
214 lines
4.7 KiB
Markdown
214 lines
4.7 KiB
Markdown
# llm-checker - Smart Model Selection for Jarvis
|
|
|
|
## Why We Needed This
|
|
|
|
**Problem:** Vi valgte models manuelt baseret på gætterier om hardware compatibility.
|
|
|
|
**Solution:** `llm-checker` giver data-drevet model selection baseret på faktisk hardware.
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
sudo npm install -g llm-checker
|
|
```
|
|
|
|
## Key Commands
|
|
|
|
### 1. Hardware Detection
|
|
|
|
```bash
|
|
llm-checker hw-detect
|
|
```
|
|
|
|
**Output for vores system:**
|
|
|
|
- CPU: Intel i5-12500H (16 cores, AVX2)
|
|
- RAM: 15GB
|
|
- GPU: None (CPU-only)
|
|
- Tier: MEDIUM LOW
|
|
- Max model size: 9GB
|
|
|
|
### 2. Model Recommendations
|
|
|
|
```bash
|
|
# Full analysis (slow - scrapes all Ollama models)
|
|
llm-checker recommend
|
|
|
|
# Quick mode with cache
|
|
llm-checker recommend --use-cache
|
|
|
|
# Smart recommendations with scoring
|
|
llm-checker smart-recommend
|
|
```
|
|
|
|
### 3. Installed Models Ranking
|
|
|
|
```bash
|
|
llm-checker installed
|
|
```
|
|
|
|
Ranks your current Ollama models by:
|
|
|
|
- Hardware compatibility
|
|
- Performance (tokens/sec estimate)
|
|
- Memory requirements
|
|
- Use-case fit
|
|
|
|
### 4. Search Models
|
|
|
|
```bash
|
|
llm-checker search <query>
|
|
|
|
# Examples
|
|
llm-checker search "code"
|
|
llm-checker search "fast"
|
|
llm-checker search "small"
|
|
```
|
|
|
|
### 5. AI-Powered Selection
|
|
|
|
```bash
|
|
llm-checker ai-check
|
|
llm-checker ai-run
|
|
```
|
|
|
|
Uses meta-analysis to pick the best model for your task.
|
|
|
|
## Integration with Jarvis
|
|
|
|
### Current State
|
|
|
|
**Ollama models installed:**
|
|
|
|
- qwen2.5:0.5b (397 MB) - Very small, fast, good for simple tasks
|
|
|
|
**Cloud models (primary):**
|
|
|
|
- gemini-3-flash-preview (Google)
|
|
- gemini-3-pro-preview (Google)
|
|
- qwen3-235b (NVIDIA)
|
|
- deepseek-v3.2 (NVIDIA)
|
|
- claude-sonnet-4-5 (Anthropic)
|
|
- gpt-5-mini (GitHub Copilot)
|
|
|
|
### Recommended Strategy
|
|
|
|
**Use llm-checker to:**
|
|
|
|
1. **Find optimal local models** for CPU-only hardware
|
|
- Target: 0.5B - 3B parameter models
|
|
- Max size: ~9GB
|
|
- Focus: Fast inference on AVX2
|
|
|
|
2. **Benchmark installed models**
|
|
|
|
```bash
|
|
llm-checker installed --benchmark
|
|
```
|
|
|
|
3. **Cost optimization**
|
|
- Simple queries → Local (qwen2.5:0.5b)
|
|
- Complex queries → Cloud (gemini/nvidia/copilot)
|
|
- Code generation → Local (codellama:7b-code)
|
|
|
|
4. **Monthly check**
|
|
- New models released constantly
|
|
- Run `llm-checker smart-recommend` monthly
|
|
- Update Ollama models based on recommendations
|
|
|
|
### Jarvis Automation
|
|
|
|
Added to tooling:
|
|
|
|
- `/home/alex/clawd/tools/jarvis-model-optimizer.sh`
|
|
- Runs hardware detection
|
|
- Logs installed models
|
|
- Suggests optimization opportunities
|
|
|
|
**Future enhancement:**
|
|
|
|
- Weekly cron job to check for better models
|
|
- Auto-install high-scoring models under 3GB
|
|
- Benchmark new models before adding to delegation
|
|
|
|
## Model Selection Matrix
|
|
|
|
Based on `llm-checker` tier: **MEDIUM LOW**
|
|
|
|
| Use Case | Recommended Model | Size | Speed |
|
|
| ----------------- | ---------------------- | ----- | ----------- |
|
|
| Simple chat | qwen2.5:0.5b | 397MB | ⚡⚡⚡ Fast |
|
|
| Code completion | deepseek-coder:1.3b | ~1GB | ⚡⚡ Fast |
|
|
| Summarization | phi3:3.8b | ~3GB | ⚡ Medium |
|
|
| Complex reasoning | Use cloud (Gemini Pro) | - | ☁️ Cloud |
|
|
| Heavy code | Use cloud (Qwen3 235B) | - | ☁️ Cloud |
|
|
|
|
## Common Issues
|
|
|
|
### Ollama crashes/hangs
|
|
|
|
- **Cause:** Model too large for RAM
|
|
- **Fix:** Use llm-checker to find smaller models
|
|
- **Setting:** `OLLAMA_NUM_PARALLEL=1 OLLAMA_MAX_QUEUE=1`
|
|
|
|
### Slow inference
|
|
|
|
- **Check:** CPU-only mode (no GPU)
|
|
- **Fix:** Use quantized models (Q4_0, Q4_K_M)
|
|
- **Verify:** `llm-checker installed --benchmark`
|
|
|
|
### Out of memory
|
|
|
|
- **Cause:** Model size > available RAM
|
|
- **Fix:** Stick to < 3B param models
|
|
- **Tool:** `llm-checker hw-detect` shows max size
|
|
|
|
## Next Steps
|
|
|
|
1. **Run full analysis:**
|
|
|
|
```bash
|
|
llm-checker recommend --use-cache > /tmp/llm-recommendations.txt
|
|
```
|
|
|
|
2. **Install top 3 recommended models:**
|
|
|
|
```bash
|
|
# Based on recommendations
|
|
ollama pull <model-name>
|
|
```
|
|
|
|
3. **Benchmark them:**
|
|
|
|
```bash
|
|
llm-checker installed --benchmark
|
|
```
|
|
|
|
4. **Update delegation rules:**
|
|
- Add best local model to `jarvis-delegate.sh`
|
|
- Route simple tasks to local
|
|
- Keep complex tasks on cloud
|
|
|
|
5. **Monitor costs:**
|
|
- Track API spend vs local usage
|
|
- Optimize delegation based on perf/cost
|
|
|
|
## Why We Didn't Use It Before
|
|
|
|
1. **Not proactive** - Didn't know it existed
|
|
2. **Cloud-first mindset** - Focused on API providers
|
|
3. **Ollama instability** - Avoided local inference
|
|
4. **No automation** - Manual model selection
|
|
|
|
**Jarvis fix:** Proactively suggest tools like this based on infrastructure needs.
|
|
|
|
## Resources
|
|
|
|
- GitHub: https://github.com/Pavelevich/llm-checker
|
|
- npm: https://www.npmjs.com/package/llm-checker
|
|
- MCP support: Built-in Model Context Protocol integration
|
|
|
|
---
|
|
|
|
**"Now we have data-driven model selection, sir." 🎯**
|