feat: add Ollama provider support and dashboard integration

This commit is contained in:
2026-02-26 13:31:09 -05:00
parent 87a3ecba07
commit 371e1f2e77
8 changed files with 375 additions and 1 deletions

View File

@@ -10,6 +10,7 @@ A unified, high-performance LLM proxy gateway built in Rust. It provides a singl
* **Google Gemini:** Support for the latest Gemini 2.0 models.
* **DeepSeek:** High-performance, low-cost integration.
* **xAI Grok:** Integration for Grok-series models.
* **Ollama:** Support for local LLMs running on your machine or another host.
- **Observability & Tracking:**
* **Real-time Costing:** Fetches live pricing and context specs from `models.dev` on startup.
* **Token Counting:** Precise estimation using `tiktoken-rs`.
@@ -52,6 +53,14 @@ A unified, high-performance LLM proxy gateway built in Rust. It provides a singl
3. Configure providers and server:
Edit `config.toml` to customize models, pricing fallbacks, and port settings.
**Ollama Example (config.toml):**
```toml
[providers.ollama]
enabled = true
base_url = "http://192.168.1.50:11434/v1"
models = ["llama3", "mistral"]
```
4. Run the proxy:
```bash
cargo run --release