feat: add Ollama provider support and dashboard integration
This commit is contained in:
@@ -10,6 +10,7 @@ A unified, high-performance LLM proxy gateway built in Rust. It provides a singl
|
||||
* **Google Gemini:** Support for the latest Gemini 2.0 models.
|
||||
* **DeepSeek:** High-performance, low-cost integration.
|
||||
* **xAI Grok:** Integration for Grok-series models.
|
||||
* **Ollama:** Support for local LLMs running on your machine or another host.
|
||||
- **Observability & Tracking:**
|
||||
* **Real-time Costing:** Fetches live pricing and context specs from `models.dev` on startup.
|
||||
* **Token Counting:** Precise estimation using `tiktoken-rs`.
|
||||
@@ -52,6 +53,14 @@ A unified, high-performance LLM proxy gateway built in Rust. It provides a singl
|
||||
3. Configure providers and server:
|
||||
Edit `config.toml` to customize models, pricing fallbacks, and port settings.
|
||||
|
||||
**Ollama Example (config.toml):**
|
||||
```toml
|
||||
[providers.ollama]
|
||||
enabled = true
|
||||
base_url = "http://192.168.1.50:11434/v1"
|
||||
models = ["llama3", "mistral"]
|
||||
```
|
||||
|
||||
4. Run the proxy:
|
||||
```bash
|
||||
cargo run --release
|
||||
|
||||
Reference in New Issue
Block a user