diff --git a/README.md b/README.md index ac31cad1..24081c22 100644 --- a/README.md +++ b/README.md @@ -5,19 +5,24 @@ A unified, high-performance LLM proxy gateway built in Go. It provides OpenAI-co ## Features - **Unified API:** OpenAI-compatible `/v1/chat/completions`, `/v1/images/generations`, `/v1/responses`, and `/v1/models` endpoints. - - The `/v1/responses` endpoint (OpenAI Responses API) is currently supported for OpenAI models only. Non-OpenAI providers (Gemini, DeepSeek, Moonshot, Grok, Ollama) return a "not supported" response. + - The `/v1/responses` endpoint (OpenAI Responses API) is currently supported for OpenAI models only. Non-OpenAI providers (Gemini, DeepSeek, Moonshot, Grok, Ollama, Xiaomi) return a "not supported" response. - **Multi-Provider Support:** - **OpenAI:** GPT-4o, GPT-4o Mini, GPT-5, GPT-5.4, o1/o3/o4 reasoning models, DALL-E 2/3 image generation. - **Google Gemini:** Gemini 2.5 Flash/Pro, Gemini 3 Flash/Pro previews, Imagen 3 image generation. - **DeepSeek:** DeepSeek Chat, Reasoner, V4 Flash, V4 Pro. - **Moonshot:** Kimi K2.5, K2.6 reasoning models. - **xAI Grok:** Grok-3, Grok-4, Grok-4.3 reasoning models. + - **Xiaomi MiMo:** MiMo v2.5 models. - **Ollama:** Local LLMs running on your network. - **Observability & Tracking:** - **Asynchronous Logging:** Non-blocking request logging to SQLite using background workers. - **Token Counting:** Precise estimation and tracking of prompt, completion, and reasoning tokens. - **Database Persistence:** Every request logged to SQLite for historical analysis and dashboard analytics. - - **Streaming Support:** Full SSE (Server-Sent Events) support for all providers. + - **Streaming Support:** Full SSE (Server-Sent Events) support with `X-Accel-Buffering: no` for unbuffered, low-latency streaming. +- **High Performance & Thread Safety:** + - **Connection Pooling:** Shared HTTP transport with connection pooling (`MaxIdleConns: 200`), TCP keep-alives, and HTTP/2 multiplexing across all provider drivers. + - **In-Memory Token Caching:** In-memory `sync.Map` TTL caching (10s valid, 2s negative cache) for client token authentication to eliminate SQLite bottlenecking. + - **Thread-Safe Core:** Full RWMutex locking across provider maps, model registry lookups, and router reloads. - **Multimodal (Vision):** Image processing (Base64 and remote URLs) across compatible providers. - **Image Generation:** DALL-E 2/3 (OpenAI) and Imagen 3 (Gemini) via OpenAI-compatible `/v1/images/generations` endpoint. - **Automatic Model Routing:**