chore: initial clean commit

OPTIMIZATION.md

@@ -0,0 +1,232 @@
+# Optimization for 512MB RAM Environment
+
+This document provides guidance for optimizing the LLM Proxy Gateway for deployment in resource-constrained environments (512MB RAM).
+
+## Memory Optimization Strategies
+
+### 1. Build Optimization
+
+The project is already configured with optimized build settings in `Cargo.toml`:
+
+```toml
+[profile.release]
+opt-level = 3      # Maximum optimization
+lto = true         # Link-time optimization
+codegen-units = 1  # Single codegen unit for better optimization
+strip = true       # Strip debug symbols
+```
+
+**Additional optimizations you can apply:**
+
+```bash
+# Build with specific target for better optimization
+cargo build --release --target x86_64-unknown-linux-musl
+
+# Or for ARM (Raspberry Pi, etc.)
+cargo build --release --target aarch64-unknown-linux-musl
+```
+
+### 2. Runtime Memory Management
+
+#### Database Connection Pool
+- Default: 10 connections
+- Recommended for 512MB: 5 connections
+
+Update `config.toml`:
+```toml
+[database]
+max_connections = 5
+```
+
+#### Rate Limiting Memory Usage
+- Client rate limit buckets: Store in memory
+- Circuit breakers: Minimal memory usage
+- Consider reducing burst capacity if memory is critical
+
+#### Provider Management
+- Only enable providers you actually use
+- Disable unused providers in configuration
+
+### 3. Configuration for Low Memory
+
+Create a `config-low-memory.toml`:
+
+```toml
+[server]
+port = 8080
+host = "0.0.0.0"
+
+[database]
+path = "./data/llm_proxy.db"
+max_connections = 3  # Reduced from default 10
+
+[providers]
+# Only enable providers you need
+openai.enabled = true
+gemini.enabled = false  # Disable if not used
+deepseek.enabled = false  # Disable if not used
+grok.enabled = false  # Disable if not used
+
+[rate_limiting]
+# Reduce memory usage for rate limiting
+client_requests_per_minute = 30  # Reduced from 60
+client_burst_size = 5  # Reduced from 10
+global_requests_per_minute = 300  # Reduced from 600
+```
+
+### 4. System-Level Optimizations
+
+#### Linux Kernel Parameters
+Add to `/etc/sysctl.conf`:
+```bash
+# Reduce TCP buffer sizes
+net.ipv4.tcp_rmem = 4096 87380 174760
+net.ipv4.tcp_wmem = 4096 65536 131072
+
+# Reduce connection tracking
+net.netfilter.nf_conntrack_max = 65536
+net.netfilter.nf_conntrack_tcp_timeout_established = 1200
+
+# Reduce socket buffer sizes
+net.core.rmem_max = 131072
+net.core.wmem_max = 131072
+net.core.rmem_default = 65536
+net.core.wmem_default = 65536
+```
+
+#### Systemd Service Configuration
+Create `/etc/systemd/system/llm-proxy.service`:
+```ini
+[Unit]
+Description=LLM Proxy Gateway
+After=network.target
+
+[Service]
+Type=simple
+User=llmproxy
+Group=llmproxy
+WorkingDirectory=/opt/llm-proxy
+ExecStart=/opt/llm-proxy/llm-proxy
+Restart=on-failure
+RestartSec=5
+
+# Memory limits
+MemoryMax=400M
+MemorySwapMax=100M
+
+# CPU limits
+CPUQuota=50%
+
+# Process limits
+LimitNOFILE=65536
+LimitNPROC=512
+
+Environment="RUST_LOG=info"
+Environment="LLM_PROXY__DATABASE__MAX_CONNECTIONS=3"
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### 5. Application-Specific Optimizations
+
+#### Disable Unused Features
+- **Multimodal support**: If not using images, disable image processing dependencies
+- **Dashboard**: The dashboard uses WebSockets and additional memory. Consider disabling if not needed.
+- **Detailed logging**: Reduce log verbosity in production
+
+#### Memory Pool Sizes
+The application uses several memory pools:
+1. **Database connection pool**: Configured via `max_connections`
+2. **HTTP client pool**: Reqwest client pool (defaults to reasonable values)
+3. **Async runtime**: Tokio worker threads
+
+Reduce Tokio worker threads for low-core systems:
+```rust
+// In main.rs, modify tokio runtime creation
+#[tokio::main(flavor = "current_thread")]  // Single-threaded runtime
+async fn main() -> Result<()> {
+    // Or for multi-threaded with limited threads:
+    // #[tokio::main(worker_threads = 2)]
+```
+
+### 6. Monitoring and Profiling
+
+#### Memory Usage Monitoring
+```bash
+# Install heaptrack for memory profiling
+cargo install heaptrack
+
+# Profile memory usage
+heaptrack ./target/release/llm-proxy
+
+# Monitor with ps
+ps aux --sort=-%mem | head -10
+
+# Monitor with top
+top -p $(pgrep llm-proxy)
+```
+
+#### Performance Benchmarks
+Test with different configurations:
+```bash
+# Test with 100 concurrent connections
+wrk -t4 -c100 -d30s http://localhost:8080/health
+
+# Test chat completion endpoint
+ab -n 1000 -c 10 -p test_request.json -T application/json http://localhost:8080/v1/chat/completions
+```
+
+### 7. Deployment Checklist for 512MB RAM
+
+- [ ] Build with release profile: `cargo build --release`
+- [ ] Configure database with `max_connections = 3`
+- [ ] Disable unused providers in configuration
+- [ ] Set appropriate rate limiting limits
+- [ ] Configure systemd with memory limits
+- [ ] Set up log rotation to prevent disk space issues
+- [ ] Monitor memory usage during initial deployment
+- [ ] Consider using swap space (512MB-1GB) for safety
+
+### 8. Troubleshooting High Memory Usage
+
+#### Common Issues and Solutions:
+
+1. **Database connection leaks**: Ensure connections are properly closed
+2. **Memory fragmentation**: Use jemalloc or mimalloc as allocator
+3. **Unbounded queues**: Check WebSocket message queues
+4. **Cache growth**: Implement cache limits or TTL
+
+#### Add to Cargo.toml for alternative allocator:
+```toml
+[dependencies]
+mimalloc = { version = "0.1", default-features = false }
+
+[features]
+default = ["mimalloc"]
+```
+
+#### In main.rs:
+```rust
+#[global_allocator]
+static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
+```
+
+### 9. Expected Memory Usage
+
+| Component | Baseline | With 10 clients | With 100 clients |
+|-----------|----------|-----------------|------------------|
+| Base executable | 15MB | 15MB | 15MB |
+| Database connections | 5MB | 8MB | 15MB |
+| Rate limiting | 2MB | 5MB | 20MB |
+| HTTP clients | 3MB | 5MB | 10MB |
+| **Total** | **25MB** | **33MB** | **60MB** |
+
+**Note**: These are estimates. Actual usage depends on request volume, payload sizes, and configuration.
+
+### 10. Further Reading
+
+- [Tokio performance guide](https://tokio.rs/tokio/topics/performance)
+- [Rust performance book](https://nnethercote.github.io/perf-book/)
+- [Linux memory management](https://www.kernel.org/doc/html/latest/admin-guide/mm/)
+- [SQLite performance tips](https://www.sqlite.org/faq.html#q19)