ChatWithSherlock
A multi-provider LLM gateway with intelligent routing, failover and rate-limit handling — demonstrated by a Sherlock Holmes chatbot.
On this page
- Ended
- Ongoing
- Type
- project
Built with
Overview
One free AI endpoint built from three free LLM providers, demonstrated with a Sherlock Holmes chatbot. Instead of wiring an app to Gemini, Groq and OpenRouter separately — each with its own format, keys and rate limits — a LiteLLM proxy exposes one address and one model name, free-auto; the proxy picks a provider, translates the request, and fails over when one is rate-limited.
How it works
Sherlock app / curl / any OpenAI-compatible client
│ model="free-auto"
▼
LiteLLM proxy (127.0.0.1:4000)
│ │ │
Gemini Groq OpenRouter
LiteLLM treats every config entry named free-auto as one pool and does a weighted random pick using each provider's real rate limit (Groq 30 rpm, Gemini 10, OpenRouter 5), translates the request into that provider's native format with its real key attached, and translates the answer back. On a 429, 404 or 5xx it retries on another provider and benches the failing one for 60 seconds. The app itself never sees a provider key — only the proxy's master key.
The Streamlit chatbot layers a Sherlock persona on top: each reply shows which provider and model actually answered and how long it took, and a Dr. Watson fallback answers in-character if the proxy itself is down, instead of showing an error. A persistent visitor counter (Upstash Redis, salted-hash visitor ids, no raw IPs) and a four-step "are you really sure?" contact-reveal flow round out the UI. Public mode caps anonymous visitors at 10 questions, tracked by hashed IP so a refresh doesn't reset it.
Architecture
One container runs both processes: the LiteLLM proxy bound to 127.0.0.1:4000 (unreachable from outside the container) and Streamlit serving the public port. A random proxy master key is generated per boot, so the default sk-local-dev dev credential never ships. On Render: one web service, health check at /_stcore/health, provider keys as env vars, free plan (sleeps after ~15 idle minutes).
Decisions
A short table of threats and the specific fix for each, which is a more honest security section than a paragraph claiming the app "is secure":
| Risk | Fix |
|---|---|
| API keys committed or baked into the image | .env gitignored and dockerignored |
| Other local users reading keys | .env permissions set to 600 |
Anyone on the network reaching the proxy (LiteLLM binds 0.0.0.0 by default) | Bound to 127.0.0.1; in Docker only reachable inside the container |
| Guessable proxy password | Random master key generated per boot |
| Strangers draining free quotas | 10 questions/visitor, hidden stress test in public mode, input/history/reply caps |
| Storing personal data | Visitors stored only as salted SHA-256 hashes; no chat content logged |
| Email scraping | Email only rendered client-side after a 4-step confirm flow |
| Stats-page timing attacks | Key compared with hmac.compare_digest; a wrong key shows the normal page, not an error |
| Container privileges | Runs as a non-root user |
Challenges
Model IDs rot fast, and tutorials lag behind. llama-3.3-70b-versatile and mixtral-8x7b-32768 were both already decommissioned by the time this was built; the fix was querying each provider's live /models endpoint instead of trusting a tutorial — including this README, which says so explicitly.
Only one config entry was actually named free-auto, so LiteLLM's "pool" was accidentally a pool of one — every request silently went to Gemini regardless of its rate limit. Renaming all three entries to the same model name was the actual fix, not a routing-strategy change.
Reasoning models spend tokens you can't see. A 800-token reply cap was silently truncating answers because some models spend hidden "thinking" tokens before the visible answer; raising the cap to 2,000 fixed it.
Results
From a real load test: 60/60 chat turns and 60/60 burst requests answered — the proxy silently recovered from roughly 10 induced rate-limit errors during the run. Groq was the fastest provider by far (median 1.1s), OpenRouter 3.7s, Gemini 10.3s. Realistic sustained capacity on free tiers: ~1,000 messages/day, mostly from Groq's allowance. Playwright smoke tests covered the contact flow, the 10-question limit surviving a refresh, and the visitor counter against a real Upstash database (10 simulated visitors counted as 10; a returning visitor correctly not recounted).
Lessons
- Model IDs go stale fast — check a provider's live
/modelsendpoint, not a tutorial or a chatbot's suggestion. - Free tiers have two separate limits — requests per day, and tokens per minute. Long chat histories hit the token limit first, not the request-count limit.
- Models don't reliably know what they are. Asked which model it was, Sherlock confidently claimed GPT-4 — it was actually
gpt-oss-120bon Groq. Trust the proxy's own headers, not the model's self-report. - Platform facts change — a planned deploy target (Hugging Face Docker Spaces) turned out to require a paid tier by the time this shipped; verify pricing before planning around it, not after.