Skip to content
All projects
In progress

ChatWithSherlock

A multi-provider LLM gateway with intelligent routing, failover and rate-limit handling — demonstrated by a Sherlock Holmes chatbot.

On this page
Ended
Ongoing
Type
project

Built with

Overview

One free AI endpoint built from three free LLM providers, demonstrated with a Sherlock Holmes chatbot. Instead of wiring an app to Gemini, Groq and OpenRouter separately — each with its own format, keys and rate limits — a LiteLLM proxy exposes one address and one model name, free-auto; the proxy picks a provider, translates the request, and fails over when one is rate-limited.

How it works

Sherlock app / curl / any OpenAI-compatible client
               │  model="free-auto"
               ▼
     LiteLLM proxy (127.0.0.1:4000)
          │        │        │
       Gemini    Groq   OpenRouter

LiteLLM treats every config entry named free-auto as one pool and does a weighted random pick using each provider's real rate limit (Groq 30 rpm, Gemini 10, OpenRouter 5), translates the request into that provider's native format with its real key attached, and translates the answer back. On a 429, 404 or 5xx it retries on another provider and benches the failing one for 60 seconds. The app itself never sees a provider key — only the proxy's master key.

The Streamlit chatbot layers a Sherlock persona on top: each reply shows which provider and model actually answered and how long it took, and a Dr. Watson fallback answers in-character if the proxy itself is down, instead of showing an error. A persistent visitor counter (Upstash Redis, salted-hash visitor ids, no raw IPs) and a four-step "are you really sure?" contact-reveal flow round out the UI. Public mode caps anonymous visitors at 10 questions, tracked by hashed IP so a refresh doesn't reset it.

Architecture

One container runs both processes: the LiteLLM proxy bound to 127.0.0.1:4000 (unreachable from outside the container) and Streamlit serving the public port. A random proxy master key is generated per boot, so the default sk-local-dev dev credential never ships. On Render: one web service, health check at /_stcore/health, provider keys as env vars, free plan (sleeps after ~15 idle minutes).

Decisions

A short table of threats and the specific fix for each, which is a more honest security section than a paragraph claiming the app "is secure":

RiskFix
API keys committed or baked into the image.env gitignored and dockerignored
Other local users reading keys.env permissions set to 600
Anyone on the network reaching the proxy (LiteLLM binds 0.0.0.0 by default)Bound to 127.0.0.1; in Docker only reachable inside the container
Guessable proxy passwordRandom master key generated per boot
Strangers draining free quotas10 questions/visitor, hidden stress test in public mode, input/history/reply caps
Storing personal dataVisitors stored only as salted SHA-256 hashes; no chat content logged
Email scrapingEmail only rendered client-side after a 4-step confirm flow
Stats-page timing attacksKey compared with hmac.compare_digest; a wrong key shows the normal page, not an error
Container privilegesRuns as a non-root user

Challenges

Model IDs rot fast, and tutorials lag behind. llama-3.3-70b-versatile and mixtral-8x7b-32768 were both already decommissioned by the time this was built; the fix was querying each provider's live /models endpoint instead of trusting a tutorial — including this README, which says so explicitly.

Only one config entry was actually named free-auto, so LiteLLM's "pool" was accidentally a pool of one — every request silently went to Gemini regardless of its rate limit. Renaming all three entries to the same model name was the actual fix, not a routing-strategy change.

Reasoning models spend tokens you can't see. A 800-token reply cap was silently truncating answers because some models spend hidden "thinking" tokens before the visible answer; raising the cap to 2,000 fixed it.

Results

From a real load test: 60/60 chat turns and 60/60 burst requests answered — the proxy silently recovered from roughly 10 induced rate-limit errors during the run. Groq was the fastest provider by far (median 1.1s), OpenRouter 3.7s, Gemini 10.3s. Realistic sustained capacity on free tiers: ~1,000 messages/day, mostly from Groq's allowance. Playwright smoke tests covered the contact flow, the 10-question limit surviving a refresh, and the visitor counter against a real Upstash database (10 simulated visitors counted as 10; a returning visitor correctly not recounted).

Lessons

  • Model IDs go stale fast — check a provider's live /models endpoint, not a tutorial or a chatbot's suggestion.
  • Free tiers have two separate limits — requests per day, and tokens per minute. Long chat histories hit the token limit first, not the request-count limit.
  • Models don't reliably know what they are. Asked which model it was, Sherlock confidently claimed GPT-4 — it was actually gpt-oss-120b on Groq. Trust the proxy's own headers, not the model's self-report.
  • Platform facts change — a planned deploy target (Hugging Face Docker Spaces) turned out to require a paid tier by the time this shipped; verify pricing before planning around it, not after.