Skip to content
All projects
ShippedFeatured

PulseIQ

Upload a spreadsheet, ask questions in plain English, and get answers whose every number was computed by real code — then checked — not guessed by a language model.

On this page
Type
project

Built with

Overview

Upload a spreadsheet, ask it questions in plain English, and get answers whose every number was computed by real code — then checked — not guessed by a language model.

An AI Analyst decides which of 16 deterministic analysis tools to run against the full dataset, not a sample, and every finding it reports is cross-checked against those tools' real output before it reaches the user. The page shows which tools ran and which findings were verified. Alongside that: natural-language-to-SQL with a real security-audited validator, a dataset explorer, dashboards, saved insights, anomaly monitors and a usage page showing today's AI spend against a daily budget.

Live demo with a shared account and a daily AI budget that resets automatically on every deploy.

The problem

Asking a language model to analyse a spreadsheet produces confident numbers that nobody can verify. The interesting constraint was making the model choose what to compute while real code does the computing — and then checking the result before showing it.

Architecture

Browser (React + TanStack Query) — X-Request-ID
  ▼
FastAPI
  ├─ JWT auth + per-user isolation
  ├─ AI Analyst engine (tool-calling loop) ⇄ Groq LLM
  │     └─ 16 deterministic tools (Polars)
  │     └─ Answer validator — checks every reported number against tool output
  ├─ SQL validator (sqlglot AST) → DuckDB (external access disabled)
  ├─ PostgreSQL (Neon)
  ├─ StorageProvider — local disk now, R2-ready
  └─ optional OTLP tracing

One request carries one X-Request-ID end to end — the browser sets it, every backend log line down to each tool call and LLM call carries it, and the root span of its trace does too when tracing is on.

Testing

316 backend tests and 24 frontend tests, with ruff, mypy, tsc and oxlint all clean. CI runs on every pull request: backend, frontend, the eval harness's deterministic checks, and a Docker image build.

19 bugs are tracked with root cause, fix and verification in the project's bug log — several found by running the real app rather than reading the code, one found only by inspecting a live trace.

A 51-question evaluation harness covers all 16 tools, NL-to-SQL and multi-step questions, with ground truth computed by the app's own tools, never by the model.

Decisions

The model never computes a number. Language models are good at choosing which analysis answers a question and bad at arithmetic they can't check. So the model only picks tools; 16 deterministic tools compute everything, and a validator checks every reported finding against the tools' actual output. A finding that can't be matched is shown and flagged, not hidden — hiding it would make the validator pointless to the person reading it.

NL-to-SQL is kept safe by construction, then attacked to prove it. Generated SQL is parsed into a real syntax tree and must be a single SELECT over the user's own dataset with known columns, standard functions and a clamped row limit; DuckDB then runs with external file/network access disabled as an independent second layer. Attacking the validator with real payloads found two genuine bypasses the original tests had missed — a bare version() call executing, and an oversized LIMIT skipping the row cap — both fixed and regression-tested. That's the result of deliberately trying to break your own safety layer instead of trusting that it works.

Rate limits are handled, not just retried. Retries back off with full jitter honoring the provider's suggested wait, but capped at 8 seconds — a request can't be allowed to hang for the 18-minute wait Groq once suggested after a quota ran out. Repeated questions are served from cache, and per-user daily token budgets stop one user from spending the whole account's quota.

Results

Honest status: the full eval run hasn't completed. It stopped at question 27 of 51 on Groq's free-tier daily limit of 200,000 tokens — 26 of 27 attempted questions were answered before the cap. Token cost per /analyze question spot-checked at 3,227–6,651 tokens; a repeated question costs 0 tokens, served from cache. Full numbers are documented to land once a complete run finishes — nothing is estimated in the meantime.

That's a deliberate choice: report what was actually measured, including where measurement got cut off, rather than project a number from a partial run.