Benchmarks
The numbers, with the receipts.
Every figure is measured live on-device against public, independently-labeled datasets — scored by the same code as the baselines, backed by raw counts, and reproducible. No cherry-picking, honest limitations included.
On-device · ~4B model, offline
Public datasets · BIRD · BFCL · AI4Privacy
Reproducible harness
Report 01 · On-device model quality
A 4-billion-parameter model, on a laptop, beats Claude 2 & GPT-3.5 at real SQL.
Measured live against the same compact on-device model the product ships — ~4B parameters, fully offline, $0 per query. BIRD text-to-SQL, official BFCL function-calling, RAG grounding, and system footprint.
NL→SQL · BIRD dev
44.65%
exec-accuracy over 1,534 Q — beats Llama-3-70B, Claude 2, GPT-3.5
Function calling · BFCL
85.7%
Non-Live AST overall — near tool-specialist models
RAG grounding
100%
recall@8 · 95% answers carry the exact fact · abstains when it can't
Footprint
~1.3GB
whole AI stack · 53 tok/s · RAG search 375 ms
- Beats bigger models. 44.65% on BIRD lands above Claude 2 (42.7%), GPT-3.5-turbo (37.2%) and Llama-3-70B (40.8% — 17× our size), 1.7 pts behind GPT-4.
- Not memorized. Rename every table/column to meaningless aliases and it keeps 82% of its accuracy — reasoning over schema structure, not recall.
- Stricter scoring than the leaderboard. We require row multiplicity to match; official BIRD only checks the set. Our number is a floor, not a ceiling.
- Honest. It does not beat 14B+ models or fine-tuned SQL specialists — and we say so, in the report.
Report 02 · Data Leakage Protection
DLP detection, head-to-head vs Microsoft Presidio — on identical texts.
A reproducible evaluation on a public, independently-labeled PII dataset (AI4Privacy · 3,000 texts · 9,491 spans), scored with identical code against the industry-standard open-source PII detector. 100% on-device.
Beats baseline
100.0%
IBAN F1 — ahead of Presidio (99.6%)
Parity
99.8%
Email F1 — tied with Presidio
Real cards
100%
recall on valid (Luhn) card numbers — 2× Presidio's catch
Trust
99.96%
of 2,329 flags landed on real PII — near-zero false alarms
- Like-for-like. Same texts, same span-overlap scoring, same code run over both tools — Presidio 2.2.363 as the baseline.
- Beyond a library. On-device scanning, attachments (PDF/Office/CSV + OCR), egress enforcement (warn or hard-block before send), tamper-evident hash-chained audit.
- Validated, not guessed. Credit card via Luhn, IBAN via mod-97, Singapore NRIC/FIN check-letter, API keys via prefix + entropy.
- Honest limits. Bare free-text names/addresses with no cue aren't caught yet (on-device NER is the roadmap); detection is tuned for English. All in the report.
Reproducibility — Datasets: BIRD dev (bird-bench.github.io), Berkeley Function-Calling Leaderboard (Non-Live AST), AI4Privacy pii-masking-200k (HuggingFace). Baselines: official BIRD-paper single-model results; Microsoft Presidio 2.2.363. Every Pyraxis figure was run live on-device during the report. Competitor scores are their published figures. Figures illustrate capability; internal implementation is proprietary.