You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
DriftBench is an open-source, fully deterministic benchmark (Apache 2.0) for how faithfully an AI tracks a user's changing beliefs across a long conversation — scored reproducibly, with no LLM judge. This grant ships v1.1: a new deterministic "ambivalence" metric (working prototype already in-repo) and a 3x larger, multi-domain scenario library.
DriftBench measures — deterministically, with no LLM judge — whether an AI correctly tracks how a user's beliefs, conflicts, and identity change across a long conversation.
Why it matters: systems that track a person's beliefs are increasingly deployed in high-stakes settings — mental-health support, coaching, advice — where misreading how someone's state is changing can cause real harm. Deterministic, reproducible evaluation lets anyone check how well a system actually does this, without trusting a vendor's own claims.
This funding covers one concrete, shippable bundle:
- Promote the working "ambivalence" (AMB) metric from prototype to a calibrated, tested metric in the core. AMB scores when an AI recognizes a person holds two conflicting beliefs at once (e.g. loyalty to the past and openness to the new) rather than treating it as a simple switch — a distinction existing direction-only metrics miss. Prototype already runs at research/v1_1_draft/ambivalence.py.
- Expand the scenario library from 7 to 20+, across new domains: relationships, health, money & risk, grief & loss, addiction & recovery (today's are all career/identity).
- Ship it: tagged v1.1 release, updated specification, and a public leaderboard.
All deliverables are verifiable in the open repository as they land.
Solo developer time over ~3 months to design domain ontologies, author and validate 13+ new scenarios, calibrate and test the AMB metric, and cut the v1.1 release, plus a small amount for LLM API costs. Based in Ukraine, where costs are well below US/EU levels, so the funding goes far. At the $5,000 minimum I'd ship the AMB metric plus ~5 new-domain scenarios; the full $15,000 funds the complete 20+ scenario library and release.
I'm the sole creator of DriftBench. To be upfront: I'm not a professional programmer — I designed the system (ontology, metrics, validation rules, scenarios) and built it by directing AI coding tools (mainly Claude) as my implementer, while I act as architect and tester. The deterministic, fully test-covered design is what makes this work: correctness is verifiable against fixed expected outputs, not dependent on my hand-coding.
Already shipped, open-source (Apache 2.0): deterministic scoring engine, five metrics, frozen ontology, seven scenarios, zero-trust validation with cryptographic integrity checks, adapter architecture, CI. The AMB prototype and a full new-domain (grief) scenario already run in-repo. https://github.com/simon9679/driftbench
The main risk is calibration: ambivalence thresholds need tuning so AMB is neither trivially easy nor impossible across diverse scenarios. This is engineering, not open research — the metric is fully deterministic and already runs; the work is validating thresholds against a broader library. Scenario authoring is labor but low-risk.
None raised yet. Applications pending: Long-Term Future Fund ($20k, for a separate adversarial safety-evaluation track), Emergent Ventures ($12k, for the base v1.1 deliverables), and Anthropic Claude for OSS (non-cash).