Project summary

AI evaluation is rapidly becoming the primary gatekeeper for frontier model deployment, yet the benchmark scoring functions themselves are almost never unit-tested or audited for determinism.

If you take a standard multiple-choice evaluation task and make minor cosmetic adjustments—such as shuffling answer choices (A/B/C/D), changing character casing, or adding a trailing newline—model scores often swing by 3 to 6 percentage points. When national safety institutes and frontier labs use these exact benchmarks to determine whether a model exceeds dangerous capability thresholds (ASL-3 containment triggers or autonomous cyber capabilities), this variance is not acceptable experimental noise; it is an evaluation safety vulnerability.

Most of the AI safety community's attention is focused on generating new benchmark questions and collecting human preference data. Very little engineering is dedicated to verifying the determinism and mathematical invariance of the scoring harnesses themselves. If a scoring script breaks under whitespace shifts or prompt permutations, the benchmark is not measuring model reasoning—it is measuring formatting sensitivity.

eval-invariance-engine is an open-source verification CLI and Python framework that treats benchmark scorers as systems requiring formal invariance testing. By executing configurable Monte Carlo perturbation chains across thousands of iterations, it empirically proves whether a scoring function satisfies the Zero-Variance Invariant or suffers from hidden flakiness.

What are this project's goals? How will you achieve them?

Where this project will be in 45 days:

1. UK AISI inspect_ai Native Integration: A drop-in evaluation adapter allowing anyone running benchmarks in the UK AI Safety Institute's Inspect framework to verify scorer invariance with a single CLI flag.

2. Public Flakiness Audit of 5 Open-Source Eval Suites: Comprehensive stress-testing of 5 widely used open-source benchmark task runners (including MMLU and GSM8K runners), documenting specific scoring vulnerabilities and publishing the full dataset on Hugging Face.

3. Multi-Model Sensitivity Dataset: Running 100,000+ perturbation trials across 20+ frontier and open-weight models (OpenAI, Anthropic, Google, DeepSeek) to map out how prompt formatting noise skews published safety leaderboards.

4. Open-Access Technical Report: A publication detailing common failure modes in evaluation regular expressions, answer extractors, and seed management, along with deterministic mitigation patterns.

How will this funding be used?

The funding is structured into three tiers:

• $8,000 (Minimum Viable): Funds 25 days of dedicated engineering to deliver the native UK AISI Inspect AI adapter, audit 2 flagship benchmark suites, and maintain the open-source repository.

• $18,000 (Scale-up): Expands the audit to 5 benchmark suites, funds multi-model API calls across 10 frontier models, and covers dedicated CI test runners for continuous benchmark scanning.

• $25,000 (Target): Delivers the full 45-day sprint: the Inspect AI plugin, the 20+ model sensitivity dataset, dedicated cloud infrastructure, and the open-access technical report published on arXiv.

Budget Breakdown ($25,000 Target):

• Engineering Stipend: $15,750 (45 days full−time @ $350/day)

• Model API and Compute Pool: $6,500 (API token inference across OpenAI, Anthropic, Google, and open-weight models)

• Cloud Runner Infrastructure: $1,750 (Dedicated sandboxed execution nodes and storage)

• Dataset Hosting and Open Publication: $1,000 (Hugging Face dataset hosting and preprint publication)

Who is on your team? What's your track record on similar projects?

Justin Arndt is a systems architect and independent AI safety engineer specializing in automated verification, formal invariants, and high-assurance evaluation infrastructure.

Track Record (Proof of Work):

• Built and published the working eval-invariance-engine core repository prior to requesting funding: https://github.com/j-arndt/eval-invariance-engine

• 31 unit, property, and stress tests passing with 97% codebase coverage and green GitHub Actions CI on Python 3.12.

• Implemented Hypothesis property-based test suites proving mathematical variance bounds on arbitrary input strings.

• Stress-tested deterministic scorers against 10,000 Monte Carlo iterations to verify zero floating-point drift.

What are the most likely causes and outcomes if this project fails?

1. Adoption Friction: Evaluation authors might continue using ad-hoc regex scorers without running pre-flight checks. Mitigation: Building the tool directly into UK AISI Inspect AI ensures zero-friction adoption for researchers already using standard safety harnesses.

2. Model Provider API Flakiness: Upstream rate limits or non-deterministic token routing from closed model providers could slow down multi-model trials. Mitigation: The engine is local-first, caching all prompt permutations and supporting local open-weight model backends.

3. Low Financial Risk: The core tool is already operational; funding directly finances the ecosystem integrations and public benchmark audits.

How much money have you raised in the last 12 months, and from where?

$0. This project has been entirely self-funded and built independently.