Project summary

I’m building AI Evaluation Lab, an independent public evaluation project for tool-using AI systems and agents.

The main idea is that getting the right final answer is not enough. An agent can complete a task correctly while still violating an execution rule, using a prohibited shortcut, escalating to a more powerful tool than necessary, or gaming the evaluation process.

The Lab is designed to capture that difference using frozen task contracts, positive and known-bad controls, trusted runner-generated traces, deterministic verification where possible, normalized Evaluation Records and regression evidence.

The first substantive evaluation, AS-001 v1.0.0, is already built and public. It tests whether an agent can complete a legitimate task while respecting a defined resource boundary.

Six live runs have already been published under the same frozen contract and verifier. Three Ollama llama3:latest runs completed the task objective and preserved the tested permission boundary, but failed an exactly-one-write execution invariant. Three Groq openai/gpt-oss-20b runs completed the objective, preserved the boundary and passed the full verifier.

I do not interpret this as evidence that one model is generally safer or better. The result is narrower: trajectory-level verification exposed a difference in execution behavior that final-answer scoring alone would not have shown.

The next step is to expand this from one real evaluation into a small public evaluation program.

What are this project's goals? How will you achieve them?

The immediate goal is to build a small set of reproducible, verifier-backed evaluations for safety-relevant agent behavior.

First, I want to run AS-001 against a third compatible model/provider configuration without changing the frozen v1.0.0 task contract or verifier.

Second, I will build AS-002, focused on least-privilege tool selection and unnecessary tool escalation. The task will test whether an agent uses a sufficient low-impact action when a more powerful shortcut is available.

Third, I will build AS-003, focused on evaluator gaming and protected evaluation state. The task will distinguish genuine task completion from attempts to manipulate protected scoring or reference artifacts.

The funded project will also add reusable regression tooling so failures can be rerun after changes, rather than treated as one-off benchmark events.

Each task will include:

• a versioned task contract;

• synthetic, independently authored fixtures;

• a positive/reference control;

• a known-bad control;

• machine-checkable verification where feasible;

• trusted execution evidence;

• normalized Evaluation Records;

• reproducibility instructions.

The initial success floor is at least 18 new normalized live Evaluation Records across the funded task family, with additional runs performed where they materially improve replication, failure analysis or cross-configuration comparison.

If time and funding allow, AS-004 and AS-005 will extend the program into regression-after-repair and untrusted-instruction / authority-boundary evaluation.

The public output will include task packs, verifier logic, evaluation records, regression artifacts and a technical report covering both positive and negative findings.

How will this funding be used?

I am requesting up to $20,000, with a $5,000 minimum viable funding level.

$6,000 — model/API inference and compute
Repeated evaluation runs across multiple model/provider configurations, replication runs, regression testing and targeted follow-up experiments.

$4,000 — isolated cloud and container execution
Ephemeral workers, CI compute, sandboxed execution environments and controlled provider execution.

$3,000 — independent technical review and adjudication
External review of task contracts, verifier logic, ambiguous results and technical-report claims.

$2,000 — evidence storage and observability
Artifact storage, normalized evidence, run metadata, logs and reproducibility retention.

$2,500 — sandbox, integration and security tooling
Provider integration, execution isolation, egress controls and evaluation-specific infrastructure.

$2,500 — reproducibility and public reporting
Public artifact hosting, data transfer, reproducibility packaging and technical-report publication.

Total goal: $20,000.

At the $5,000 minimum level, I can still complete:

• third-provider AS-001 validation;

• AS-002;

• AS-003;

• initial regression tooling;

• a first public cross-task research release.

I will contribute the core engineering work myself. The requested funding is for project execution and external resources, not personal equipment or unrelated expenses.

Who is on your team? What's your track record on similar projects?

I currently lead and engineer AI Evaluation Lab independently.

The strongest evidence of execution is the public repository itself.

I have already built the evaluation path end-to-end:
frozen task contracts, positive and known-bad controls, trusted runner-authored traces, deterministic verification, provider-configurable model execution, normalized Evaluation Records, canonical record hashing, CI evidence validation and public cross-provider evidence.

AS-001 is already live rather than proposed.

Its contract was frozen before model execution, and six public runs are now available under the same contract and verifier.

The interesting part of the first result is that both evaluated configurations completed the task and respected the tested permission boundary, while the verifier still detected a meaningful execution-level difference.

Because I am currently the primary engineer, I am also deliberately allocating some funding to independent technical review rather than presenting internal review as independent validation.

What are the most likely causes and outcomes if this project fails?

The biggest risks are methodological rather than engineering.

Some task designs may turn out to be too synthetic to support useful conclusions.

Provider integrations may introduce confounds that make comparisons difficult.

A verifier may contain assumptions that need to be revised before the result is trustworthy.

Some experiments may produce null results or too little variation to support a meaningful comparison.

I do not consider null results to be a failure. If models consistently satisfy a tested constraint, or if a task turns out not to measure what I intended, that result should still be documented.

I would consider the project unsuccessful if it produced impressive-looking results that could not be reproduced, or if its public claims went beyond the evidence.

The main safeguards are:

• freezing task contracts before interpreting results;

• using reference and known-bad controls;

• separating provider/integration failures from model-evaluation outcomes;

• preserving negative and inconclusive findings;

• using deterministic verification where possible;

• inviting independent technical review.

Even if the broader program does not scale as hoped, the task packs, verifier code, Evaluation Records and methodological lessons will remain public and reusable.

How much money have you raised in the last 12 months, and from where?

AI Evaluation Lab has not yet received a cash grant.

I have submitted a $10,000 BlueDot Rapid Grant application, which is currently pending.

Development so far has mainly been self-funded and supported through local, free and credited technical resources.

If multiple funders support overlapping parts of the project, I will disclose that and adjust budgets so the same costs are not funded twice.

Main repository:
https://github.com/kaulastudies/ai-evaluation-lab

AS-001 methodology/results:
https://github.com/kaulastudies/ai-evaluation-lab/blob/main/evaluations/AS-001/README.md

Groq evidence:
https://github.com/kaulastudies/ai-evaluation-lab/tree/main/runs/AS-001/groq/gpt-oss-20b

Ollama evidence:
https://github.com/kaulastudies/ai-evaluation-lab/tree/main/runs/AS-001/ollama/llama3-latest

Cross-provider PR:
https://github.com/kaulastudies/ai-evaluation-lab/pull/9