Why I am pursuing this research

My research begins with a question that I believe deserves greater attention in the evaluation of AI agents: when an evaluation reports that an agent has succeeded, what evidence allows us to verify that conclusion?

I have been studying the relationship between an agent’s actions, the results attributed to those actions, and the evidence used to support scientific claims about its performance.

An agent may complete its assigned task while also exhibiting behavior that raises security or policy concerns. Likewise, an evaluation may produce a numerical result without preserving a sufficiently clear record of how that result was obtained.

These are not identical problems, but they point to the same methodological concern. We should not confuse a reported outcome with the integrity of the evidence supporting it.

This concern led me to develop AgentDojo Trace Audit, a research-software project focused on the reconstruction and verification of evidence derived from AI-agent execution records.

What my previous work established

In the published AgentDojo Trace Audit implementation, I worked with a frozen AgentDojo v1.2.1 corpus containing 11,624 selected records.

The software reconstructs relationships between source records, selection rules, denominators, derived results and the claims associated with them.

I wanted these relationships to be inspectable rather than assumed.

The research also involved deliberately introducing faults into controlled datasets to examine whether the software could detect them and identify their locations.

One result is particularly important to me. In a frozen test catalog containing 408 mutations, the software detected and classified every mutation, but correctly localized only 336. I preserved the remaining 72 localization failures in the published evidence.

Later, a separately frozen post-repair test catalog achieved detection, classification and localization for 360 out of 360 mutations. Two benign-control suites produced no alarms in 120 tests.

I regard these findings as evidence about the tested implementation under specified conditions. They are not proof of general AI safety, nor do they establish that the software will perform equally well on unrelated datasets.

Independent external replication remains an open requirement.

I believe scientific credibility depends partly on making these distinctions visible, including the findings that do not support our expectations.

The problem I want to address

Many AI evaluation workflows concentrate on final scores, success rates and benchmark comparisons.

Those measurements are useful, but they do not always reveal whether every relevant execution record was included, whether exclusions were justified, or whether an individual conclusion can be reconstructed from the underlying data.

I am interested in the point where raw technical records become scientific evidence.

If a record is missing, its absence should be reported. If an exclusion changes a denominator, that change should be visible. If the available evidence is insufficient to support a conclusion, the system should not silently treat the conclusion as verified.

My proposed project addresses these problems through a practical, open-source Python toolkit.

What I intend to develop

I want researchers and smaller technical teams to have a way to examine the evidence behind AI-agent evaluations without building a separate auditing framework for every project.

The toolkit will concentrate on:

• Reconstructing links between source records and derived evaluation results.

• Detecting missing records, duplicate entries and inconsistent population counts.

• Examining exclusions and changes in evaluation denominators.

• Testing evidence-integrity checks against controlled synthetic faults.

• Generating machine-readable reports that distinguish verified structural checks from unresolved problems.

I intend to use explicit PASS and HOLD states so that an unresolved problem is not mistaken for successful verification.

The software will examine evidence structure and consistency. It will not independently authenticate every original record, establish the scientific validity of a benchmark, or certify an AI system as safe.

Those boundaries are part of the research design.

Research plan and funding

I am requesting $15,000 to support a twelve-week development and evaluation program.

The minimum funding threshold is $7,500.

During the first six weeks, I plan to produce an initial documented audit schema, a Python command-line auditing tool, at least 40 reproducible synthetic test cases, automated tests and public documentation.

That initial phase will provide a defined deliverable even if the project receives only its minimum funding.

With full funding, I intend to extend the work to at least 150 controlled test cases, develop versioned audit reports and investigate compatibility with two publicly documented evaluation-data schemas, subject to access and licensing conditions.

The completed release will include documentation, reproducibility instructions, test results and an assessment of the software’s limitations.

The proposed budget allocates $9,000 to research engineering and documentation, $1,750 to synthetic fixtures and validation, $1,250 to documentation and release testing, $1,700 to external technical feedback and compatibility work, $550 to testing and archival resources, and $750 to contingency and maintenance handover.

These are planned allocations, not expenditures already made. External technical reviewers have not yet been contracted.

How I will judge the results

I do not consider the number of implemented features a sufficient measure of scientific progress.

I want the resulting software to answer concrete questions.

Can another researcher reconstruct the evaluation records used in a reported result?

Can the software identify deliberately introduced faults?

Does it produce false alarms on valid records?

Does it clearly distinguish a detected problem from a problem whose precise location remains unknown?

Can another developer reproduce the published tests and examine their limitations?

The answers, including negative findings, will be included in the public technical reports.

I will publish implementation updates and make the funded research outputs publicly accessible under appropriate open-source licenses.

About me and the existing research

My name is Abdullah Alharbi. I am an independent researcher and Python developer based in Saudi Arabia.

My work concentrates on AI-agent evaluation, scientific reproducibility, evidence integrity and the relationship between technical outcomes and the claims made about them.

I have developed this research independently, without presenting it as an institutional certification program or claiming validation beyond the published evidence.

I see this project as a continuation of a specific research problem, not an attempt to provide a universal solution to AI evaluation.

My objective is to make the evidential foundations of reported results more accessible to inspection, criticism and independent reproduction.

A credible evaluation should allow us to understand not only what a system reportedly achieved, but also what the available evidence can genuinely establish.

That is the contribution I hope to make through this project.

Research references

Published software and source code:
https://github.com/albdrani37-hash/agentdojo-trace-audit

Archived research release:
https://doi.org/10.5281/zenodo.22217485

ORCID:
https://orcid.org/0009-0009-7201-0295