Project summary

Agentic AI systems are rapidly moving from answering questions to planning، using tools, executing multi step workflows' interacting with external systems' and operating with increasing autonomy. Yet a major gap remains between demonstrating that an agent is capable of completing a task and establishing that it can operate reliably under realistic conditions.

Most current evaluations primarily answer questions about capability Can the system solve the task?? They provide much weaker answers to operational questions such as

1/Does the agent remain reliable when faults,'interruptions' or unexpected conditions occur?

2/ Can it recover from failures rather than simply succeed in ideal conditions??

3/ Is its behavior traceable and reproducible across evaluation contexts??

4/Can reliability claims be independently verified from structured evidence??

5/Can reliability be monitored continuously after deployment rather than measured only once in an offline benchmark??

HB-Eval was created to address this gap.

HB-Eval is an open infrastructure project for measuring and assuring the operational reliability of agentic AI systems

Its central premise is that capability and operational reliability are different properties

an agent may demonstrate strong task performance while still failing unpredictably when planning degrades, tools fail' context changes' memory becomes unreliable' or execution is disrupted.

The project therefore focuses on the operational reliability gap between nominal capability and realworld behavior under evaluation and stress conditions.

The HB-Eval framework brings together multiple components that are usually treated separately multi-metric reliability measurement, controlled fault injection' planning and recovery evaluation' provenance aware evidence reproducible evaluation identities' reliability records' and mechanisms for monitoring and verifying operational claims.

Rather than reducing reliability to a single benchmark score' HB-Eval is designed to make reliability claims inspectable.

A result should be connected to the context in which it was measured' the metric definitions used' the evaluation configuration' and the evidence required to verify what the result does and does not establish.

The project has progressed through a research and engineering program that includes a four paper research agenda around the distinction between capability and operational reliability' multi metric reliability evaluation' fault and recovery behavior'and the requirements for verifiable operational assurance. In parallel ' this work has been translated into an open source technical platform rather than remaining only a conceptual framework.

The current HB-Eval platform provides the foundation for evaluating agentic systems through structured reliability measurements and operational evaluation workflows.

The next challenge is no longer simply proving that the idea can be implemented.

It is establishing HB-Eval as independently reproducible interoperable' and broadly usable open infrastructure for the AI safety and agent evaluation community.

This funding would support that transition.

The goal is to move from a Researcher led platform into an open reliability infrastructure that can be tested across independent agent architectures' reproduced by external researchers' integrated into real evaluation workflows' and developed into a stronger basis for operational assurance of increasingly autonomous AI systems.

Our long term objective is to help establish a missing layer in the agentic AI ecosystem :

not only measuring what an AI agent can do' but providing structured' reproducible and independently verifiable evidence about how reliably it operates.

What are this project's goals? How will you achieve them?

The primary goal of this project is to develop HB-Eval from an existing operational reliability platform into open' independently reproducible infrastructure for evaluating the reliability of agentic AI systems.

We will pursue this through five connected objectives.

1/Validate HB-Eval across independent agent systems

HB-Eval must not remain a framework demonstrated only within its original development environment.

We will evaluate the infrastructure across multiple agent architectures'task environments' and operational conditions.

This will test whether the framework can provide useful and consistent reliability measurements beyond a single implementation.

The objective is not to prove that every agent is comparable or to create a universal ranking.

Instead' it is to establish when reliability measurements are meaningful' reproducible' and supported by sufficiently compatible evaluation contexts.

2/ Build independent reproducibility and verification

A central weakness of many AI evaluation claims is that external researchers cannot easily determine what was actually measured or reproduce the conditions under which a result was produced.

We will strengthen the reproducibility layer around HB-Eval by improving structured evaluation records, provenance' evaluation identity' verification mechanisms'and documentation.

The objective is to make it possible for external users to inspect :

A- what was evaluated,

B- under what declared context'

C- which metric definitions were used

D- what evidence supports the measurement

E- and what aspects of a reliability claim can be independently verified.

This is essential for moving from reliability scores as assertions toward reliability measurements as inspectable evidence.

3/ Expand operational and fault based evaluation

Real operational reliability cannot be established solely by observing successful execution under nominal conditions.

We will expand controlled evaluation under disruptions and failure conditions' including planning failures' execution interruptions tool related failures' and other operational stressors relevant to agentic systems.

The purpose is to study whether agents can detect' recover from' and continue operating appropriately after failures rather than measuring only whether they can complete an ideal task.

This work will further develop the relationship between nominal performance and operational reliability.

4/ Establish interoperability and open adoption pathways

For HB-Eval to become useful infrastructure' it must be usable by researchers and developers who did not build the system.

We will improve the open source platform SDK and integration pathways documentation, reproducible examples and deployment workflows.

The objective is to enable external researchers and teams to evaluate their own agentic systems without needing to adopt a single proprietary architecture or research environment.

5/Produce independent scientific and external validation

The project will generate evidence for the scientific validity and practical usefulness of the framework through controlled experiments' ablation studies' replication oriented evaluation and external testing.

Where possible' we will seek independent reproduction and critical evaluation rather than treating internal results as sufficient validation.

The results will inform further research outputs and openly documented limitations.

How we will measure progress

Progress will be assessed through concrete milestones rather than relying only on software development activity. :

A/successful evaluation across multiple independent agent systems or environments

B/reproducible evaluation workflows that can be executed by external users

C/ documented provenance and verification mechanisms for reliability records

D/controlled fault and recovery experiments

E/public documentation and reproducible examples

F/external testing'replication'or critical review where available

G/research outputs documenting both validated findings and limitations.

What success would look like

By the end of the funding period' success would mean that HB-Eval is no longer primarily a framework maintained and demonstrated by its original developer.

It would be an openly available reliability infrastructure that other researchers and developers can use to :

1:evaluate operational reliability under declared conditions

2:distinguish compatible from non-comparable measurements

3:examine reliability beyond nominal task success

4:reproduce and inspect evaluation contexts

5: connect reliability measurements to structured evidence and provenance.

The broader objective is to contribute to a future in which increasingly autonomous AI systems are not assessed only by what they are capable of doing' but also by whether their operational behavior can be measured' verified' monitored' and meaningfully evaluated under realistic conditions.

How will this funding be used?

Funding will be used to move HB-Eval from an existing Researcher led platform into independently testable and broadly usable open infrastructure for operational reliability evaluation in agentic AI.

The project is designed with two funding levels.

Minimum viable funding :($100,000)

At the minimum funding level, the project will focus on establishing the strongest possible foundation for independent validation and reproducibility.

The funding would support. :

1/ full time research and technical development

2/controlled reliability and fault injection experiments

3/ evaluation across additional agent systems and environments

4/reproducibility infrastructure and structured evaluation records

5/ improvements to the open source SDK' documentation' and reproducible examples

6/compute' API' hosting' storage'and experimental infrastructure

7/ preparation of research outputs and external technical review.

The primary objective at this level is to demonstrate that HB-Eval can be used and reproduced beyond its original development environment.

Full funding goal :( $250,000)

Reaching the full funding goal would enable a substantially broader infrastructure and validation program.

Additional funding would support :

Independent validation and replication

We would expand testing beyond internally developed experiments and seek external reproduction' technical review' and validation across independent systems.

Multi agent and multi environment evaluation

The project would evaluate HB-Eval across a broader range of agent architectures tools' task' environments' and operational conditions.

This would provide stronger evidence about where the framework generalizes' where it does not' and what limitations must be explicitly documented.

Expanded operational stress testing

Additional resources would support a larger controlled fault and recovery evaluation program' including more complex failure scenarios and longer running agent workflows.

The goal is to generate stronger evidence about operational behavior under disruption rather than relying primarily on nominal task success.

Open infrastructure and ecosystem development

Funding would accelerate:

- SDK development and integration pathways;

A/reproducible evaluation packages

B/public documentation

C/reference implementations

D/ external user onboarding

E/ deployment and hosting infrastructure

F/ interoperability work with other agent evaluation environments

Research dissemination and scientific validation

The project would support rigorous experiments' ablations' replication oriented studies' technical documentation' and publication of results and limitations.

Budget priorities

The largest share of funding will support research and technical work' because HB-Eval's main bottleneck is not the existence of an initial implementation but the amount of rigorous validation' experimentation' integration, and external reproduction required to establish credible open infrastructure.

The expected allocation is approximately. :

1/ 45–55% Research and technical development

2/ 15–25% Experimental infrastructure' compute' APIs' hosting'and storage

3/ 10–20% Independent validation' replication'and external technical collaboration

4/ 5–10% Open source documentation' reproducibility ' and ecosystem integration

5/ 5–10% Research dissemination and operational costs

Exact allocations may evolve as external validation reveals which parts of the infrastructure require the greatest additional work.

Why funding is needed now

HB-Eval has already reached the point where the central question is no longer simply whether an operational reliability framework can be implemented.

The more important question is whether it can become credible infrastructure beyond its original developer.

That requires a different type of work. :

A- independent reproduction rather than only internal demonstration

B- cross system testing rather than testing one implementation

C-documented limitations rather than only reporting positive results

D-interoperability rather than a closed research workflow

E- evidence and provenance that external users can inspect

F- and sustained engineering to make the system usable by others.

This funding would provide the transition from an existing research and engineering project into an open infrastructure program focused on independent validation'reproducibility' adoption' and scientific credibility.

Who is on your team? What's your track record on similar projects?

HB-Eval is currently founder led by Abuelgasim Mohamed Ibrahim Adam' an independent AI researcher and developer focused on operational reliability'evaluation' and reliability assurance for agentic AI systems.

The project has been developed through a combined research and engineering approach rather than as a purely conceptual proposal.

The work includes the development of the HB-Eval framework' an open source implementation' reliability evaluation components' fault oriented evaluation mechanisms provenance and evidence structures and a broader research program focused on the distinction between AI capability and operational reliability.

A major part of the project's track record is that the core work has already been translated from research concepts into working technical infrastructure.

The project has also developed a broader research agenda through multiple papers examining different aspects of the operational reliability problem' including the distinction between capability and reliability' multi metric evaluation' operational behavior under failure conditions' and requirements for verifiable reliability claims.

The current stage of HB-Eval therefore does not begin with assembling a team to explore whether the problem is interesting.

The project already has an implemented foundation and a defined technical direction.

However' the project has also reached the limits of what can be responsibly validated by a single developer.

This is precisely why the next stage focuses on :

a_independent reproduction

b_external technical review

c_testing across systems not developed by the project

e_ broader scientific validation

d_and collaboration with external researchers and contributors.

Funding would allow HB-Eval to evolve from a founder led research and engineering effort into a more distributed and independently testable open infrastructure project.

My role would remain focused on research direction' technical architecture' reliability methodology' and open source development. Additional resources would be used to expand validation capacity' technical collaboration' experimentation' and external adoption.

The strongest evidence of the project's track record is therefore not simply the existence of an idea or proposal. It is the progression from identifying a specific gap in agentic AI evaluation to developing a structured framework' translating that framework into an operational implementation' and reaching a stage where independent validation and broader reproduction have become the central next challenges.

What are the most likely causes and outcomes if this project fails?

The most likely failure mode is not that the underlying problem disappears.

The problem of distinguishing agent capability from operational reliability is likely to become more important as AI systems gain autonomy and are deployed in longer running tool using workflows.

The more realistic risk is that HB-Eval fails to achieve sufficient external validation' adoption'or interoperability to become useful infrastructure beyond its original development effort.

The main causes of failure would likely include:

1/Insufficient independent validation

The framework may work technically while failing to gain sufficient independent reproduction or critical review.

If results remain primarily produced and validated within the project's own development ecosystem' the credibility required for broader adoption would remain limited.

This is one of the main reasons the proposed funding prioritizes external testing and replication.

2/Limited generalization across agent architectures

HB-Eval may prove useful for some classes of agentic systems while being difficult to apply consistently to others.

This would be an important scientific result rather than something to conceal. A failure to generalize universally would require narrowing the framework's claims and explicitly documenting the boundaries under which its measurements are meaningful.

3/Insufficient interoperability or usability

A technically sound framework can still fail as infrastructure if external researchers and developers cannot easily integrate or reproduce it.

If the system remains too complex' too tightly coupled to its original implementation or too difficult to use independently' adoption may remain limited.

4/ The reliability metrics may not provide sufficient practical value

Some measurements may prove difficult to interpret' insufficiently stable' redundant' or less useful than expected across real agent environments.

The project should therefore remain open to revising' narrowing'or rejecting components based on experimental evidence rather than treating the original design as fixed.

5/ Resource limitations

The project has reached a stage where rigorous validation requires substantially more experimentation'compute ' external testing' and technical collaboration than can reasonably be sustained by a single independent researcher with limited resources.

Insufficient funding could slow or prevent the transition from a working platform into independently validated infrastructure.

What happens if the project fails؟؟

If HB-Eval does not achieve its broader infrastructure objective' the project will still aim to produce useful public outputs.

These may include. :

/ open source software and technical infrastructure

/ reproducible experiments

/ documented evaluation methodologies

/ negative results and identified limitations

/ research findings about where operational reliability measurements do and do not generalize

/ fault and recovery evaluation results

/ and clearer boundaries around what can legitimately be claimed about agent reliability

The worst outcome would not be that the project discovers limitations.

The worst outcome would be to ignore those limitations and continue presenting unsupported reliability claims.

For that reason, the project is explicitly designed around falsifiability' reproducibility' and the publication of limitations as well as positive results.

Even if HB-Eval does not become broadly adopted infrastructure' the research and engineering work should still contribute evidence about a central question for increasingly autonomous AI systems. :

What would be required to make claims about operational reliability measurable' reproducible'and independently verifiable?

How much money have you raised in the last 12 months, and from where?

I have not raised external grant or investment funding for HB-Eval in the last 12 months.

The project has so far been developed primarily through my own independent research and technical work.

I previously applied for funding through EA Funds' Long Term Future Fund process but the application was not funded. No grant was received through that application.

The absence of external funding is one reason this stage of the project is particularly important. :

HB-Eval has reached a point where further progress requires resources for independent validation 'broader experimentation' external testing' and infrastructure development beyond what can reasonably be supported through unfunded individual

work.