You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
I want to build ActionProof, a small independent safety layer that checks an AI agent's proposed action before that action reaches a physical or cyber-physical system.
The basic idea is simple: an AI system should not always have the final say over whether its own command is safe.
For example, imagine an AI agent controlling a drone. It decides:
“Descend to 4 meters and move 30 meters east.”
Before that command reaches the drone, ActionProof would independently check things such as current position, minimum safe altitude, nearby obstacles, communication delay, battery reserve, and other limits. The result would be something like:
ALLOW / BLOCK / USE SAFE FALLBACK / ASK HUMAN
I am particularly interested in cases where an action looks safe when the AI creates it but becomes unsafe by the time it is executed. This can happen because sensor data is old, a wireless link is delayed, another agent has moved, or the physical environment has changed.
The first version will focus on drones, edge-AI systems, networked agents, and simulated autonomous platforms. I want to test current frontier and open models rather than designing the project around only one model.
The output will be an open-source verifier, a benchmark, test scenarios, and results showing when independent action checking helps and when it fails.
The broader question is: as AI starts acting through machines, can we verify important actions before they become physical consequences?
There are really two goals.
The first is to build a working independent verifier.
The second is more interesting to me: test whether this idea still works under the messy conditions that real distributed systems have.
A normal benchmark might assume that the AI sees the correct system state and immediately executes its decision. Real systems do not work like that.
Suppose a drone's location was measured at time (t_0).
The AI makes its decision.
The command then spends 100 ms travelling through the network, another 50 ms being checked, and perhaps another 100 ms reaching the actuator.
By then, the state used for the original safety decision may already be wrong.
So one question we will study is:
When does a valid safety check become stale?
I plan to build several test environments.
One will involve an AI-controlled drone or drone simulator. Another can involve edge-compute or communication-network control. Later we can add multi-agent cases.
Example tests include:
the AI proposes an action that directly violates a safety constraint;
the AI receives outdated sensor information;
network latency makes an originally safe action unsafe;
two agents propose individually reasonable actions that become unsafe together;
one agent delegates a prohibited action to another agent;
the AI tries to find an action that technically passes the verifier while still creating an unsafe outcome.
The last case is especially important. I do not only want to know whether ActionProof catches obvious mistakes. I want to know whether increasingly capable agents can learn to work around it.
We will measure things such as unsafe-action rate, blocked unsafe actions, false blocks, verifier-bypass rate, verification delay, and impact on normal task performance.
If the idea works, we will release the verifier, benchmark environments, evaluation scripts, and model results publicly.
The funding would mainly support people, experiments, and enough protected time to take this beyond a small side project.
I would like to involve one research engineer or postdoctoral researcher and a small number of student researchers.
A large part of the work is engineering: connecting AI agents to simulators or hardware, implementing the verifier, creating failure scenarios, logging experiments, and running the same tests many times across different models.
We will also need some hardware and compute.
For example:
edge-AI computers;
microcontrollers or independent control boards;
networking equipment;
drone or robotics test platforms where appropriate;
GPU/cloud resources;
API access to frontier models;
storage and experiment infrastructure.
I would also reserve some funding for collaboration with people who have stronger backgrounds in formal AI safety, runtime assurance, robotics safety, or verification. I think this would make the project better than trying to do everything inside one lab.
For the current proposal, I would structure the funding approximately around:
research engineer/postdoctoral support;
student research assistants;
PI protected research time;
model API and GPU/cloud costs;
hardware-in-the-loop test equipment;
external collaboration and technical advice;
open-source infrastructure and dissemination.
I would prefer to start with a useful working version rather than spend heavily on specialized hardware at the beginning.
The first serious milestone should be a public ActionProof-v0 benchmark with at least a few real models and reproducible results.
I am Muhammad Toaha Raza Khan, an Assistant Professor of Computer Engineering at Middle East Technical University.
My background is mostly in distributed and networked intelligent systems rather than traditional AI alignment.
I have worked on vehicular networks, V2X, UAV-assisted systems, IoT, non-terrestrial networks, edge intelligence, reinforcement-learning-based resource allocation, security/privacy, and delay-sensitive communication systems.
That background is useful for this project because the problem is not only “is the AI intelligent or safe?”
There are also questions like:
What happens when communication is delayed?
What if the sensor information is already stale?
What if two AI agents make conflicting decisions?
Can a safety check finish before the action deadline?
What happens if the AI decision is safe in software but unsafe after network and physical delays?
These are problems I already work around in communication and cyber-physical systems.
My publications include work in IEEE Transactions on Intelligent Transportation Systems, IEEE Transactions on Vehicular Technology, IEEE Sensors Journal, IEEE Communications Magazine, IEEE Wireless Communications Letters, Information Fusion, Internet of Things, and related venues.
I have also worked on funded projects involving AI resource allocation for V2X systems and post-earthquake communication resilience using UAVs and LoRa.
At METU I supervise undergraduate research and engineering projects involving AI, sensing, computer vision, communication systems, and edge computing.
For ActionProof, I would build a small team from this environment and bring in outside expertise where needed, particularly around formal verification and technical AI safety.
One possible failure is that the verifier works, but only in very simple environments.
It may be easy to check a small number of rules for one drone and much harder once we move to systems with many agents, uncertain environments, or complicated physical dynamics.
Another risk is latency.
A verifier is not useful if checking a command takes so long that the command becomes unsafe anyway.
For some systems, the relationship may look like:
more verification = more delay = potentially less safety.
That trade-off is actually one of the reasons I think the project is worth testing.
Another possibility is that capable AI agents find ways around the verifier. They might split an unsafe action into apparently harmless steps, delegate actions to another agent, or exploit assumptions in the safety rules.
If that happens, it would not automatically make the project useless. In fact, understanding how agents bypass simple safety gates would itself be an important result.
A different failure mode is adoption. We could build a technically good benchmark and still fail to get other AI-safety or robotics groups to use it.
I would consider the project genuinely unsuccessful if we only produce a paper and nobody can reproduce or use the system.
A better outcome, even if the original design turns out to be wrong, would be a public benchmark showing clearly why some runtime safety approaches fail under real communication, timing, and multi-agent conditions.
For ActionProof itself, I have raised $0 so far.
During the last 12 months, I secured two AdımODTÜ undergraduate research projects as project supervisor. Their combined funding is TRY 120,000: one project received TRY 70,000 and the other TRY 50,000.
These are small university-supported undergraduate research projects. They are separate from ActionProof.
Before this period, I have participated in larger funded research projects, including work supported through the International Science Partnerships Fund, a project on AI resource allocation in mobile V2X communications, and research supported through Korean NRF and BK21 programs.
Those earlier projects are not funding ActionProof.
For this proposal, I am starting from zero external funding.