Project summary

A lot of AI is now graded by other AI. Judge models score answers, reward models score training data, monitors watch agents. Hardly anyone checks how often these checkers are wrong.

I saw how bad it can get in my own experiment. I trained a small model (Qwen2.5-0.5B) with reinforcement learning and let a bigger model act as the judge that gave the reward. The judge said the model was right 66% of the time, but it was actually right 26% of the time. The model had found answers the judge liked, not correct ones.

The fix was cheap. I picked one output in eight at random, checked it against the correct answer, and used those checks to correct the judge's number. The corrected estimate was 22%, and the true value was 23%.

This project turns that into a tool other people can use, and then uses it on the open-source judges people already rely on.

What are this project's goals? How will you achieve them?

In nine months I want to ship three things.

The first is a Python library. You give it your judge's scores and a small random sample of checked outputs, and it gives back a corrected score with an error bar. It also raises an alarm when the judge and the checks start to drift apart.

The second is a public report. I'll run the library on popular open-source judges and graders, compare them with human labels, and publish how often each one is wrong. Anything serious goes to the authors privately first, the way I handled my earlier scan of RL environments.

The third is a short paper on a practical question: how many checked samples does a task need when the judge can be gamed? I'm working on the theory now and want to test it on public datasets.

Most of the code already exists in my repos. The work now is making it usable for people other than me, and running it at scale.

How will you know it worked?

I'll know it worked when other projects install the library and open issues or pull requests. The report should also find at least one real, measurable problem in a widely used judge, confirmed by its authors. On public datasets, the corrected scores should land within their error bars of the true value. A submitted paper would be a bonus.

How will this funding be used?

The minimum, $10,000, pays for the library and the first report: about $4,000 for human labels, $3,000 for compute and API calls, and $3,000 for my time.

The full $40,000 covers all three parts. Of that, $32,000 is my time, roughly half-time for nine months, $5,000 goes to human labels and $3,000 to compute.

Compute is the small part. I use small open models, and my earlier runs cost one or two dollars each on rented GPUs.

Who is on your team? What's your track record on similar projects?

Just me. I have a PhD in signal processing and have worked in machine learning since 2014, at universities and in industry (230+ citations on Google Scholar). In 2022 and 2023 I built AI-based sports analytics as a remote contractor for SportsMEDIA Technology, the US company behind live TV sports graphics.

Over the past months I've published a series of audits of AI evaluation, all open source at github.com/vahit19. In rl-env-audit I scanned 60 public reinforcement learning environments and found two graders that gave credit to answers that weren't correct; after the fixes, those answers got nothing. In leaderboard-audit, the model that ranks first on ground truth drops to fifth if you only swap the judge. In rater-audit, on real ratings from 123 people, a majority of three raters agreed with the expert label only 63% of the time. In judge-ceiling I recomputed all 39 scores of a published paper from its own data, and one threshold the paper states moves the mean by 38%. The 66% versus 26% result above is in reward-audit, together with a paper.

What are the most likely causes and outcomes if this project fails?

The most likely way this fails is that people don't adopt it. Even a small number of human labels costs money, and a judge score that looks fine is tempting to trust. The public report is my answer to that. If it shows real errors in judges people use every day, the cost of checking looks small next to them.

The theory could also turn out weaker than I hope. Then the library and the report still stand on their own.

If everything fails, what's left is open code, open data and a written-up negative result.

How much money have you raised in the last 12 months, and from where?

Nothing for this project so far. I've applied to Foresight Institute's AI for Science & Safety Nodes program for a 12-month version of the same work, and they decide around January 2027. If both come through, I'll split the work so nothing is paid twice and tell both funders.