A six-week study of wrong answers accepted and correct answers rejected, with independent review and public results.
I’m building Plumbline Grader to check whether AI grading systems reward wrong answers or reject correct ones. We have software and research prototypes, and a lot of work that needs independent checking. I want to keep our commercial pilot moving while doing a public study that other people can use.
I’m asking for $20,000 for a six-week study. We would compare original and repaired graders on a small, defined set of coding tasks. Independent reviewers would check which answers are actually correct, and a separate person would resolve disagreements and rerun the evidence. We would publish the methods, task pack and results, including failures or a finding that the repairs do not help.
What are this project's goals? How will you achieve them?
The question is simple: can we reduce the wrong answers a grader accepts without causing it to reject more correct answers?
We plan to use 12 deterministic, function-level coding task families. Two would be used to calibrate the process. The remaining 10 would each have 10 valid and 10 invalid held-out cases, for 200 confirmatory cases. We would fix the task manifests, correctness rules, repair budget, analysis and exclusion rules before the confirmatory run. The held-out cases would not be used to tune repairs.
Two independent reviewers would judge correctness. A distinct adjudicator and rerunner would handle disagreements and reproduction. We would compare the original and repaired grader on the same cases and report results by task family. Ambiguous cases, missing results, crashes and timeouts would be recorded rather than quietly removed or counted as successful grading.
The study has not started. Reviewers have not been contracted. The six-week timetable begins after funding, reviewer availability, rights and safe execution arrangements are settled, and only if calibration supports the proposed design.
How will you know it worked?
We will report both error directions: wrong answers accepted and correct answers rejected. We will show the paired changes for each family, the uncertainty and the disagreements. A useful result could be an improvement, no improvement, or evidence that a repair causes harm.
The public output would include a rights-cleared regression pack, correctness rubric, evidence manifests, analysis scripts and a report that another team can inspect and rerun. This is a feasibility study. It cannot establish rare-error rates, how common defects are across the industry, general superiority over other products, or certification readiness. It is not a test of our memorization detection system.
How will this funding be used?
The planning budget is $17,100 plus up to $2,900 in contingency, for a total cap of $20,000:
• Two independent correctness reviewers: 32 hours each at $100/hour, $6,400.
• Task preparation, baseline runs and fixed-budget repair work: 40 hours at $100/hour, $4,000.
• Separate adjudication and rerunning: 12 hours at $125/hour, $1,500.
• Statistical analysis: 8 hours at $150/hour, $1,200.
• Rights and terms review: 4 hours at $400/hour, $1,600.
• Public reproducibility materials: 12 hours at $100/hour, $1,200.
• Coordination: 8 hours at $75/hour, $600.
• Compute: $600.
• Contingency: up to $2,900, used only if needed within the agreed scope.
These are planning allowances, not contracted quotes. Taxes, fees and transfer costs must fit within the cap. The minimum and maximum request are both $20,000 because we have not established a smaller funded design that can deliver this full study.
This request is for public research, not founder salary, advertising, paid-client work or private engine development. Plumbline may benefit from what we learn, but the public outputs must be useful on their own. Existing proprietary code and customer data are excluded. We will settle the public-output licenses and background-IP boundaries before work or award acceptance. We would not accept study funding from owners of the systems being rated.
Who is on your team? What's your track record on similar projects?
I’m Shane Cope, founder and CEO of Plumbline Grader, Inc. Our existing work includes audit and repair software, public and synthetic task studies, and reports to benchmark maintainers. Those projects give us material and tools to build on. They do not substitute for this study’s independent review.
AI agents do much of our software development. This grant would pay for the separately scoped study work and independent checking that our internal build process does not provide. Reviewer and adjudicator roles are planned, not filled, and no outside person is represented here as having agreed to participate.
What are the most likely causes and outcomes if this project fails?
We may find that correctness is too ambiguous, that the review budget is insufficient, that repairs do not generalize to held-out cases, or that they trade fewer wrong accepts for more wrong rejects. Reviewer recruitment, rights clearance or execution problems could also delay the work.
Calibration is an early stop point. If the full design is not feasible, we would disclose that and agree any changed scope with the funder before proceeding. We would publish what was learned and the limitations rather than present a smaller or inconclusive study as validation of the whole business.
How much money have you raised in the last 12 months, and from where?
We have received no funding.
A $20,000 BlueDot Rapid Grant application for this same study was submitted on November 16, 2026 and is pending. A separate $117,700 individual application to the Transformative AI Fund, submitted on October 20, 2026, covers a broader January–December 2027 public program and has potentially overlapping review, methods and repair work. A Lightcone draft for this study is saved but not submitted. These are applications, not awards.
This Manifund request is an alternative or co-funding route, not an additional $20,000 to spend twice on the same work. Before accepting funds, we would disclose other awards, reconcile scope and cost allocations, and keep invoices and time records so no expense is reimbursed twice. The TAIF application is individual funding, not company funding.
This proposal was prepared with AI assistance and should be assessed on the specific plan and evidence above.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!