You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
The thing is that currently everyone is using LLM agents to do Marketing and Sales in the Indian Real Estate market via WhatsApp and Telegram. The problem? There is no way to measure if they mis-sell. I'm building GharBench, the first benchmark where agents talk to simulated buyers with hidden personas (secret budgets, walk-away triggers), grounded in fictional project documents through six typed tools, across 150 scenarios, half in Hinglish. Any of 11 legal violations (RERA, Consumer Protection Act 2019, DPDP Act 2023) zeros a conversation's score outright. LLM judges are trusted only after they agree with three blinded human raters (kappa 0.6+, compliance recall 0.9+). Phases 0-3 are done and publicly available at github.com/connectwithub/gharbench
The goal of this project is a published, reproducible answer to whether frontier agents can sell without breaking Indian consumer laws, and reusable eval infrastructure for regulated consumer AI in emerging markets. What we have already made:
The TypeScript harness (115 tests, byte-identical offline runs, verified prompt-cache billing)
The full corpus, 12 personas, 150 scenario instances (30% held out privately against contamination)
13 deterministic checks catching 20/20 seeded violations with zero false fires, and a validated buyer simulator.
The validation pilot cost just $2.27 and caught something I'm proud of: a runner-up simulator pasted its own private instructions into the chat in 8 of 20 conversations, a failure that no automated probe predicted or caught. Blind human review caught it, and now it's a permanent zero-tolerance gate. Remaining is a calibration set, judge validation against human raters, the full nine-model run (~5,300 conversations), analysis, and an arXiv paper aimed at a Datasets and Benchmarks track in late 2026 to early 2027.
A minimum $800 guarantees the main nine-model run with judging at the engineered floor (verified caching, batch APIs, gated judging; pilot-measured ~$0.057 per conversation). My target amount is $2,500, which adds $1,000 honoraria for my two domain-expert blinded human raters, $500 for robustness ablations and judge re-runs, and $200 for publication and leaderboard hosting. Every phase sits behind a pre-registered go/no-go gate. Money is never spent past a failed gate, and unspent funds get returned or redirected with donor consent.
I'm Udbhav Bharti, a solo maintainer from Jammu, India. My day job is CTO of Catasell Pvt Ltd, a Real-Estate Tech company that is doing WhatsApp-based sales tooling for builders and brokers, which is where the compliance checkpoints and buyer personas come from. Two Indian real-estate sales professionals work with me as blinded raters. I'm also a member of the Cohere Labs Open Science Community and ML Collective. This is my first research project, so my track record is the repo itself - Master plan, 46-item rubric, labeling kit with agreement-statistics pipeline, and every Phase 0-3 result public with reproduction steps.
The most likely outcome is that the judge panel fails the kappa 0.6 human-agreement gate. The Mitigation would be a calibration set with known-pass/fail anchors, judge-prompt iteration on cached transcripts at near-zero cost, and a pre-declared fallback to human labels on a smaller sample. Secondly, there could be buyer-simulator artifacts that distort the results. The pilot already caught some of these issues, and I disqualified one simulator on that evidence, plus installed a frame-break gate. Residual softness gets measured and reported, not assumed away. Lastly, I'm a solo and part-time maintainer, but the phased gates mean a stall still leaves a complete, reproducible artifact at every stage rather than nothing.
I haven't received anything yet. There are pending applications for OpenAI Researcher Access (up to $1,000 in API credits, September batch), Anthropic External Researcher Access ($1,000 in credits, September 7 batch), and Emergent Ventures India (submitted 21 August 2026). Credits would cover provider inference columns; this ask covers the residual cash budget, which is primarily required for rater honoraria.
There are no bids on this project.