You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$25,000 left to reach the funding goal.
PROBE builds a local pipeline of AI evaluators who can design, run, and critique safety-relevant evaluations of frontier and open-weight AI systems. We plan to start in Asia/greater China region, where capable open-weight models ship fastest and there is a strong need for evaluators Each PROBE workshop pairs hands-on evaluation design talk with an interactive audit game, which feeds an open-source practitioner handbook. We are requesting $23,000 for a pilot of two workshops in Asia (December 2026 – Q2 2027), supporting the learning of 40–70 participants looking to upskill evaluation.
Why it matters for safety: evals are how labs, auditors and governments see dangerous behavior before deployment, and evals are getting harder to build well.
Why China/Asia first: China is a major producer of open-weight models with a stated government mandate for AI safety, but the AI safety ecosystem is relatively small. This holds true across greater Asia, where the AI safety hubs are still in development stage.
Why independent workshop : There is a lack of support for independent, safety-focused evaluator commons. That is the gap PROBE fills.
Safe deployment of advanced AI is bottlenecked by lack of independent evaluation talent who can turn standards into rigorous, safety-aligned evaluations. Best practices already exist in the US, UK/EU, and China. However, adoption remains limited because implementation is nuanced and the talent is scarce.
Evals are getting harder. As capabilities grow, especially in agentic systems, measuring risk needs more care in design and oversight. Frontier evaluators such as METR and Apollo Research increasingly point to the limits of current measurement.
The evaluator pipeline is concentrated in the EU and the US. There are relatively fewer AI safety aligned hubs in Asia that can support evaluation activities and information exchange. China is shipping increasingly capable open weight models that need corresponding development of “human in the loop” talents for evaluation.
Automation is not a substitute. Delegating evaluation and oversight to LLM-based monitors and classifiers needed a stronger human in the loop approach since LLM based judges have known failure modes. Human judgement and evaluators remains essential at high-stakes checkpoints, which requires skilled and trained evaluators.
Each PROBE session takes 20–40 ML engineers, researchers and risk professionals from reading about evals to designing and critiquing one. The format was piloted at PROBE Berlin in October 2026.
Evaluation design jam: small teams learn about the latest eval tools and research. Participants have the option of scoping a safety-relevant eval for a real deployment case, covering threat model, task design, and human-in-the-loop checkpoints.
Interactive audit game: a paper-based Grader-Failure / Classifier Audit exercise. Participants inspect model transcripts, find where automated classifiers fail, and decide where a human must stay in the loop. Participants also have the option to comment and review evaluation design
Facilitation kit: game instructions, eval templates and a facilitation guide so future partners can run similar PROBE independently.
Workshops delivered in Asia
2 (minimum)
Participants reached
40–70
Open source Resources
1 audit game kit and handbook from evaluation materials reviewed
Outcome
Gain in participant eval design skill and awareness of tools and techniques
Participants applying eval design or safeguards at work (6-month follow-up): over 75%
PROBE Shenzhen (Dec 3, 2026): $7,500
Primary anchor event: venue, speaker and organizer travel & honoraria compensation, game printing, and documentation
PROBE Iteration 2 (2027) $6,500
Second city deployment (venue, speaker travel, compute, local logistics)
Series Infrastructure $3,500
Support for community contributors for open Facilitation Playbook, interactive game kits, evaluator network support
Working-Group Seed & Content Curation Fund $5,500
Support for community contributors for capturing, structuring, and curating expert workshop inputs for workshop and handbook integration
Total Funding Request. $23,000
Min-Si Wang (Program Lead & Coordination): project strategy, overall event execution, and workshop curriculum development. Background: Product and research lead in fintech and risk analytics. Main curator of www.aisafetynode.com for cross border AI safety development and field building. https://www.linkedin.com/in/min-si-wang-867395a/
Jaspreet Singh (Program Advisor): Role: Design workshop format, evaluation methodology, workshop activity structure, and participant experience. Background: experienced product and design lead experience. https://www.jaspreetsingh.net/#projects
Bryan Chan (Developer & AI Safety Specialist): Role: Design and development of the interactive audit game kit, evaluation design jam exercises, and theory of change alignment. Background: AI Safety researcher with background in evaluation(https://www.linkedin.com/in/cnm13ryan/)
Downscaled reach for Evaluation Training: We will reach less people with limits to scaling the workshop. The breadth and quality of the workshop will also be less comprehensive in scope in terms of the interactive activities and followup resources.
We have not raised funding so far.
There are no bids on this project.