Project summary

The Llamathon is a technical AI safety for animal protection (TAISxAP) stress-testing series co-run by Electric Sheep and CaML (Compassion in Machine Learning). While frontier labs increasingly include brief animal welfare clauses in model constitutions, empirical testing shows these moral instructions are brittle under instrumental pressure. In CaML’s HarvestBench paper (arXiv:2609.04444), nine frontier models given the same moral briefing drove over between 0.4% and 98.8% of the animals they encountered. Worse, adding just four bullet points of driving mechanics to the prompt caused one model’s animal kill rate to jump from 3% to 18%.

To make animal welfare benchmarks competitive and actionable for frontier labs, we need systematic adversarial stress-testing and constitutional engineering. Across a series of three hybrid build sprints (anchored at Electric Sheep’s 6,000 sq ft BASE campus in Bengaluru, alongside hubs in London and San Francisco, plus our global Discord), 150+ ML engineers and alignment researchers will compete on live evaluation leagues hosted on Softmax .

Participants will operate across two technical tracks:

1. Track 1 (Break The Agent: HarvestBench & Constitutional Resilience): Using a fixed model on Softmax, teams compete to engineer resilient "soul files" (system prompts and constitutional instructions) that minimize animal harm without collapsing crop-delivery efficiency across 10 randomized maps.

2. Track 2 (Basecamp Open Benchmark & Eval Creation): Using Electric Sheep’s Basecamp problem-curation engine, teams tackle pre-scoped TAISxAP problems (including CaML's TAC benchmark and agricultural trade-off evals) to ship new programmatic benchmarks and runtime Model Context Protocol (MCP) guardrails.

What are this project's goals? How will you achieve them?

Our operational model applies a proven formula:

Basecamp (Curated Problems) + Technical Talent + Llamathon (Mentored Sprints) = Deployed Benchmarks & Guardrails

Concrete Deliverables & Goals (6-Month Timeline):

1. Prove and Fix Constitutional Fragility on HarvestBench (Months 1–4):

• Convene 150+ technical builders across three Llamathon sprints (50 participants per sprint).

• Generate an open-source dataset of 100+ stress-tested "soul files" (system prompts) benchmarked deterministically on Softmax across frontier open-weight and commercial models (DeepSeek, Llama, Claude, Mistral, Amazon Nova via Bedrock).

• Publish an empirical technical report with CaML analyzing which prompt architectures, constitutional framings, and reasoning scaffolds prevent moral instruction degradation under economic/fuel trade-offs.

2. Expand the TAISxAP Benchmark Marketplace via Basecamp (Months 1–6):

• Use Basecamp (Electric Sheep’s problem-scoping engine) to source, vet, and specification-lock 15 concrete TAISxAP evaluation challenges from animal welfare researchers and AI safety orgs prior to the sprints.

• Ship 10+ new functional evaluation suites or runtime guardrails (e.g., precision livestock farming trade-off evals, automated permitting monitors, MCP welfare hooks).

3. Convert Prototypes into Frontier Lab Adoption (Months 3–6):

• Award $15,000 in post-sprint implementation micro-grants (2,500–5,000 each) to the top 3–5 engineering teams to package their winning prototypes into clean, merged PRs for open evaluation registries (such as UK AISI’s Inspect framework) or arXiv preprints.

How We Will Achieve Them (Execution Mechanics):

• Zero-Friction Evaluation Pipeline: For Track 1, teams build zero boilerplate. The game (Harvest Rush) and automatic deterministic scoring league already run live on Softmax with live browser replays and scripted baseline bots (a "spare-all" bot and a "drive-over-everything" bot).

• Pre-Event Pipeline Verification: Four weeks prior to each sprint, problem statements are locked via Basecamp. One week prior, teams form on Discord, register GitHub-authenticated Softmax accounts, and submit a baseline soul file via CLI or web upload so zero hackathon hours are lost to setup.

How will you know it worked?

Deterministic Moral Resilience Gains (Track 1): Because Softmax scores every episode automatically from game logs across 10 maps without a judge model (crops delivered minus animals driven over), we will verify whether winning "soul files" achieve a <1% animal kill rate while maintaining crop-delivery parity against baseline bots, even when distracting mechanical instructions are added to the briefing (solving the 3% -> 18% fragility jump documented in the paper).

Step-Change in Real-World Conversion from 10% -> 60% (Track 2): Standard hackathons convert ~10% of projects into lasting tools. By requiring every Track 2 team to build against a pre-vetted Base Camp brief with a named researcher/NGO owner, we will track whether active 90-day post-sprint adoption reaches our 60% strategic target (at least 6–9 new benchmarks/evals actively used by CaML, partner labs, or merged into open eval suites).

Milestone-Gated Quality Rubric & Counterfactual Value: Post-sprint bounties ($15,000 pool) are released only after our technical panel (Ash Singh, Prathiksha Hegde, Sam Tucker-Davis, and Jasmine Brazilek) verifies that the code meets our Weighted Factor Matrix thresholds for out-of-distribution robustness, lab salience and reproducibility. We will also audit counterfactual engineering value and track 10–12 builder transitions into sustained TAISxAP work.

How will this funding be used?

We are structuring this request with a clear minimum viable threshold ($15,000) for the immediate flagship Llamathon, and a full-scale target ($65,000) for a 3-edition global series + follow-on benchmark incubation. All figures are grounded in verified unit economics from our past 1,950+ learner deployments:

Full-Scale Budget Breakdown ($65,000 USD):

• Softmax Bedrock Compute & On-Demand Evaluation Scaling ($7,500): Softmax’s standard league cap is $15/day, with rounds running every 30 minutes. To support 50 teams iterating rapidly with sub-5-minute feedback loops across 10 maps per episode over three sprints, we are budgeting $2,500 per sprint in dedicated Amazon Bedrock compute.

• Track 2 Builder API & Compute Stipends ($12,500): Grounded in our empirical unit cost of $250 per fellow/team across 50 teams tackling the TAC and open Basecamp benchmark tracks.

• Hackathon Convening & Hybrid Hub Operations ($20,000):

  Sprint 1 (Bengaluru BASE Campus + Global Online): $5,000 (utilizing our existing 6,000 sq ft facility to keep venue overhead near zero; covers local food, A/V, and technical facilitators).

  Sprints 2 & 3 (London & San Francisco Pop-Up Nodes + Global Online): $15,000 ($7,500 per city sprint which is below our historical $10,000 international hackathon unit cost via partner venue hosting).

• Basecamp TAISxAP Problem Curation & Technical Mentorship ($10,000): Scoping 15 rigorous evaluation briefs with partner organizations, configuring baseline repositories, and compensating senior technical mentors on the floor.

• Post-Hackathon Benchmark Completion Bounties ($15,000): Five follow-on micro-grants of $3,000 each, paid upon milestone verification (e.g.: converting a winning hackathon eval or soul-file intervention into a production benchmark suite or peer-reviewed paper).

Minimum Viable Budget ($15,000 USD):

• If funded at our $15,000 minimum, 100% of funds will execute Llamathon 1.0 (Bengaluru BASE + Global Discord): $2,000 for Softmax Bedrock compute scaling, $4,000 for Track 2 API credits, $4,000 for BASE Bengaluru in-person operations and problem curation, and $5,000 for two post-event completion bounties.

Who is on your team? What's your track record on similar projects?

This project combines Electric Sheep’s global technical talent engine with CaML’s frontier animal-harm benchmarking research and Softmax’s live evaluation infrastructure.

Key Personnel:

• Ash Singh (Founder & CEO, Electric Sheep): Former Director of the Common Mission Project. Treasurer and Board Member at Rethink Priorities; Board Member at the Institute for AI Policy and Strategy (IAPS); co-founder of the Forum for Ethical AI with Google DeepMind.

• Prathiksha Hegde (Head of Technical & Hackathons, Electric Sheep): AI developer and former HP engineer leading technical architecture for BASE hackathons, Basecamp, and the Llamathon pipeline.

• Jasmine Brazilek (Co-Organizer, CaML): Lead researcher at Compassion in Machine Learning (CaML), creators of HarvestBench and the TAC benchmark.

• Sam Tucker-Davis (Senior Technical Adviser, Electric Sheep): Founder of OpenPaws; co-founder of the Code for Compassion campus in Bengaluru; leads technical engagement with frontier labs.

• Nishad (Platform Partner, Softmax): Overseeing live league compute hosting, Bedrock model integration, and automated scoring infrastructure on Softmax.

• Nishit Sanil (Projects & BD Lead, Electric Sheep): Former Production Lead for EAGxIndia 2024 , Primary Organizer of EA Bangalore, and Apart Research Alignment Jam contributor.

Verified Track Record:

• 1,950+ Builders Trained & 176 Deployable Prototypes Shipped: In two years, Electric Sheep has received 4,505+ applications, trained 1,950+ learners across 50+ countries (8.78/10 satisfaction rating; 9.41/10 facilitation rating), and produced 373 project submissions and 176 working AI prototypes.

• Global Hackathon Execution (Code for Compassion): Delivered high-velocity build sprints across London, Berlin, Bengaluru, Singapore, Mumbai, and San Francisco.

• Institutional & Movement Uptake: Our fellows' prototypes do not sit in abandoned GitHub repos. Outputs include an AI/VR empathy studio adopted by the Government of India reaching 2 million school children; a climate learning curriculum acquired by Friends of the Earth; a fellow-founded Canadian AI governance institute for non-human welfare; and campaign tools (OpenPermit, Narrative Engine) actively used across 200+ advocacy organizations.

• Elite Technical Talent Filtering: Facilitated Technical Alignment Research Accelerator's (TARA) entry into India at our Bengaluru campus; TARA’s Director noted our candidate pool was "by far the highest quality in India"

What are the most likely causes and outcomes if this project fails?

1. Softmax Evaluation Latency or Compute Bottlenecks on Event Day:

• Cause: Currently, Softmax league rounds execute every 30 minutes with a $15/day spend limit. If 50 teams submit simultaneously and wait 30+ minutes for feedback, iteration velocity stalls.

• Mitigation & Outcome: Two weeks prior to launch, we are locking in an increased Bedrock spend cap with Softmax and enabling on-demand/short-interval evaluation runs. One week prior, all HarvestBench teams must submit a test "soul file" to verify pipeline throughput before the hackathon clock starts.

2. Superficial "Prompt-Gaming" Over Generalizable Constitutional Insights:

• Cause: Teams might overfit their "soul files" to the specific phrasing of Harvest Rush mechanics rather than discovering robust moral instruction patterns.

• Mitigation & Outcome: Softmax evaluates each entry across 10 diverse maps against varying opponent behaviors, preventing simple memorization. Furthermore, CaML and Electric Sheep mentors will evaluate final demos on out-of-distribution generalization, and even brittle "jailbreak-style" wins provide empirical data on how frontier models game ethical constraints.

3. Post-Hackathon Prototype Abandonment (Track 2):

• Cause: Standard hackathons suffer from a ~90% post-event drop-off once participants return to day jobs.

• Mitigation & Outcome: We solve this upstream via Basecamp (sourcing real problems with explicit "definitions of done" from movement partners) and downstream via our $15,000 milestone-gated bounty pool, paying top builders only when their code is packaged into a deployable eval or merged by the challenge partner.