Training Small Language Models to Write Provably Correct Code Using Deterministic Compiler Rewards and Group Relative Policy Optimization
We’re building an open-source RLVR (Reinforcement Learning with Verifiable Rewards) harness specifically for training small language models to generate formally verified code. The goal isn't to build another chatbot; it's to create a deterministic training pipeline where the reward signal is a passing compiler or theorem prover, not human preference. We want to prove that a 7B parameter model can reliably write provably correct Lean4 and Dafny code if trained with strict, rule-based feedback instead of subjective RLHF.
We’ll achieve this by wiring up a GRPO training loop directly to ephemeral sandbox environments. When the model generates a proof candidate, we spin up an isolated container, run the formal verifier, and return a binary 1.0/0.0 reward based solely on compilation success. No neural reward models, no hallucinated grading. We’re starting with a cold-start SFT phase on Qwen2.5-7B to teach the output format, then letting the RLVR loop optimize for actual correctness.
Three concrete metrics:
Pass 1 on held-out formal verification benchmarks increases by at least 15% over the base model after RLVR training.
Zero reward hacking incidents in our adversarial test suite (we’ve built specific tests for format spam, hardcoded answers, and sandbox escape attempts).
The harness itself is reproducible. Another team can clone the repo, point it at their own dataset, and get identical training dynamics within 5% variance. If we can’t replicate our own results, it didn’t work.
We’ll also track proxy metrics during training: KL divergence stability, group reward entropy, and parse success rate. If all rewards in a group collapse to zero or one for more than 20% of batches, we know the curriculum is broken.
$25k – Compute: Cloud GPU time for RLVR training runs and sandbox orchestration. GRPO with group size 16 on 7B models eats VRAM fast. We’re budgeting for ~3 months of sustained A100/H100 usage plus Modal/SandboxFusion credits for ephemeral verifiers.
$15k – Dataset Curation & Verification Infrastructure: Paying domain experts to audit our seed dataset of formal proofs and build out the adversarial test suite. Also covers licensing for any proprietary benchmark subsets we can’t redistribute.
$10k – Engineering & Ops: Contract support for sandbox hardening, CI/CD for the harness, and documentation. I’ll be doing the core RL engineering myself, but we need help making this production-grade for open-source release.
$5k – Contingency & Dissemination: Conference travel, open-source hosting, and buffer for compute overruns. RL training always has surprises.
Cause 1: Small models lack latent capacity for formal reasoning. A 7B model might just not have enough pretrained knowledge to bootstrap formal proofs, even with perfect RLVR. Outcome: We pivot to distillation-first (using DeepSeek-R1-Distill as base) and document the capability threshold. The harness still works; we just adjust the target model size. This is still a valid research contribution.
Cause 2: Sandbox overhead kills training throughput. Spinning up containers per rollout might be too slow for stable GRPO updates. Outcome: We switch to persistent sandbox pools with aggressive caching, or move verification to in-process subprocesses with strict seccomp filters. Worst case, we reduce group size and accept slower convergence. The harness architecture is designed to be modular here.
Cause 3: Reward signal is too sparse. If >80% of rollouts get zero reward consistently, GRPO has no gradient. Outcome: We implement curriculum learning with easier problems first, add partial credit for syntactically valid but incorrect proofs, or increase temperature during early training. We’ve already built monitoring for this exact failure mode.
Worst-case outcome: We don’t beat baseline Pass@1. But we still ship a battle-tested, open-source RLVR harness for formal verification that others can build on. The infrastructure has value independent of the model performance.
$0 from external sources. This project has been self-funded and bootstrapped so far. We’ve covered initial compute costs (~$3k out of pocket) and dataset curation through volunteer effort. The MVP was built on personal hardware and free-tier cloud credits.
This grant would be our first external funding. We deliberately waited until we had a working prototype before asking for money—we wanted to prove the concept wasn’t vaporware before taking anyone’s capital.
Comments
No comments yet. Sign in to create one!