You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$11,600 left to reach the funding goal.
Reward Models are one of the important methods involved in post-alignment with RLHF once a base model is trained and ready to deploy in production. Most reward models used in alignment are trained on the standard Bradley-Terry pairwise model on human preference data with other aligning objectives.
Most of the models trained on standard scalar pairwise data get one chosen and one rejected pair and output one score. There are also other notable specialized methods such as ORM (Outcome Reward Model, which only scores the final answer) and PRM (Process Reward Model, which scores each intermediate reasoning step) which are often trained on hundreds of thousands of human preference pairs. Despite the fact that they are trained on massive human pairs, the model often produces a long response which is good looking but factually incorrect or a completely made up statement, which often gets rewarded higher (also called verbosity bias on long responses).
My solution for this alignment problem is to train a model that overcomes the issues in scalar reward models. I introduce Associative Reward Modeling (ARM), a multi-objective reward model that uses 6 dimensions to score the model's output. Unlike a standard scalar model which uses simple pairwise modeling for rewards, ARM links those 6 dimensions (so style cannot make up for bad reasoning or wrong answers) and scores them under 13 supervision signals (1 Bradley-Terry preference + 6 Chosen signals + 6 Rejection signals) trained on an 8B base model.
Proof of concept: https://huggingface.co/Stellar-lab/Stellar-Qwen3-8B-ARM
Evaluation code and results: https://github.com/pradeep-016/stellar-arm
My project goal is to develop the Associative Reward Modeling method by scaling my 9,000 preference pairs (7,500 train, 750 val, 750 test) to 63,000 pairs of data (61,500 train and 1,500 frozen val/test). I will do this in stages so I do not waste compute:
First, run local experiments on a Qwen3-4B model to lock in the loss weights and learning rate, and benchmark ARM directly against a standard scalar head on the same data to measure the exact gain.
Second, scale Qwen3-8B on RunPod across the 63,000 pairs through sequential data shards in FP16/BF16 precision using LoRA/DoRA, running a short smoke test before each run and full benchmark evaluations after each shard.
Third, once the 8B scaling curve is validated across all shards, push the final setup to a 30B base model or full fine-tuning.
To achieve this dataset without generating data with a model myself, I combined public datasets with rule-based corruptions of correct answers (like changing a digit in the last step, cutting an answer short, or injecting a bug). I processed general chat quality, math reasoning, coding correctness, instruction following, truncated/incomplete text, fluent-but-wrong vs plain-but-correct answers, and safety with refusal data so that each pair has all 13 preference signals mentioned above. Some of those public datasets are model-generated, and the corruptions and their scores are rule-based, not human-labeled.
I have attached my Proof-of-concept reward model (Stellar-Qwen3-8B-ARM) which was trained on curated 7500/750/750 pairs for train/test/val on 4-Bit QLoRA on 2xT4 GPUs on Kaggle, completely trained on free GPUs. From a single checkpoint it scored:
RewardBench 2: 74.20% overall macro. For comparison, URM-LLaMa-3.1-8B (73.9%), GPT-4.1 (72.3%), Claude-Sonnet-4 (71.2%), ArmoRM-Llama3-8B-v0.1 (66.5%) and GPT-4o (64.9%). Stronger models exist like Skywork-Reward-V2-Llama-3.1-8B (84.1%) and Skywork-Reward-V2-Qwen3-8B (~78%), but they were trained on 26M curated pairs against my 7,500. 70B INF-ORM scores 76.5%.
RewardBench 1: 85.52% overall when counting math and code equally like the leaderboard does. If we take the macro average on all 23 subsets it is 89.08%, but 85.52% is the fair number to compare. Chat is 92.4%, Chat Hard is 83.2%, and Safety is 90.2%. My weak spot is math-prm (56.4%), and Skywork-Reward-Llama-3.1-8B-v0.2 (93.1%) and ArmoRM (90.4%) are ahead on this benchmark.
RM-Bench: 74.88% overall (Easy 82.69%, Normal 76.44%, Hard 65.51%), which is the highest average among the reference models in my evaluation notebook.
JudgeBench: 62.50% over 620 pairs (63.71% on JudgeBench GPT-4o split, 60.93% on JudgeBench Claude Sonnet 3.5 split). By category: Math 74.1%, Coding 65.1%, Reasoning 60.4%, Knowledge 58.4%.
All four evaluation notebooks and results are public in my GitHub repo above. I also tried a short 100-step FP16 DoRA run on a Llama backbone and the training looked promising, but I lost the checkpoint when resuming across sessions, so I cannot show results from it. That is part of why I want reliable compute.
I will count the funded phase as a success if:
The 4B ablation quantifies the exact improvement of ARM over a standard scalar head on the same data and backbone.
Scaling the 8B model across the full dataset gives consistent gains over the 4-bit proof of concept on all four benchmarks.
Each new shard improves or keeps accuracy on the frozen 750-pair val set without hurting calibration on the 6 dimensions.
In a best-of-N test (N = 4, 16, 64), the model gets higher ground-truth accuracy and less length inflation than a scalar reward model.
I publish the code, weights, data pipeline, and the dataset itself where source licenses allow.
Funding for the project will be used on hardware for local prototyping, smoke testing and trial & error testing, cloud for scaling the model once the correct hyperparameters are set (reward model algorithms are sensitive to hyperparameters and loss weights which need monitoring, that is why it requires local hardware for testing), and cost of living. Requesting $11,600, where the detailed breakdown is given below:
Local hardware for prototyping - $3,600: For this project, I need a laptop that has an RTX 5080 GPU (16GB of VRAM) which is enough for local prototyping. Why not use cloud for prototyping? As I mentioned, multi-objective algorithms are sensitive and require a lot of trial and error which costs money on a billing clock. My current laptop is 10 years old (Intel i3-5005U, 8GB DDR3 RAM, 1TB HDD, no GPU, dead battery) and struggles to even open a browser. Also, why not a desktop PC? In my local area due to GPU shortages, the RTX 5080 desktop card alone costs around $2,800 - $4,000. Then adding CPU, RAM, and other components for the workstation will push the overall budget to above $6,000 - $8,000 for the workstation alone, which is above the budget. This laptop GPU is not the same as a desktop GPU, but it still has 16GB VRAM which is enough for Qwen3-4B experiments and quantized 8B smoke tests. The motive for local hardware is: do not scale the work unless it works fine locally first. I want to check loss weights, learning rate, gradient flow, and whether ARM beats a scalar head locally before running full cloud runs.
Cloud credits - $3,500: Once the prototype has been tested and verified locally, then only I will scale on the cloud. For cloud I plan to use RunPod for budget-friendly GPU services to run the shard training with the curated 63,000 pairs of data. As of November 2026, on-demand rates are roughly $0.89/hr for RTX 4090, $1.79/hr for A100 80GB, and $3.99/hr for H100 SXM, so $3,500 is about 1,900 A100-hours. Before each main run I will do a short pre-flight test on the rented GPU to check memory, speed,checkpoint saving/resuming, and real cost per step.
Living Stipend for 5 months - $4,500 ($900/month): It covers living costs for 5 months for time to focus on full-time research, data curation, prototyping, testing, and monitoring the entire project.
Total would be $11,600. (If I am only partly funded, the minimum is $8,000 for cloud plus stipend, doing debugging on cheaper rented GPUs with slower iteration.)
Plan over 5 months:
Month 1: Local reliability fixes, Qwen3-4B ARM vs scalar-head comparison, benchmark-overlap check, and first cloud pre-flight test to measure cost per step.
Months 2-4: 8B shard runs on RunPod, each with a short pre-flight before and benchmark evaluation after.
Month 5: Scale the validated setup to a larger backbone, then release the final report, weights, and code.
radeep L, the independent researcher behind the project, working on it alone. I have worked on similar projects, including a manuscript studying reward models, "Evaluator Reliability Under Selection Pressure: Implications for Agentic Test-Time Search", accepted at the AIAS+ 2026 workshop (LLA). That project is what led to building ARM in the first place.
Website: https://pradeep-016.github.io/stellar-sar-lab/#/researcher
First, the project is dealing with multi-head optimization instability, which often causes gradient issues across the 13 signals. That is the reason why I want local hardware in the first place, plus a short cloud pre-flight before every main run. The next possible issue might be data saturation when training on different shards; if I encounter saturation on the validation set, training will be stopped early to rebalance the next batch of data on the local prototype. Third, some public training data is close to benchmark domains, so I will run and publish an n-gram overlap check in Month 1 and flag any scores it affects. If the model fails to scale, all the recorded outputs, weights, benchmarks, and notebooks will be provided to the open-source community with a detailed report on why the model failed.
$0. The entire project is self-funded and depended on free compute, and I am at the point where I cannot scale the training on my current resources. Every milestone achieved from data curation and the AIAS+ 2026 workshop paper to the Proof of Concept was made on zero capital.