You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$11,600 left to reach the funding goal.
Reward models are an important part of post-training alignment in RLHF. They are used to score the model's responses based on human preferences, helping the model learn which outputs are preferred during alignment. Most reward models follow the standard Bradley-Terry pairwise training approach, where they are trained on human preference data containing one chosen response and one rejected response. The model learns to assign a higher score to the chosen response.
There are also other specialized methods, such as ORM (Outcome Reward Model), which is trained to score the model's final answer, and PRM (Process Reward Model), which evaluates the model's reasoning through step-by-step supervision. Even when trained on large amounts of preference data, these models can still suffer from verbosity bias. Sometimes, a longer and well-formatted response gets a higher reward even when the reasoning is incorrect or the final answer contains completely fabricated information.
I wanted to explore a different approach to this problem. This led me to develop Associative Reward Modeling (ARM), a multi-objective reward model that evaluates the model's output across six quality dimensions instead of relying on a single score. These dimensions are connected so that qualities like writing style cannot outweigh failures in reasoning or correctness. Each preference pair is trained using 13 supervision signals (1 Bradley-Terry preference signal + 6 chosen response signals + 6 rejected response signals). I developed the initial proof of concept using an 8B base model.
Proof of concept: https://huggingface.co/Stellar-lab/Stellar-Qwen3-8B-ARM
Evaluation code and results: https://github.com/pradeep-016/stellar-arm
The current dataset contains 9,000 preference pairs (7,500 training, 750 validation, and 750 test pairs). Our goal is to expand this dataset to approximately 63,000 pairs and evaluate how ARM performs with a larger amount of training data.
To achieve this, I plan to work in three stages:
Local experiments: I will start with a Qwen3-4B model to experiment with different loss weights, learning rates, and data mixtures. I also want to compare ARM against a standard scalar reward head using exactly the same training data. This will help identify whether the multi-objective approach actually provides an advantage.
Scaling the 8B model: Once the training configuration is stable, I will move to RunPod and scale Qwen3-8B using sequential data shards. The training will use FP16 or BF16 precision with LoRA/DoRA. Before each major run, I will perform a short pre-flight test and evaluate the model after each shard.
Moving to larger models: I will consider moving to a larger backbone, such as 30B, or using full fine-tuning only if the results from the 8B experiments justify the additional compute requirements.
Regarding the data, I did not generate additional preference data using an LLM myself. I combined publicly available datasets with rule-based corruptions of correct answers. For example, changing a digit in the final mathematical derivation, cutting an answer short, or introducing a bug into a code response.
The dataset covers seven areas: general chat quality, mathematical reasoning, coding correctness, instruction following, incomplete or truncated responses, fluent-but-wrong versus plain-but-correct answers, and safety with refusals. Some of the original public datasets contain model-generated text, while the corruptions and their associated supervision targets are generated using predefined rules rather than human annotations. Every preference pair carries all 13 supervision signals mentioned above.
Our proof-of-concept model, Stellar-Qwen3-8B-ARM, was trained on 7,500 training pairs, 750 validation pairs, and 750 test pairs using 4-bit QLoRA on free Kaggle 2x T4 GPUs. From a single checkpoint, it achieved the following results:
RewardBench 2: 74.20% overall macro accuracy. This is ahead of URM-LLaMA-3.1-8B (73.9%), GPT-4.1 (72.3%), Claude Sonnet 4 (71.2%), ArmoRM-Llama3-8B-v0.1 (66.5%), and GPT-4o (64.9%). There are stronger models available, such as Skywork-Reward-V2-Llama-3.1-8B (84.1%) and Skywork-Reward-V2-Qwen3-8B (approximately 78%). However, these models were trained on around 26M curated preference pairs compared to the 7,500 training pairs used in my proof of concept. The 70B INF-ORM also scores 76.5%.
RewardBench 1: 85.52% overall using the official 50/50 weighting for math and code reasoning. A macro-average across all 23 subsets gives 89.08%, but I consider 85.52% the more appropriate number for comparison. The model scored 92.4% on Chat, 83.2% on Chat Hard, and 90.2% on Safety. Math-PRM is currently the weakest area at 56.4%, while models such as Skywork-Reward-Llama-3.1-8B-v0.2 (93.1%) and ArmoRM (90.4%) perform considerably better.
RM-Bench: 74.88% overall, with 82.69% on Easy, 76.44% on Normal, and 65.51% on Hard. This is the highest average among the reference models included in my evaluation notebook.
JudgeBench: 62.50% across 620 pairs, with 63.71% on the GPT-4o split and 60.93% on the Claude 3.5 Sonnet split. By category, the model scored 74.1% on Math, 65.1% on Coding, 60.4% on Reasoning, and 58.4% on Knowledge.
All four evaluation notebooks and their predictions are publicly available in the GitHub repository.
I also tried a short 100-step FP16 DoRA experiment on a Llama backbone. The training loss curves looked promising, but I lost the checkpoint when the session reset, so I cannot provide benchmark results from that experiment. Working across temporary GPU sessions has made it difficult to maintain longer experiments, which is one of the reasons I need more reliable compute resources.
I will consider the funded phase successful if I can achieve the following:
The comparison between ARM and a standard scalar head clearly shows whether ARM provides an advantage when both use the same data and backbone.
Scaling the 8B model across the full dataset produces consistent improvements over the initial 4-bit proof of concept across all four benchmarks.
Each new data shard maintains or improves accuracy on the fixed 750-pair validation set without affecting calibration across the six dimensions.
In best-of-N experiments (N = 4, 16, and 64), ARM achieves higher ground-truth accuracy and less answer-length inflation compared to a standard scalar reward model.
I release the model weights, code, training scripts, data pipeline, and the dataset itself wherever the source licences permit redistribution.
The funding will be used for three main purposes: local hardware for prototyping and testing, cloud compute for scaling the model, and living expenses for five months. I am requesting $11,600.
1. Local hardware – $3,600
I need a laptop equipped with an RTX 5080 GPU with 16GB VRAM. Multi-objective reward modeling requires a lot of experimentation with loss weights, learning rates, and other hyperparameters. These experiments often involve several rounds of trial and error, and repeatedly using cloud GPUs for every small change can become expensive.
My current laptop is around ten years old. It has an Intel i3 processor, 8GB DDR3 RAM, a mechanical hard drive, no dedicated GPU, and a dead battery. It struggles to keep even basic browser tabs open.
You might wonder, "Why not use cloud GPUs for all the experiments?" The main reason is that I need to make frequent changes to the training configuration and test different settings before committing to larger runs. Having local hardware allows me to perform these experiments without continuously paying for rented GPU time.
Another question might be, "Why not build a desktop PC?" In my local area, an RTX 5080 desktop graphics card alone costs around $2,800–$4,000 because of GPU shortages. After adding the CPU, RAM, storage, and other components, a complete workstation would cost around $6,000–$8,000, which is beyond my budget.
Although a laptop RTX 5080 is not equivalent to its desktop counterpart, its 16GB VRAM should be enough for Qwen3-4B experiments and short, quantized 8B smoke tests. The main purpose of having local hardware is to test the implementation, identify issues, and verify that everything works before moving to expensive cloud training runs.
I want to avoid scaling the work before confirming that the training setup is working properly. Local hardware will allow me to experiment with different configurations, check the training behaviour, and make adjustments without worrying about cloud billing for every small experiment.
2. Cloud credits – $3,500
Once the local experiments have been completed and the training configuration is stable, I will move to cloud GPUs for scaling. I plan to primarily use RunPod because it offers relatively budget-friendly GPU rental options for larger training runs.
At the estimated rates of $0.89/hour for an RTX 4090, $1.79/hour for an A100 80GB, and $3.99/hour for an H100 SXM, the $3,500 allocation would provide approximately 1,900 A100 GPU-hours.
This budget will cover the sequential data shard runs, experiments with multiple seeds, benchmark evaluations, storage, and additional testing. Before starting each major run, I will perform a short pre-flight test on the rented GPU to check memory requirements, training speed, and whether checkpoints can be saved and resumed properly.
3. Living stipend – $4,500
This amount covers my living expenses for five months ($900/month), allowing me to focus full-time on the research. This includes data curation, local prototyping, training experiments, monitoring, and evaluation.
Total requested: $11,600
If the project receives only partial funding, my minimum requirement is $8,000 ($3,500 for cloud compute and $4,500 for the living stipend). In that situation, I will use cheaper rented GPUs for debugging, although the experiments will take longer to complete.
Month 1: Local experiments, training reliability improvements, ARM versus scalar-head comparison, benchmark-overlap check, and the first cloud pre-flight test.
Months 2–4: Scale the 8B model on RunPod using sequential data shards, with a pre-flight test before each major run and benchmark evaluations afterwards.
Month 5: Move to a larger backbone only if the results justify it. Complete the final evaluations and prepare the research report, model weights, and code for release.
I am Pradeep L, the independent researcher behind this project, and I work on it alone.
I previously worked on a manuscript titled Evaluator Reliability Under Selection Pressure: Implications for Agentic Test-Time Search, which was accepted at the 3rd Workshop on Lifelong Agents (AIAS+ 2026). That research helped me understand some of the limitations of reward model evaluation under increasing selection pressure and eventually motivated me to develop ARM.
Website: https://pradeep-016.github.io/stellar-sar-lab/#/researcher
One of the main challenges is multi-objective optimization instability. Since ARM trains multiple objectives simultaneously, balancing the 13 supervision signals may introduce gradient conflicts or make training unstable. This is why I want to experiment with different configurations locally before moving to larger training runs. I will also perform a short pre-flight test on the cloud before each major run.
Another possible issue is data saturation. As the model is trained across successive data shards, its performance may stop improving after a certain point. If I notice that validation accuracy has stalled, I will stop the current training run and revisit the data mixture using the local prototype before proceeding with the next shard.
There is also a possibility of overlap between some of the public training datasets and benchmark content. I will run an n-gram overlap check during the first month and publish the findings, including any overlaps that could affect the evaluation results.
If ARM fails to scale as expected, I will still release the available model outputs, weights, evaluation notebooks, benchmarks, and training logs, along with a detailed report explaining what went wrong and why the approach did not work as expected.
$0. The entire project has been self-funded so far and developed using free Kaggle GPU resources and my existing laptop. Everything I have achieved, from data curation and the workshop paper to the 8B proof of concept, has been completed without external financial support.
I have now reached the point where I cannot scale the training further using my available resources. Additional funding is necessary to continue the experiments and evaluate ARM on a larger scale.