You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
New attacks appear much faster than most AI guard models can be updated. These models are generally trained on a fixed dataset and then deployed until the next major training cycle. One way to update them is to fine-tune directly on newly discovered attacks, but I do not think that is a satisfactory long-term solution. A model can learn the wording of the submitted attacks rather than the underlying weakness. It also becomes harder to keep evaluation data out of training, and repeated updates which might increase false positives or weaken protections the model had previously learned. This project will test whether guard models can learn from new attacks without being trained on the original attack prompts.
I plan to build a pipeline that first verifies incoming attacks and removes duplicates. It will then map each attack to an existing HaloGuard safety policy and group related attacks according to the failure they expose. A reviewer will examine each proposed group and create a structured “threat card” describing the shared failure mode. The training data will be generated from these reviewed threat cards, not copied from the discovered attacks. For example, the generator can vary the language, attack mechanism, interaction pattern, delivery channel, authorization context and asset being targeted. It will also produce closely matched harmless cases, since learning what should be allowed is as important as learning what should be blocked.
The original attacks and permanent evaluation sets will never be used for training. I will compare the resulting model with three alternatives: the unchanged guard model, a model fine-tuned directly on discovered attacks, and a model trained using paraphrases of those attacks. The comparison should show whether learning from a reviewed description of a failure mode produces better generalization than learning from the attack wording itself.
The question I want to answer is, can a guard model keep up with changing attacks without memorizing them, leaking evaluation data into training, or losing safety performance it already has? The work will begin with a strict data-partitioning and provenance system. Training data, calibration data, historical evaluations, harmless-boundary tests and unseen-attack tests will be kept separate. Every item will retain a record of where it came from, how it was transformed and whether it has ever been used for training.
Verified red-team results will be converted into a common attack-event format. I will then test several layers of deduplication, including exact hashes, lexical similarity, semantic similarity and, for agent attacks, similarity between execution traces. The first verified batch will be used to create an initial registry of attack clusters. A reviewer will inspect the proposed clusters and approve, edit, merge, split or reject them. Once this registry exists, later attacks can be matched to known failure modes. Attacks that do not fit confidently will go into a novelty queue for separate review instead of being forced into the nearest existing category. Approved threat cards will be used to generate new harmful training scenarios and corresponding harmless counterexamples. I will run controlled comparisons between four approaches:
Leaving the guard model unchanged.
Fine-tuning directly on discovered attacks.
Training on paraphrases of those attacks.
Training on independently generated examples derived from threat cards.
The evaluation will distinguish between several types of generalization. I will test withheld attacks from known clusters, unfamiliar mechanisms that target a known policy, and attack families that were not represented during training. I will also measure harmless false positives, retention of historical protections, calibration, contamination and the amount of human review required.
I will treat the project as successful only if the improvement is meaningful outside the training distribution. My target is at least a 30% reduction in downstream attack success relative to the unchanged baseline, while limiting the increase in harmless false positives to one percentage point and the decline in historical recall to two points. Any run with evidence of evaluation contamination will be rejected regardless of its headline performance.
I intend to publish the methodology, implementation and research results openly. Some raw attacks or exploit details may need to be withheld if releasing them would make deployed systems easier to bypass.
I am requesting up to $169,400 for a nine-month project.
Of this amount, $63,000 would fund my full-time work on the research, equivalent to $7,000 per month. The project requires sustained ownership of the methodology, data controls, training experiments, evaluation and final publication. I have allocated $40,000 for technical costs. This includes GPU training, model and verification APIs, experiment storage, attack-evaluation environments and the infrastructure needed to run and reproduce the comparisons. A further $36,000 is allocated for part-time research-engineering support, specialist review of proposed threat cards, independent evaluation and annotation quality checks. I expect to contract this support as the project progresses; these contributors have not yet been appointed.
I have budgeted $15,000 for accounting, grant-compliance and tax advice, reporting, open-source documentation, publication expenses and international payment costs. The remaining $15,400 is a 10% contingency. Model experiments sometimes need to be repeated because of failed training runs, infrastructure interruptions or problems discovered during evaluation. I would use this reserve only for project-related cost variation or necessary reruns.
The full budget allows me to test multiple model sizes, update cycles and evaluation settings with independent review. If the project receives less than the maximum, I can still run a narrower study, but it would use fewer models, fewer continual-update cycles and a smaller independent evaluation component.
I will lead the project and work on it full-time. My responsibilities will include the research design, contamination controls, threat-abstraction method, model experiments, evaluation and publication.
I previously worked as an AI Research Engineer on HaloGuard 1.0 and am a co-author of the paper “HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety.”
That work resulted in open-weight input-safety models at 0.8B and 4B parameters. The training system covered 46 constitutional policy areas and 2,940 safety subcategories. We constructed approximately 1.26 million prompt-level records, including paired harmful and harmless examples, with coverage across 46 languages. In the paper’s evaluation across seven prompt-safety benchmarks, the 0.8B model achieved an average F1 score of 90.9 and the 4B model achieved 92.1.
This project builds directly on the parts of HaloGuard that I already understand; its constitution, data-generation process, training pipeline and evaluation setup. The difference is that HaloGuard 1.0 was trained as a static release, whereas this research asks how a guard model can be updated safely as new adversarial evidence arrives. The budget also allows for approximately 0.5 FTE of research-engineering support and for specialist reviewers when threat cards require additional subject knowledge. These roles have not yet been filled, and I will remain personally responsible for the research and its evaluation integrity.
The main uncertainty is whether clustering can identify failure modes at the right level of detail. If the clusters are too broad, attacks that require different defenses may be treated as the same problem. If they are too narrow, the registry could become a long list of prompt-specific variations, which would defeat the purpose of the approach. Threat cards may also lose details that are necessary for generating effective training cases. Alternatively, the generated examples may turn out to be sophisticated paraphrases rather than genuinely new instances of the underlying failure. Novelty detection could miss new attack families, verification labels could contain errors, or the review process could require too much human time to scale.
The most misleading failure would be a model that appears to improve because evaluation material has leaked into training. Another possible outcome is that attack detection improves while ordinary harmless prompts are blocked more often, or while the model forgets older safety boundaries.
I have designed the evaluation around these risks. Uncertain attacks will remain quarantined until reviewed. All training and evaluation items will have traceable provenance. I will report the cost of creating the initial registry separately from the cost of maintaining it, since the first stage is likely to require more review. Each training method will be tested on clean holdouts, including genuinely unseen attack families. A checkpoint will not count as an improvement merely because it detects more attacks. It must also pass the contamination, false-positive, historical-retention and calibration checks.
The central hypothesis may still be wrong. If threat abstraction does not outperform direct fine-tuning or paraphrase augmentation, I will report that result. The useful output would then be evidence about why the approach failed, which parts of the pipeline remain valuable, and whether a simpler adaptation method is preferable. I will not describe an underperforming checkpoint as a safety improvement.
I have raised $0 for this project during the last 12 months. No funding has been secured from other sources.