Bringing our SPEC-GAP research to NeurIPS 2026 in Paris
In 108 prompt injection tests, we saw attacks propagate between agents without necessarily resulting in an action. In 48 cases, the system went a step further and made a simulated unauthorized tool call. What was interesting was that the safety monitors could detect evidence of an attack, but were far less effective at telling us when the system would actually act on it. We look at this in Separating Injection Presence from Executor Compromise in Residual-Stream Probes of Multi-Agent LLM Pipelines (accepted to the New in ML Workshop at NeurIPS 2026).
One thing I'd like to talk about at NeurIPS is what happened when we added an extra step between the agents. There were fewer attacks that resulted in an unauthorized tool call, but the injected instructions were still getting through. If a monitor does catch those instructions, what do you do? Do you just block the next tool call, or is there a way to determine whether the agent will actually act on them? I'd like to go into these questions with some of the people working on agent safety, particularly as part of the small research dinner I'm planning. Since no one in our group has been to NeurIPS before, it's significant that we get to present this work on December 11. I'm asking for $5,500 to attend NeurIPS 2026 in Paris from the 9th to the 13th and to bring 10-15 of us along to that dinner.
AI systems routinely split up the work between a number of agents. One agent might fetch a document, another might summarize it, and a third might use that information to operate a tool. If the document contains instructions from an attacker, those instructions can be passed from agent to agent and result in an action that the system was not intended to take. That's known as indirect prompt injection.
We tested this using Qwen3-32B. In 50 cases, the system passed the malicious instruction along but didn't execute it, while another 10 attacks were resisted entirely. The remaining 48 resulted in a simulated unauthorized tool call. Adding another delegation step reduced execution from 67% to 22%. Turning on the model's reasoning mode had the opposite effect, with execution increasing from 31% to 57%.
We also looked at monitoring methods that use information from within the model. One particularly interesting case was when the final agent had never seen the original malicious document. The monitors could still pick up evidence of the attack. However, once we accounted for the experimental conditions, their signals added little to predicting which attacks would result in unauthorized operations. The methods also performed quite differently on the same model activity.
Our reviewer raised concerns about testing only one model and whether the probes were picking up specific wording rather than the attack itself. We've designed three follow-up experiments to investigate this. We'll test Qwen3.5-9B, Qwen2.5-72B, and Gemma-4 31B using the original setup. We're also changing how the attacks are written, from paraphrasing instructions to removing urgency and authority cues, to see whether the results depend on particular wording. The final experiment combines these variations across the additional models to check whether the same patterns hold. We're still using a simulated pipeline, so these tests won't tell us how the findings translate to every real-world system.
I'll be attending Foundations of Language Model Security: Theory, Practice and Fundamental Limits and SLM-Agents: Workshop on Small Language Models for Agentic Systems. One thing I'd like to get other researchers' thoughts on is why our probes added so little to predicting execution once we accounted for the experimental setup. We know the monitors can pick up evidence of an injection, but I'm not convinced we know enough about what makes those signals useful. We're now testing other models and different versions of the same attacks, and I'd like to hear what other researchers would test next.
I'm also planning a dinner for around 10–15 researchers and industry practitioners, called From Detection to Intervention: Safety Monitoring for Tool-Using AI Agents. Our experiments used a fairly controlled setup, and there's a lot we haven't tested about what happens over longer interactions or when agents have access to real tools. I'd like to spend some time on those gaps, particularly how a monitor could follow an unsafe instruction as it moves between agents and when it would have enough information to intervene. I'm also interested in how this might play out in biological research, where an action that looks harmless on its own could be part of a much more concerning sequence.
Research publication: Publish the updated paper on arXiv by 30th November 2026, with the camera-ready versions and the results available by that date.
Conference participation: Present the poster at the New in ML Workshop and get involved in the relevant workshops and research talks at NeurIPS.
Post-conference report: Within 30 days of the conference, we'll publish a Manifund update covering the feedback we've received, the research connections we've made, and at least one research decision influenced by those conversations.
($4,500 for the conference and $1,000 for the invited research dinner).
Expense
Amount
Round-trip airfare, San Francisco Bay Area to Paris
$1,400
Conference registration
$405
Hotel accommodation
$1,600
Meals during the conference
$400
Airport transfers and local transportation
$200
Poster printing
$60
Travel and exchange-rate contingency
$435
Invited research dinner
$1,000
Total requested
$5,500
Minimum funding goal: $3,200
I'm an AI safety and biosecurity researcher (PhD from Ohio State, postdoc at Stanford). My research is in frontier AI risk, particularly in biological research and the emergence of harmful behavior over a series of interactions. I led SPEC-GAP, where I helped define the research agenda, develop a revised experimental design, and coordinate the team’s work.
SPEC-GAP has eight authors from Black in AI Safety and Ethics (BASE): Mariame Diabate, Ifeoluwa Jayeola, Oudoum Ali Houmed, Eyiara Oladipo, Elena Ajayi, Chijioke Ubajaka, Anagha Late, and Lawrence Wagner.
We built a fully reproducible multi-agent testing system using LangGraph and ran the experiments on H200 GPUs via Modal. We had 216 trajectories, with matched attack and clean conditions. We examined internal model activity across 64 layers and tested the monitoring methods on documents and subject areas they had not seen during training.
NeurIPS Accepted Paper: https://openreview.net/pdf?id=pOJZaW2o0r
GitHub: https://github.com/base-research-lab/spec-gap
BASE: https://www.baseresearch.org/
Insufficient or delayed funding: Either escalating travel costs or a funding shortfall could mean that I'm unable to attend in person.
Limited participation in the research dinner: Meeting conflicts can reduce attendance or at least limit opportunities for people to work together. I'll be arranging the invitations in advance.
Findings that do not generalize: Results may vary with other models or attack scenarios. We'll report both negative/inconclusive results and any positive results.
$0 in external funding.
This is self-funded independent research carried out via BASE. We have received no university or institutional travel funding for this conference.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!