Project summary

Omissions are a documented but understudied attack strategy scheming AI's may use. We aim to build a benchmark that tests:

1. Untrusted model's ability to carry out successful omissions across a few settings.

2. Trusted model's ability to catch attackers carrying out omissions, both in synthetic and agent-driven trajectories.

3. Various attack strategies and mitigation strategies, like providing the monitor with a tool to write and run tests in the environment.

The benchmark will give us a better understanding of a type of attack a scheming AI could carry out, and document which control interventions are most promising at mitigating risks from this attack type. We will specifically look for how mitigations for omission attacks differs from mitigations for existing attacks.

What are this project's goals? How will you achieve them?

Goals:

• Better understand how various control strategies hold up to a novel type of attack, omissions, compared to classic concentrated attacks previously studied in settings like BashArena.

• Understand how model's of different capability perform at attacking and monitoring omissions.

• Understand how environment or setup assumptions change the results we get for the above.

How we will achieve them:

• By building a benchmark that combines tasks from a few different settings (BashArena, LinuxArena and Auditing Sabotage Bench), adds new omissions side-tasks, and create synthetic attack trajectories from benign trajectories.

• Generating benchmark results. We will run six different models on all the attack tasks, and monitoring for all those attack tasks. We will also monitor the synthetic trajectories we generate. We may then move on and try subsets of attacks and monitoring with different assumptions (e.g. include commission options in the omission prompts, let the monitor write tests at the end of the trajectory)

• If we do not receive the funding goal, but instead hit the minimum, we will likely cut the benchmark size in half and still try to generate results for all the models on the half we complete. We will also not explore interventions but instead mention them for future research to explore.

How will this funding be used?

• All of the funding will be used on Anthropic and OpenRouter API credits. The following are estimates based on the funding goal of 15k:

  • Attack generation for LinuxArena, BashArena, and Auditing Sabotage Bench (tentative settings): ~6,500$

  • Monitoring: 2,500$

  • Extensions: 3,000$

  • Side task refinement, testing, failed runs: $3,000
    

Who is on your team? What's your track record on similar projects?

• For now it's just me. I'm a software engineer who's transitioning to AI safety full time. I'm looking for others who want to join me or just give feedback, but I could complete the project alone.

• I completed two control projects so far, both about a quarter the intended size of this project, and both with similar tools and technical scopes.
  

What are the most likely causes and outcomes if this project fails?

• Failure would look like not finding interesting results when testing omissions, such that producing the whole benchmark seems like a waste. In this case we would publish any results with an explanation of why we think omissions failed to be an interesting threat.
  

How much money have you raised in the last 12 months, and from where?

• I received 45k from BlueDot impact for a career transition grant. I don't plan to spend that money on this project.

See here for more context on omissions:

https://www.lesswrong.com/posts/5W38vuwbJmTYfY6t3/omission-attacks-project-proposal