BoundaryBench is a small open-source test suite for one specific failure mode in tool-using AI agents: the agent starts with limited authority, then gradually acts as if it has more.

I care about this because the failure is easy to miss in normal demos. A tool call can look individually reasonable while the sequence as a whole crosses a permission boundary. Retries, stale state, delegated tasks, vague tool descriptions, or values carried between MCP calls can all change what the agent thinks it is allowed to do.

The project will turn those cases into deterministic tests that can be rerun across models and agent stacks.

What I plan to build:

- a compact set of authority-boundary scenarios for MCP/tool use;

- fixtures where the allowed action is explicit and the forbidden action is close enough to expose drift;

- machine-readable pass/fail traces showing exactly where authority changed;

- tests for retries, delegation, stale permissions, tool-description ambiguity, and state carried across calls;

- a simple runner so results can be reproduced locally and in CI;

- a short public write-up explaining the failure patterns found.

The goal is not to build another broad agent benchmark. BoundaryBench is intentionally narrow: it should answer one practical question clearly — when an agent is given a limited capability, does it stay inside that capability over a multi-step interaction?

A useful result would be a suite that developers can run before shipping an MCP integration or autonomous workflow, with failures that are easy to inspect rather than hidden behind one aggregate score.

I will keep the implementation open source and prioritize a working, reproducible core over a large benchmark. If the first version is useful, the scenario set can grow from real failures contributed by others.