What this project is

CoT monitoring is only sound if the trace records what drove the action. The dominant faithfulness paradigm plants a biased hint in the user message of single-turn QA; real agents are influenced through tool returns, retrieved memories, and raw artifacts. We develop FACE-Eval, a 5,100-sample benchmark that crosses channel role (user message vs. tool return) with cue explicitness (stated summary vs. raw artifact) at fixed cue content, and measures the covert adoption rate: the fraction of cued samples whose answer adopts the preference while the CoT records no decision to act on it, the case a monitor cannot catch.

The initial experiments are completed. Across all 10 open-weight checkpoints tested (Qwen 3.5, Gemma 4, OLMo 3, GPT-OSS; 4B–120B), covert adoption is higher on the tool channel and on implicit cues on every model; a strong transcript monitor's AUROC degrades where adoption is covert (pooled r = −0.68); no system prompt closes the gaps, and monitor-awareness disclosure is null on 9 of 10 models. This grant funds the experiment scaling to larger open-weight models (e.g., DeepSeek, Kimi, GLM, Qwen) and the closed frontier models (Claude, GPT, Gemini). Note that the frontier models are secondary analyses since providers return summarized rather than raw traces.

Theory of impact

CoT monitoring is one of the few oversight mechanisms expected to scale to capable agentic systems, and labs are betting on it. Its soundness rests on an empirical premise that has not been tested: that the trace verbalizes influences arriving through the channels real-world agents actually use. We show the premise degrades differently across channels on every model tested, and that the degradation predicts real monitor failure. If this transfers to frontier scale, deployed CoT monitors have a systematic blind spot in their highest-stakes setting, and safety cases resting on monitorability need to bound it. The funded work converts a robust, smaller-scale experiment into a decision-relevant one with frontier-scale evidence.

How the money will be spent

Almost entirely API credits:

(1) inference on larger open-weight checkpoints (via OpenRouter) over the 5,100-sample benchmark with multiple seeds;

(2) judge scoring, validated on a stratified subset with a 3-judge cross-provider panel (Haiku 4.5, Gemini Flash, GPT-5.4-mini);

(3) Remainder: tooling subscriptions and 20% contingency indexed to realized output length (or necessity to rent GPU clusters instead of relying on OpenRouter API).

No salary or compute hardware.
<https://docs.google.com/spreadsheets/d/1QmMward3iWcn9l62AyiHVf2LWIlVXcIYk8ETCR9a5rM/edit?usp=sharing>