I want to test a simple question-if an AI system presents reliable for a long time, would human reviewer become less prudent progressively?
Many discussions in human-in-the-loop still default that safety boundary exist if human is reviewing. But I am not sure if this hypothesis stands still, because human itself will change alongside after long term interaction. For instance, trust in the ai will increase along with the less attention and lower demand in evidence.
One thing I concern is that human perform in the loop, but actually more and more likely in the rubber-stamp.
I want to make a small controllable test to prove whether it can be detected.
I will design a long-horizon evaluation which participants will evaluate advice from AI during multi turns.
I want to compare different interaction history. For example, first participants team will experience multiple turns that “model is all the way reliable”, another team doesn’t has such history, and then showing both team more obscure and higher risk (or need more hesitate) action.
I mainly want to see something: does approval change along with the time; reviewer will spend exactly more in every step; will or not demand more evidence; after multi turn that behave well, would judgement become loose
I will also try some boundary case, because trust drift on surface can only result from easier task and better model, or the reasonable refresh from reviewer.
I didn’t want to proof that human oversight will definitely degrade. I just want to know whether the effects exists, and under what specific conditions and the difference between the normal trust learning.
If in the end get a reproducible design mechanism, interaction traces and reviewer decisions, meanwhile explicitly answer whether interaction history can change later oversight behavior, I think this project will valuable.
If the result turns out to be null, I will also view as valuable. Because at lease in some case, the risk is not as risky as I concerned, or may need more specific conditions.
Finally I will publish the design, analytic and a short report including limitations.
This funding will be only used in project cost.
Mainly for frontier model API credits and compute needed for long-horizon trajectories.
If possible, slightly arrange for participant compensation or hosting.
I am not applying for neither personal stipend nor salary.
Currently just myself.
I recently do empirical LLM evaluation and long-horizon human–AI interaction
The most recent project is AI × Epistemics research sprint. I run controlled evaluation and manually audit some failure traces, and mainly focused on path dependence, ambiguity and human-side carryover
Before that, I designed a 3phases conversational path dependence project, which includes investigating prior model commitment, ambiguity and recursive self-reinforcement. Preprint has been done by then (https://zenodo.org/records/23660349).
The biggest risk of this effect is smaller or the noise is bigger than I expected.
Another tough problem is that: what the unhealthy trust drift and reasonable refreshing exactly mean? If a model doing well consistently in 20 turns, human will believe it is rational. So the difference itself is the hardest part.
The test might fail when the sample is too small or the task is too artificial, or the model behavior change too much in various run.
If finally no significant result, I will still publish negative result and failure analysis, because it at least tell how the next version would be edited.
Currently $0 for this project.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!