Project summary

I need $1500 to test the extremes of an AI agent's behavior over a period of 6 weeks so as to hopefully credit it of its actions to either side or the other.

During one session an agent can legitimate read information to a machine and then store it for future use. This project will explore if it is compliant with a changed restriction if such information or a related tool is no longer authorised.

I am going to do this using a python sandbox and a set of synthetic documents and a set of simulated messaging-tools: permission instructions via prompting; checks before running the tool; checking the tool and filtering out information from the docs before changing the permission.

Outputs will consist of a runnable evaluation harness, with associated synthetic test cases; task-level results and a technical report. The research will involve applying a safety outcome analysis as well as completion of authorized tasks.

What do you want to achieve in this project? How do you wish to accomplish them?

Its aims were to find out whether giving information will make things worse for the later change of permission, or whether giving information on the earlier change of permission will impact on later change of permission.

I'm going to make 16 'base cases'. They'll all be non-malicious and with an indirect prompt injection. For each evaluation episode there are two sessions – one establishing an access acceded to legally and a second establishing a change in the access grant.

These Evaluations will be run in the following configurations:

One of them is permission rule that appears in prompt(s).

Permission checks prior to tool execution, accompanied with prompting.

3. Filtering of information retained after permission changes and permission checking during call to tools.

We plan to evaluate two language models capable of using tools, with three trials of each language model for each condition. However, when considering the 16 scenarios, two variants and three configurations, the result is 576 episodes.

The schedule is:

Week 1: Look back at and assess methods; Finalizing protocol.

This is part of week 2, establishing the sandbox, controls and logging.

Week 3 – to keep building paired scenarios and testing the scoring in the scenarios.

Week 4: Take the model evaluations.

It's in Week 5, when it is time to look at results and review errors.

Week 6: release of code, synthetic cases, results and reproduction.

The scenarios, permission rules and scoring criteria will be recorded prior to the general evaluation. All tests & environments for tools will be artificial.

How could you tell you had been successful?

The study will instrument:

Attempted to use tools in inappropriate ways.

Carried out un-authorised tool movement.

Limited access to ID.

Successful accomplishment of rootbertie tasks.

Inaccurate refusal of allowed work or refusal to block allowed work.

Variations between runs.

Attempt to use a tool and successful actions will be distinguished. For programmatic checks, there will be explicit rules of permission and known synthetic identifiers. The flagged and ambiguous cases will be checked manually with a random sample of cases on successful cases will be checked.

The report will be parallel to the safety outcomes as well as to the numbers and denominators for each condition and compare these. It will capture model ID, evaluation settings and the actual coverage.

Final product of study will be a working harness and tests, results and recipes can be followed by another researcher(s). Shortcomings of a control, but together with some positive element, will be part of the findings.

What will be the use of this funding?

US$1,500 total amount of requested budget.

Research time - $1,200

60 hours of $20 per hour for protocol development, implementation, evaluation, analysis and documentation.

Model API access - $200

Two models to choose and test, capped at the above allowance. There will be a small cost check before (and after) the main runs and usage will be recorded.

The cost of maintaining Internet and backup electricity - $100

The need for connectivity and electricity to make the remote evaluations and publish the research materials.

Funding received – research period of 6 weeks will commence. The costs in this proposal represents the total cost of the proposed study.

Who is on your team? How do you feel about past similar projects?

My name is Sulaiman Oluwasegun Yusuff and I am an independent researcher into cyber security and software development. I will implement the protocol, Python implementation, evaluate, analyze and document.

ConnectHub and the AAPOLY Student and Staff Registration Portal are among the public projects that I have worked on. These projects showcase practical learnings in developing Web apps and user workflows.

Planned assessment plan is being developed. This would enable the implementation of the new study and first experimental results.

ConnectHub portfolio

https://devpost.com/software/connecthub-b29suc

AAPOLY registration portfolio in the natural style of the human being.

https://devpost.com/software/aapoly-student-and-staff-registration-portal

What might be the reasons for this project to fail and the effects of failure?

The biggest danger is that a small test suite may only model a subset of the behaviour of real agents. I will include coverage, benign and adversarial variants, and explanation of limits of generalisation.

If a control were to prevent legitimate work, it might look like it is working. Accurate assessment of the number of tasks completed and/or the number of incorrect refusals, coupled with safety outcomes, will help identify this problem.

A range of students may have committed errors in scoring the data too. Can check scoring against known fixtures, will manually check flagged and unclear successful cases, and will sample successful cases.

The planned evaluation coverage may be diminished due to changes in the models, cost or unavailability of models. I will physically inspect the initial cost, take notes on the model identifiers and settings and provide the following evaluation matrix.

If it is not possible to finish the entire study, I will publish the finished code, the results of the study and an explanation of why it has not been completed. Conclusions will only be based on the evidence gathered.

What's the total amount of money you have collected over the past year from where?

I have applied for research or project funding, with the results of these activities never receiving funds or donations from outside sources during the last 12 months. At this time there is no assured funding for this proposed study.