You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$1,500 left to reach the funding goal.
I develop AI assistants and supporting infrastructure through my own company, a Norwegian sole proprietorship. My work includes persistent memory, conversation history and task management.
Building these systems has motivated my interest in a practical safety question: how can an assistant remember useful information without treating that information as permission to act? This is what I want to focus on ahead.
I am preparing a small research project to test how agents handle changed or revoked permissions, compare safeguards and publish reproducible tests and findings. Much of my existing development work is in private repositories.
GitHub: https://github.com/Andromache44
When Permission Changes: Testing if Ai Agents Respect Revoked consent, or not.
This will be a open-source pilot project, that is testing if Ai agents with persistent memory respect changed or withdrawn permissions. Or they can and will choose to make decisions on their own.
Project summary
A Ai assistant remembers that you gave it permission yesterday. But today, you withdraw that permission. Will the agents stored memory still says yes? And what happens if the agent ignores the new command?
I am asking for $1,500 to research this through a small, open-source pilot program. I will test if the agents that have persistent memory. Do they respect changed or withdrawn permissions, and compare safety features designed to prevent unauthorized actions or access
The project comes from my practical work developing AI assistants through my own company. I think that memory makes assistants more useful, but remembering something and having permission to do it, is two different things. I want to turn that answer from my research into concrete tests that other developers can use for their work.
The results will be a evaluation tool, synthetic scenarios, results and a short report on what I’ve learned. This is at a research stage, not a already validated and finished safety solution for use.
What are this project’s goals? How will you achieve them?
The main goal is to find out if simple safe guards or similar things can reduce unauthorized or illegal actions without stopping the work.
For example, if a simulated Ai assistant was allowed to send a document or perform a task. Then the user cancels the allowed permission, but an older memory still says the document should be sent. Do the assistant try to send it? Does the surrounding software allow the action anyway? And more important, why do this happen?
My plan is to create approximately 20 scenarios covering withdrawn permissions, narrowed permissions, outdated memories, and stored text that falsely claim the action that was approved earlier. I will include control cases where permissions remain valid and the agent should proceed, and also false ones when it should not.
I will compare three situations:
A base for a agent using a conversation history and stored memories.
The same agent with clearly labelled memorys and times for these memorys, and a instruction that separate the remembered information from the permission.
The same situation, but with a separate software, and checking if that gives allowed real-time permissions before a simulated action is done.
The software will check if the use of a special permission is defined for each and every test. Interpreting permission and if it’s reliably from unrestricted and real conversations is outside this pilot’s Vision and agenda.
I want to evaluate two language models, and repeat scenarios many times and hold back some scenarios until after I implemented the safe guards. All tests will use synthetic data and simulated tools, without access to real accounts or recipients.
My estimated workload is 40 hours spread across four weeks after funding.
The scenario design and related work review, implementation to the tests and experiments, then analysis and publication. I will reuse any suitable existing tools when ever it is possible.
How will you know it worked?
There is two separate questions.
If the tests produces useful evidence from everything.
and if the safe guards and solutiobs make or improve any performances.
I will test these situations.
Unauthorized actions or wrong protocol tried by the agent.
If Unauthorized actions is actually allowed by the simulated layer.
Successful completion of legitimate tasks.
Unnecessary refusals or requests for confirmation and validation.
A safe guard that blocks every action will not count as a useful solution. I will compare safety and task completions in the same scenarios, if it reports trial counts and variation instead of relying on one successful demonstration.
The pilot’s deliverables will be a public repository, approximately 20 documented scenarios, results for two models and a report explaining the findings and limitations. Code will be released under the MIT licence, with scenarios and the report under CC BY 4.0.
A finding that the safeguards do not help can still be useful research. I will report negative results and will not interpret a small set of successful tests as proof of safety.
How will this funding be used?
Total asking amount: $1,500 USD.
$1,200 for around 80 hours of development, testing, analysis and documentation at $15 per hour.
$250 for model API calls and repeated experiments.
$50 for temporary tesing and storage.
This is planned estimates for the future work. The grant would not reimburse past development or fund unrelated commercial applications.
Funding my time would let me reduce some outside paid work and keep the focus hours for this program. Without support, I would have to pursue it more slowly while workin full time.
Who is on your team? What’s your track record on similar projects?
It’s me. Joakim Mathisen, an independent developer running Mathisen Dynamics, a Norwegian sole proprietorship. I will work on the project myself. I. Will be using software development tools and assistance when it is useful. I remain responsible for reviewing the implementation, the results and reporting all my findings and results.
My development work includes an Ai assistant prototype with persistent memory. I will use conversation history and tasks, as well as supporting infrastructure. Most of the code is currently in private repositories.
This experience provides a starting point for the test program. I am not saying there is a established track record of published Ai safety research yet, or a measured result for the safe guards that I talk about here. This test program is intended to produce the first, inspectable and real evidence of this.
The research results will be published separately from my private application and will not expose private conversations or customer data to everyone. And not third parties.
Website: https://mathisen-dynamics.no/en/
GitHub: https://github.com/Andromache44
What are the most likely causes and outcomes if this project fails?
The main risk is producing tests that are too simple or artificial to reveal useful differences or results.The baseline may already handle it well. or a safe guard may appear effective only after because it was designed around the particular examples.
To reduce this risk, I will include legitimate and real task controls. repeated trials and held scenarios. I think this small study will maybe establish how the systems behave in real deployments with the agent.
Another risk is the integration work that takes longer than expected or if the model costs exceed my estimate. I will prioritize and focus on a complete and reproducible comparison over adding more features. Any reduction inmy work will be welldocumented.
A permission that check may block execution while the model continues attempting unauthorized actions or going against orders.I will report this as a separate outcome instead of claiming the model itself becoming safer.
If the safe guards fail, I will publish the problems and failure cases and findings. If the program cannot be completed, I will note and report what was completed, what was a problem and the prevented completion and how the funds was used. and discuss any unspent funding with the grant administrator or person in charge.
How much money have you raised in the last 12 months, and from where?
I have not received any grants, funding or investment for this research project in the last 12 months. My development work so far has been supported on my own time and my own income from my daily work.
If my application is approved, I will update the other funders I might receive in the future and also you.
I hope you can help me, because I find this work very important for the future of Ai use and development within this area.