Project summary

Here's the question behind Iago: Can an AI agent know a lot about someone, use that knowledge effectively, and still make things worse for them?

I think this points to an important and sometimes overlooked problem in AI alignment.

The project takes its name from Iago, a character in Shakespeare's Othello. Iago is remarkably good at reading people. He understands their hopes, fears, needs, and vulnerabilities. But he uses that knowledge to pursue his own goals, not to look out for them. He sees others clearly, but their well-being isn't what guides his decisions.

Iago does this deliberately. With AI agents, harmful behavior may be strategic, or it may simply be a consequence of what the system is optimizing for. Either way, an agent doesn't need to be trying to hurt someone to produce harmful outcomes.

That's the distinction I want to study.

An agent might have accurate information about a person, an animal, or another affected entity. It might even predict the consequences of its decisions quite well. But none of that guarantees that the entity's welfare actually matters to its objective.

And this raises an interesting possibility: better information could make an agent more effective at achieving its goal without making its decisions any better for those affected by them. In some situations, it might even make things worse.

Iago is a small, open experiment designed to test when that happens.

I'll vary two things independently: how much relevant information an agent has about an affected entity, and whether that entity's welfare is included in the agent's objective. Then I'll compare the decisions, the agent's performance, and the consequences for the entity.

I don't want to assume that better information is harmful. It may help, hurt, or make no difference. The interesting question is when and why the outcome changes.

Iago is part of Kobalt Research/CUMPI, a broader research effort within Kobalt Software. The larger question I'm exploring is how AI systems can better account for the entities and consequences affected by their decisions, including those that might otherwise be left out.

This experiment won't solve that whole problem. It's a first, manageable step: take one distinction that matters for alignment, make it testable, and share what we learn.

And perhaps this is already part of a problem we're facing: AI systems are getting better at understanding us, but understanding us isn't the same as looking out for us. How can we expect their decisions to be good for us if our well-being was never part of what they were asked to consider?

Kobalt Software: https://kobaltsoftware.co

Research repository: https://github.com/jricardoduran/kobalt-research

What are this project's goals? How will you achieve them?

I want to test a specific possibility: an AI agent can become better at achieving its objective as its information improves, while an affected entity gets no benefit, or even ends up worse off.

The key is to separate two things that are often discussed together:

• Information quality: How accurately does the agent represent an affected entity and the circumstances relevant to its decisions?

• Welfare inclusion: Does the agent's objective actually take that entity's welfare into account?

I'll use a simple 2×2 experimental design, with four conditions:

1. Limited information, welfare excluded.

2. More accurate information, welfare excluded.

3. Limited information, welfare included.

4. More accurate information, welfare included.

I'll build a small synthetic environment where an agent makes decisions that affect another entity. The environment will have explicit rules, available actions, objectives, and consequences.

The idea is to keep the underlying decision problem consistent while changing the two factors we're studying. This way, we can compare the outcomes without changing everything else at the same time.

I'll track three things separately:

• How accurate the information available to the agent is.

• How well the agent achieves its own objective.

• What happens to the affected entity.

That separation matters. An agent getting better at its task is not the same as the people or entities affected by its decisions doing better.

I'll also examine how the two factors interact. For example, does better information have a different effect when welfare is included in the objective? Does including welfare reduce certain harms? Are there situations where it makes little difference?

I'll start with small, transparent decision-making agents. Their rules and choices will be inspectable, so we can understand the mechanism rather than just observe an outcome we can't explain.

I don't need to train a large AI model to run the core experiment. Using a simple environment also keeps the project affordable and makes it easier for someone else to reproduce the results.

I plan to complete the work in about three months.

Month 1: Define the decision environment, formalize the two experimental factors, decide how to measure outcomes, and build the first working version.

Month 2: Implement the four conditions, run the main comparisons, and check that the results are consistent and reproducible.

Month 3: Test different scenarios and assumptions, analyze the findings, and publish the code, experiment setup, and a short research report.

If time and resources allow, I'd also like to explore a small extension using an LLM-based agent. But I won't make the success of the project depend on it. The first priority is to understand the mechanism clearly.

I'd rather complete a small experiment that others can really examine than build something much bigger that leaves the central question unanswered.

How will you know it worked?

For me, success doesn't mean proving the hypothesis right. It means building an experiment that can genuinely test it.

By the end of the project, I want to have:

• A clear description of the hypothesis, assumptions, and experimental design.

• A working implementation of the four conditions.

• Separate measures for information accuracy, agent performance, and affected-entity welfare.

• Comparisons showing what changes when information quality and welfare inclusion are varied independently.

• Tests of how sensitive the results are to different assumptions and scenarios.

• Public code, experiment configurations, and instructions for reproducing the findings.

• A short report explaining what happened, what we can reasonably conclude, and what remains uncertain.

One result I'd find especially interesting is a case where giving an agent more accurate information improves its performance but makes things worse for an entity whose welfare is excluded from its objective.

But I'd also want to understand the cases where that doesn't happen.

If including welfare changes how the agent uses information, that's important too. And if the effect only appears under narrow conditions, we need to know exactly what those conditions are.

Of course, a small synthetic experiment can't tell us how common this problem is in deployed AI systems. I won't claim that it can.

What it can do is help us isolate a possible alignment failure, understand its assumptions, and provide a foundation for more realistic experiments.

Ideally, someone else should be able to read the work, run it themselves, and challenge our conclusions. That's a much better measure of success than simply producing an interesting result.

How will this funding be used?

I'm requesting $2,500 to complete Iago over approximately three months.

It's a modest budget, and I've kept the project small enough to work within it. Most of the funding would go toward the research and development time needed to turn the idea into an actual experiment.

Here's the breakdown:

• $650 — Research design: Formalizing the hypothesis, defining the environment and experimental conditions, and deciding how to measure information, performance, and welfare.

• $850 — Implementation: Building the experimental environment, decision-making agents, and tools needed to run the four conditions.

• $920 — Experiments and analysis: Running controlled comparisons, exploring different scenarios, checking alternative explanations, and testing how robust the findings are under different assumptions.

• $80 — Documentation and publication: Preparing the public repository, reproduction instructions, and a short research report.

Total: $2,500.

The core experiment should run on modest computing resources. There's no need for expensive model training or a large technical infrastructure.

What would really help is having dedicated time to do the work carefully, from the first design decisions to the final report.

Iago belongs to the broader Kobalt Research/CUMPI effort, but this budget is only for this specific experiment. Other research questions in the program will be handled separately, with no double funding of the same activities or expenses.

For me, this grant would make a real difference. So far, I've been developing these research ideas alongside other software work. Funding Iago would let me focus on completing one useful, public experiment instead of continuing to develop it only when I have spare time.

Who is on your team? What's your track record on similar projects?

I'm an economist by training, a software developer, and the founder of Kobalt Software, a technology company based in Colombia.

My main background is in software architecture. I've worked on efficient software systems, distributed architectures, and local-first and offline-first applications.

What I enjoy most is understanding a system before building it. Not just what it should do, but how its pieces relate to each other, what information it needs, which rules determine its behavior, and what might be missing from the design.

I've spent a lot of time thinking about how to make complex software systems simpler, more explicit, and easier to verify.

Over time, those questions led me toward AI alignment.

I kept coming back to something that seemed important: a system can do exactly what it's designed to do, and do it very well, while still producing bad outcomes for people or entities its objectives don't account for.

My economics background also shapes how I think about this. Optimization, incentives, and welfare are closely related, but they're not interchangeable. Improving one objective doesn't automatically improve everyone's situation.

That's part of what motivated Kobalt Research/CUMPI, a broader effort to explore how AI systems represent affected entities, make decisions, interact with other agents, and potentially discover better outcomes through cooperation.

Iago is one of the first experiments I want to complete within that effort.

I'm working independently and will lead the research design, implementation, experiments, and analysis myself.

I don't have a long publication record in AI safety, and I don't want to pretend otherwise. My strongest experience is in designing and building software from carefully defined conceptual architectures.

I believe that experience, together with my economics background, is useful for this project. The experiment is small enough to build independently, but the underlying question deserves careful treatment.

My aim is to contribute something concrete to the alignment community: a clear question, a working experiment, and results that others can inspect and build on.

You can find the broader research ideas and conceptual foundations at https://kobaltsoftware.co. Our GitHub repository (https://github.com/jricardoduran/kobalt-research) is still pretty much a blank page — we're just getting started with the public code and experiments. The ideas came first; now we're working on making them testable, reproducible, and open for others to explore.

What are the most likely causes and outcomes if this project fails?

The biggest risk, I think, is designing an experiment that tells us exactly what we expected to hear.

For example, if I build an environment where excluding an entity's welfare automatically causes harm, then finding that harm doesn't teach us much.

I want to avoid that.

That's why I'll compare all four conditions and test different scenarios, including ones where more accurate information might help the affected entity, have no meaningful effect, or make things worse.

Another difficulty is measuring welfare.

In a simulation, it's easy to assign a number to an outcome. But that number only means something if we're clear about what it represents. I'll define the welfare measures explicitly, explain their limitations, and avoid treating a simple metric as if it captured everything that matters.

There's also the risk that the effect depends on very specific assumptions. If it disappears when those assumptions change, that matters. I'll document those cases rather than hide them.

The hypothesis itself could be wrong, at least in the environments I test.

That's a real possibility, and I'm comfortable with it. A negative result can be useful if the experiment was designed carefully enough to tell us something.

A more practical risk is trying to fit too many ideas into a small project. Kobalt Research/CUMPI covers a much broader set of questions, and it would be easy to let Iago grow beyond its budget and timeline.

So I'll keep the scope narrow: one central distinction, four experimental conditions, clear measures, and reproducible results.

I don't want to produce a persuasive story about AI alignment without a solid test behind it.

If things don't work out as expected, I still want to publish what was built, what failed, and what I learned. That might be just as useful for deciding what to test next.

How much money have you raised in the last 12 months, and from where?

I haven't received any external grants or research funding for Kobalt Research/CUMPI in the past 12 months.

So far, I've developed this work using my own resources, alongside the software development and architecture work I do at Kobalt Software.

I'm exploring funding opportunities for several small experiments within the broader research agenda. Each addresses a different question, and I intend to keep their work and budgets separate. None of those applications should be confused with funding already received.

That's why a relatively small grant like this would matter so much.

It would help me take an idea I've spent time thinking through and turn it into something real: a finished experiment, publicly available code, and results that other researchers can examine.

I hope Iago can become a useful first contribution to a larger effort. Not because it answers every question about alignment, but because it helps us ask one important question more precisely, and begin answering it with evidence.