Should AI Agents Have a Conscience?

 

Nir Eyal’s Hooked made a huge impression on me.

It shaped a large part of how I think about product creation. Not because I wanted to “hook” users at any cost, but because the book made one thing very clear: if you understand how to influence behaviour, you also become responsible for how you use that power.

That responsibility is easy to ignore.

A product person, designer or founder can hide behind the usual excuses: we are just improving engagement, we are just reducing friction, we are just helping users build a habit, we are just optimising the funnel.

But those words can cover very different realities.

Helping someone build a useful habit is one thing. Making it harder for them to leave is another. Helping someone learn a language, exercise more or manage their finances better is one thing. Building a dopamine machine that eats their evening, attention and self-control is another.

This is why the ethics topic in Hooked mattered so much. Nir Eyal did not just describe a model for creating habit-forming products. He also pushed product creators to ask uncomfortable questions.

• Are you genuinely making people’s lives better?
• Are you solving a real problem?
• Would you use the product yourself?
• How could this feature be misused?
• What happens if it works too well?

Some teams turn this into a game. Before shipping something, they ask: what is the worst possible way this could be used?

It sounds dramatic, but it is a useful question.

Because most harmful products do not start with someone saying: “Let’s build something evil.”

They start with something more innocent.

• Let’s improve retention.
• Let’s increase daily active users.
• Let’s add one more notification.
• Let’s make the reward more exciting.
• Let’s make the next action obvious.
• Let’s personalise the feed a bit more.

And suddenly you are not solving the user’s problem anymore. You are solving your own growth problem by exploiting the user’s weakness.

Now Add AI Agents

Lately, I see more people using AI agents to create content, give and take advice, write code, design interfaces, analyse markets and even build complete digital products.

This is powerful.

A solo founder can now do what previously required a small team. A product manager can prototype an idea without waiting for engineering capacity. A designer can generate flows, copy and variants in minutes. A non-engineer can vibe-code a game, a tool or a landing page.

I love that.

But there is an uncomfortable question hiding underneath all this excitement:

Are we baking ethics into AI agents?

And I do not mean safety in the narrow sense.

It is obviously good that AI systems refuse to help someone build a bomb, create malware or design a new virus. That kind of guardrail is necessary. It is also relatively easy to understand. The request is clearly dangerous, so the system should refuse.

But product ethics is rarely that clean.

What if I ask an agent to build a more addictive version of TikTok?

What if I ask it to design a casino game that keeps vulnerable people playing for longer?

What if I ask it to optimise a children’s game for maximum screen time?

What if I ask it to create a subscription flow where cancellation is technically possible but emotionally exhausting?

What if I ask it to write push notifications that exploit loneliness, insecurity or fear of missing out?

None of these requests sounds like “build a bomb.”

But they can still create harm.

And this is where the current generation of AI agents becomes interesting. They are extremely good at execution. Give them a goal, and they will often help you reach it. Increase conversion. Improve retention. Make the game more engaging. Make users come back more often. Make the onboarding more persuasive. Make the paywall harder to ignore.

The agent may not know whether your goal is good.

It may not understand the human cost of achieving it.

It may simply optimise.

That is both the promise and the danger.

Ethics Are Not the Same as Safety

We need to separate two ideas that are often mixed together: safety and ethics.

Safety says: do not help users do clearly dangerous things.

Ethics asks a much harder question: should this thing exist in this form at all?

A safe AI system might refuse to explain how to make a weapon. Good.

But the same system might happily help you design a streak mechanic that makes teenagers anxious about missing a day. It might help you write loss-chasing copy for a gambling product. It might help you create dark patterns that technically follow the law but still manipulate people into doing things they would not otherwise choose.

That is not a safety failure in the traditional sense.

It is an ethics failure.

And those are harder to detect because they live in the grey zone. They depend on context, intent, audience, business model and consequences.

• A notification can be helpful.
• A notification can also be manipulative.
• A streak can motivate.
• A streak can also create anxiety.
• A recommendation feed can help discovery.
• A recommendation feed can also trap people in an endless loop.
• A game can be fun.
• A game can also become an extraction machine.

The difference is not always in the feature itself. It is often in the intent behind it and the system around it.

Should Agents Push Back?

I think they should.

Not always. Not in an annoying, moralising, bureaucratic way. Nobody wants an agent that turns every product discussion into a philosophy seminar.

But agents should be able to recognise ethically dubious goals and push the creator to think.

• If I ask an agent to improve retention for a meditation app, it could help.
• If I ask it to improve retention by increasing user anxiety when they miss a session, it should challenge me.
• If I ask it to make a game more fun, it could help.
• If I ask it to make a game more addictive for children, it should push back hard.
• If I ask it to improve a subscription flow, it could help.
• If I ask it to hide cancellation, guilt-trip users or make opt-out confusing, it should refuse or redirect me towards a fairer design.

This does not mean the AI agent becomes the ultimate moral authority.

That would be ridiculous.

The responsibility still sits with the human. The human defines the goal. The human ships the product. The human takes the money. The human owns the consequences.

But agents are becoming part of the product creation process. They are not just autocomplete anymore. They plan, write, design, code, test and sometimes make decisions across multiple steps.

So they should not be ethically blind.

The “Worst Way It Could Be Used” Test

One simple improvement would be to make agents better at asking the question good product teams should already ask:

What is the worst way this could be used?

Before building a feature, an agent could help run a quick abuse and harm review.

• Who could be harmed by this?
• What user weakness does this exploit?
• Could this create addiction, anxiety, financial loss or social pressure?
• Could this be especially harmful to children, vulnerable users or people in distress?
• Does this feature make it easier for users to act in their own interest, or harder?
• Are we giving users control, or taking it away?
• Are we solving a real problem, or manufacturing dependency?

This does not need to become a massive compliance process. Most indie builders, PMs and founders will not run a formal ethics review for every feature. But an agent can make this lightweight.

It can be as simple as:

“You are asking me to optimise for retention. This could be fine, but some tactics may become manipulative. Do you want me to optimise for healthy retention instead?”

Or:

“This mechanic may increase engagement by exploiting loss aversion. A less harmful alternative would be to reward positive return behaviour without punishing absence.”

That small nudge could matter.

Because when you are building quickly, especially with AI, speed becomes dangerous. You can go from idea to implementation before you have properly thought about the consequences.

AI agents make execution cheaper.

That means ethical reflection needs to become cheaper too.

The Agent Should Not Replace Your Conscience

I do not believe AI agents should block every questionable product idea.

There are too many edge cases. Too many legitimate businesses live close to sensitive territory: gaming, finance, social media, dating, health, education, productivity, news, advertising.

• A gambling product can be legal.
• A social media product can be useful.
• A game can include rewards without becoming exploitative.
• A fitness app can motivate without shaming.
• A productivity app can encourage focus without turning work into self-harm.

The point is not to ban everything that can influence behaviour. All products influence behaviour. That is what products do.

The point is to make the influence visible.

A good AI agent should not just ask: “Can I build this?”

It should also ask: “What happens if this works?”

That is the question product creators often avoid.

If your product works, will people become healthier, calmer, richer, smarter, safer, more capable?

Or just more dependent?

Will they thank you later?

Or will they feel tricked?

Will they use your product because it helps them achieve something they care about?

Or because you learned how to press the right psychological buttons?

We Need Better Defaults

The future I would like to see is not one where AI agents constantly refuse to help.

It is one where they guide us towards better defaults.

• Healthy retention over addictive retention.
• Transparent choice over dark patterns.
• User agency over manipulation.
• Long-term trust over short-term conversion.
• Useful habits over compulsive loops.
• Fair monetisation over exploitation.

This would not solve everything. Bad actors will still try to bypass guardrails. Companies will still chase growth. Some founders will still choose extraction over value.

But norms matter.

Tools shape behaviour. If the default behaviour of AI agents is to help creators think through harm, more people will do it. Not everyone. But more.

And that is worth having.

Because the next wave of digital products will not be built only by large companies with legal teams, research teams and design ethics playbooks. They will be built by solo founders, product managers, designers, teenagers, small teams and anyone with an idea and access to an agent.

That is exciting.

It is also risky.

We are giving more people the power to create software that can influence behaviour at scale. That power should come with better questions.

Before we ask an AI agent to optimise the product, maybe we should ask it to stress-test the product.

Before we ask it to increase engagement, maybe we should ask whether that engagement is healthy.

Before we ask it to make something habit-forming, maybe we should ask whether the habit is worth forming.

The ultimate responsibility is still ours.

But if AI agents are going to help us build the future, they should also help us avoid building the worst version of it.