Cognitive debt: when nobody understands the system AI wrote

It has never been easier to turn an idea into a product. Over a weekend, with an AI and a few prompts, you can ship a SaaS with login, payments and a dashboard. It works. The first customers show up.

Then the business does what every healthy business does: it changes. A customer asks for a different plan, the billing rule needs an exception, a new law changes what you're allowed to store. And the question no prompt answered shows up: where do I change this without breaking everything else?

The gap between those two things has a name: cognitive debt.

The debate is asking the wrong question

The conversation about AI in software development tends to get stuck between two extremes.

On one side, speed. How many features shipped, how many releases per week, how much time was saved. AI delivers, so the market measures what it delivers: quantity.

On the other side, distrust. "Every line the AI wrote has to be reviewed." It sounds prudent, but it cancels out the gain and doesn't scale. Nobody is going to carefully read tens of thousands of generated lines a month.

Both sides are looking at the code. And the code, honestly, is no longer the problem. For most of the work, AI is the one writing it, and I don't think that's a bad thing.

The question that matters is a different one: do you have a way to know whether what was built is right, and how you'll change it tomorrow?

Every healthy business changes

Software doesn't exist for its own sake. It exists to support a business, and a business that doesn't change has probably stopped growing. Rules change because the market changes: competitors, customers, laws, prices.

That's why architecture, design patterns and decoupling exist. They aren't decoration, and they aren't big-company bureaucracy. They are ways to make change cheaper.

In The Software Architect Elevator, Gregor Hohpe compares good architecture decisions to financial options: you pay a little up front for the freedom to change later, and that freedom is worth the most precisely when the future is uncertain. Decoupled parts can be swapped without bringing down the rest. Well-separated rules can change without side effects.

In the end, the goal is financial. Architecture decides whether the next change costs an afternoon or a month.

Fear of change comes from lack of familiarity

Hohpe also talks about something every developer has felt: the fear of touching the code.

Nobody is afraid of changing what they understand. The fear shows up when you don't know what depends on that piece, why it was written that way and what might break somewhere else.

Before AI, that lack of familiarity built up over time: the legacy system, the person who left the company, the module "nobody touches". Now it can exist from day one. It's code nobody wrote, nobody read carefully and nobody had to understand to put it in production.

That was lack of familiarity accumulated over years. With AI, you can accumulate the same thing in weeks.

Technical debt lives in the code. Cognitive debt lives in people's heads: it's the gap between what the system does and what the team actually understands about it.

The term gained traction in 2025 with an MIT Media Lab study of people writing essays with help from ChatGPT. Those who used AI showed less brain engagement and remembered less of their own text than those who wrote without help. The study is about essays, not software, and it's still being debated. But the idea carries over: the effort you skip today becomes the understanding you lack tomorrow.

Context is what makes change possible

The problem isn't AI writing the code. The problem is asking for a magic solution with no criteria, no documentation and no context, and then expecting to be able to change the result.

In practice, context means:

• Written business rules: in a PRD (product requirements document), with the problem the system solves, what it must guarantee, the exceptions and the business vocabulary.
• Recorded decisions: ADRs (architecture decision records), proposed by Michael Nygard in 2011, capture what was decided, why, and which alternatives were ruled out.
• Team rules: the technologies used and avoided, patterns, project structure and what AI may or may not do. These are the guardrails.
• Validation criteria: tests that express business rules and automated checks that tell you whether something still holds.

That context serves people and AI at the same time. With it, AI stops guessing: it knows the rules, the decisions and the limits. Without it, every request starts from scratch, and every answer can contradict the previous one.

For decades, documentation was the good practice that lost to the deadline. Writing and maintaining it was expensive, and nobody had the time. Today AI writes, updates and checks that documentation alongside us.

That's why the question should shift from "how do we deliver faster" to "how do we deliver better". Speed and quality aren't opposites, but the priority has to be quality. Speed comes as a bonus when every change stops being a gamble.

When this doesn't apply

Not everything needs this rigor. A prototype built over a weekend to test an idea doesn't need ADRs or documented rules. If the idea doesn't hold up, it all goes in the trash, and that's fine.

The risk is the prototype becoming the product without anyone noticing. The day someone pays for the system, it stops being an experiment, and the debt starts to count.

It's also worth being clear about what I'm not arguing for: validating doesn't mean reading all the code. Validating means having ways to know whether the result respects the rules. That comes from tests, clear criteria and reviews focused on what's critical.

In practice: 10%, 80%, 10%

A simple way to think about working with AI:

• 10% before, human: understand the business and write the context, meaning the rule, the constraint and the decision. If you can't write the rule down, AI won't get it right either.
• 80% in the middle, AI: the execution, within the guardrails you defined.
• 10% after, human: validate against the criteria set at the start and update the documentation with what changed.

A few ways to start:

1. Write a PRD with the system's business rules, and make the team and the AI read it before any change.
2. Record as an ADR every decision that will one day raise the question "why was it done this way?".
3. Turn the most important business rules into tests.
4. Ask the AI to explain what it did and compare the explanation with the documentation. If they don't match, something is wrong: in the code or in the document.

What AI didn't solve

AI solved the writing part. It didn't solve the understanding part, and understanding is what lets you change things.

Generating code got cheap. Changing what nobody understands is still expensive.