Where AI Falls Apart

Hallucinations: Confident, Detailed, and Completely Wrong

In Lesson 1.1, you learned that AI predicts the next word based on patterns. Most of the time, this produces useful output. But sometimes, the prediction engine generates something that sounds perfectly reasonable and is entirely made up.

This is called a hallucination. And it's one of AI's most dangerous failure modes — not because the output looks wrong, but because it looks right.

Here's a real-world example: people have asked AI tools to recommend legal cases to support an argument. The AI responded with case names, court details, dates, and legal reasoning — all presented with total confidence. The problem? The cases didn't exist. The AI invented them because its pattern-matching predicted that "a legal case citation should look like this," and it generated one that fit the pattern perfectly.

Hallucinations happen because AI optimizes for what sounds plausible, not what's true. It has no way to check facts against reality. It has no internal sense of "I'm not sure about this." It just predicts — and sometimes those predictions land in fiction disguised as fact.

Bias: Garbage In, Garbage Out

Remember that AI learns from training data — text written by humans. And humans, as it turns out, have biases. Those biases end up baked into the training data, and the AI absorbs them as patterns.

If the training data contains more stories about doctors who are men, the AI learns a pattern: "doctor" associates with "he." If the data contains stereotypes about certain neighborhoods, the AI will reproduce those stereotypes in its outputs. It's not being intentionally unfair — it's just matching patterns it found in the data.

This is the "garbage in, garbage out" problem. The model is only as fair, accurate, and balanced as the data it was trained on. And since its training data comes from the internet and published text — which reflect centuries of human biases — the AI inherits those biases.

As a builder, this matters. If you're building something that makes recommendations, generates content, or interacts with people, you need to think about what biases might be hiding in the AI's outputs. We'll go deeper on this in Module 6, but keep it on your radar starting now.

Context Windows: AI Has a Short Memory

When you have a conversation with an AI tool, it might feel like the AI remembers everything you've said. But here's what's actually happening: every time you send a message, the AI receives your entire conversation so far as input and generates a new response. It doesn't "remember" — it re-reads.

The catch: there's a limit to how much text the AI can re-read at once. This is called the context window. Think of it like a desk that can only hold so many papers. Once the desk is full, the oldest papers fall off the edge. If your conversation gets long enough, the AI literally can't see what you said at the beginning anymore.

This has practical consequences for builders. If you're working on a complex project and your conversation with AI gets very long, it may start contradicting things it said earlier or forget instructions you gave it. Not because it's being difficult — because those earlier messages scrolled out of its context window.

Math and Reasoning: Where Prediction Hits a Wall

If AI is a pattern-matching machine, you'd expect it to struggle with tasks that require actual logic and precise calculation. And you'd be right.

AI can seem great at math when the problem is common enough to appear in its training data. Ask it "What's 15% of 200?" and it nails it — because that type of calculation appeared thousands of times in its training text. But give it an unusual multi-step problem, and it can confidently produce a wrong answer because it's predicting what a correct-looking answer should look like rather than actually computing.

The same goes for logical reasoning. AI can follow common reasoning patterns, but it can trip on basic logic puzzles that a human would solve easily. It might tell you something contradicts itself in the same paragraph without noticing. Why? Because it generates text one word at a time. It doesn't step back, re-read its own output, and think "wait, that doesn't make sense."

The takeaway for builders: Never trust AI math without double-checking. Never assume AI reasoning is logically sound. Use AI to draft, brainstorm, and accelerate your work — but when precision matters, verify everything.