Capabilities and limits

A good model can explain code, follow examples, transform text, write programs, read images, and reason through problems it has never seen. It’s tempting to think of that as one score. This model is smart, that one is smarter.

In practice, capability is uneven. A model can solve a hard algorithm and then miss a simple requirement two paragraphs earlier. It can explain an API well and invent one parameter. It can refactor the happy path and forget what happens when the network fails.

Knowing and having access are different things

A model knows general facts from training. That doesn’t mean it knows today’s version of your framework, your production config, or what happened five minutes ago.

Training has a cutoff date. Anything after that date is missing. This bites developers all the time: you ask about a library, and the model confidently writes code for a version from a year ago. The API changed, the answer didn’t.

You have to do a little work up front. Tell the model where to find current information. In practice that means one of these:

• attach the docs for the exact version you use
• give the agent a tool that fetches current documentation, such as the Context7 MCP server
• install a skill for the framework you’re working with

We’ll cover tools, skills, and MCP later in the course. For now, remember the rule: don’t rely on training data for anything version-specific.

Tools help, but they don’t make the model infallible. It can call the wrong tool, misread the result, or stop too early. When you judge an answer, ask:

• Was the needed information in the context?
• Did the model use the right source or tool?
• Does the conclusion follow from the evidence?
• What important case did it not check?

Confidence is a writing style

Models are trained to write useful, coherent answers. When one says I am certain, that’s generated text. Not a reading from a truth detector.

You can ask a model to state its assumptions and what it’s unsure about. I do that often, because it shows me what to check. But the model’s confidence never replaces a source, a test, or an observed result.

Match the check to the claim

Different outputs need different checks:

• a factual claim: read the current primary source
• code: run it, test the expected case and the failure case
• a visual change: open the page at a couple of screen sizes
• a calculation: recompute a few known examples
• a consequential decision: keep a qualified human responsible

Treat every output as a candidate. The more an error would cost, the more evidence you should ask for before you accept it.