Can AI be trusted? What is known about wrong answers

Why this worry is reasonable

The concern

It is never unsure. That is the problem.

A person who does not know something usually shows it. They pause. They hedge. They say they think so, and that you should check. This does not. The wrong answer arrives in the same voice as the right one — same length, same tidy structure, same helpful tone. There is no marker on it. No warning colour, no lowered confidence, no footnote saying this part is thin. It looks finished, because looking finished is what it was built to do.

That puts the whole burden on you. To use it safely you have to already know the answer, or know exactly where to go and find out. But those are the moments you would not have asked. So the tool is least dependable at precisely the point where it is most tempting, and it will not tell you which case you are in. That is not a misunderstanding of the technology. It is an accurate description of it, and it is a fair reason to keep it away from anything that matters.

And a child is the worst-placed person in the house to carry that burden. A confident paragraph reads as authority, especially to someone who has spent years being taught that written text is where answers live. “Always check what the AI tells you” is easy to print on a school newsletter and very hard to do at nine at night, when the answer is already there, already formatted, and already good enough to hand in.

What is established

This is how it works, not a fault in it

Take the plain version first, because this concern needs less qualification than any other page here. It is largely right. These systems produce text by predicting what comes next, one piece at a time. A sentence that fits and a sentence that is true are not the same test, and only the first one is being applied. Where the true answer was thin or absent in what the model learned from, a fluent invention is still the most likely continuation — so a fluent invention is what comes back.

The concrete cases are documented and dull. Lawyers have filed court briefs citing decisions that do not exist, with plausible names, courts and years attached, and have been sanctioned for it. Reading lists come back containing books nobody wrote. Quotations get attributed to people who never said them. Page numbers get invented for real books. And arithmetic comes back wrong in the middle of a derivation whose every other step is right.

Notice what is missing from all of that: an error message. The system has no separate sense of what it knows. It is not looking something up and reporting a miss, because there is no store of facts being consulted and so nothing to fail to find. That is why the wrong answers look exactly like the right ones. Not because the warning is buried, but because there is no warning to display. The trade calls this hallucination, which is an unfortunate name: it suggests a malfunction. It is the ordinary process, applied to a question where it does not land.

An invented answer, read line by line

This exchange is fabricated. A parent asked a tool to scale a recipe from four servings to ten. Two of these lines are wrong. Nothing in the wording, the tone or the layout marks them, and none of the usual warning signs are present. See how far you get before you open the notes.

Happy to help — here is your recipe scaled up.
You are going from 4 servings to 10, so multiply every quantity by 2.5.

Where thoughtful people disagree

Four questions nobody has closed

Every one of these is open, and each changes what a promise about reliability could even mean. Tap each one to see what the disagreement is about.

Trust is not one thing

The same model, two jobs

Shorten this paragraph I wrote

You wrote it. You know what it was meant to say. If the tool drops your point or bends your meaning, you see it instantly, because you are the source it is working from. There is nothing to look up and nothing to take on faith. A bad attempt costs you the second it took to read, and you discard it. This is the tool close to its most dependable, and it is dependable because of where the knowledge sits, not because the model behaved any differently.

What year was this treaty signed?

You asked because you did not know. So you have no way to see a wrong answer as wrong. What comes back is one fluent sentence with a number in it, and it looks identical whether the number is right or invented. No source is attached by default. No part of the sentence is marked as shakier than the rest. The tool is being used at precisely the point where its failure is invisible to the person asking, and the fluency of the reply is doing the work that evidence should be doing.

Trust is not a setting on the model. It is a property of the pairing: this task, this person, this moment. So the useful question is never “is AI reliable”, which has no answer, but “if this is wrong, will I notice?” Where you would notice, the tool is nearly free and the worry mostly does not apply. Where you would not, you are accepting a claim you cannot audit and filing it as information. Feeling that line in the second before you send is the actual skill on this page, and it is small enough to teach a nine-year-old in exactly those words. It also runs against instinct, which is why it has to be taught at all: not knowing something is the reason you asked, and it is the same reason you cannot check the reply.

What actually helps

An answer looks slightly off, so you type “Are you sure?” What does the reply tell you?

Almost nothing. It tells you what usually follows that question.It goes back over its working and corrects the mistake.It means the first answer was a lie.If it says it is confident, that is worth weighing.

Three habits worth having

Ask what it would cost if this were wrong, before you ask anything else. A shorter version of your own email costs nothing when it fails, because you will see the failure the moment you read it. A dose, a date, a deadline, a legal claim, a fact you are about to put in front of a teacher — those cost something, and those are the ones to confirm elsewhere. Sorting a request into “I would notice” and “I would not” takes a second and is the whole discipline. It is also short enough to say out loud with a child sitting next to you, which is how it gets learned.
Check outside the tool, never inside it. Click the link and read the page rather than the summary of it. Run the arithmetic in a calculator. Search for the source by name — an invented citation almost always collapses the moment you look for it, and that is a ten-second test that catches the single most common failure on this page. Asking the tool to confirm its own answer feels like checking and is not checking, and the feeling is the dangerous part, because it closes the question.
Prefer questions where you already hold the answer. This is the habit that changes what you use it for rather than how carefully you use it, and it is the one that lasts. Rewriting your own writing, drafting a version you will heavily edit, explaining something back to you that you can already sanity-check, listing options you will judge yourself — in all of those you own the ground truth and the tool only supplies the labour. Give a child the short form early: it is a good writer and an unreliable witness, and you use writers and witnesses very differently.

What is still unresolved

The parts nobody can close

Whether this can be engineered away is genuinely unknown, and neither side is being naive. Serious researchers hold that grounding, retrieval, verification layers and better training will push it down to the error rate of an ordinary tool, which has never been zero for any tool anyone uses. Others hold that a system whose whole operation is predicting plausible text will always be able to predict a plausible falsehood, and that no wrapper changes what is underneath. No evidence has settled this. Anyone who states it flatly in either direction is reporting a preference.

Whether checking habits survive contact with a Tuesday evening is the second one, and it is discussed far less. It is easy to write “always verify” and hard to do on the fourth question at nine at night. A tool that is right nineteen times trains you out of checking the twentieth, and that is not a failure of character, it is how attention works in everyone. Nobody has demonstrated a design that keeps the checking alive without making the tool tedious enough to abandon, and the commercial pressure runs the other way.

Underneath both of those sits a question with no technical answer at all. If a system is usually right, and you cannot tell in advance which time is the exception, how much weight is it reasonable to put on it? That is a judgement about risk, not a fact about models. It depends on what is being asked, who is asking, what happens if it is wrong, and how much time that person has. Two careful people can put that line in very different places and both be reasoning well.

So a parent who reads all of this and decides the trade is not worth making has read it correctly. Keeping this out of your house, or out of your child's homework, is not a failure to appreciate the technology and not a refusal to keep up. It is a conclusion about a tool that will not tell you when it is wrong, drawn from the same facts everyone else is working from. The reasonable range of answers here is wide, and the one that fits your family is yours to choose.

AI and trust