Checking an AI answer that sounds right

Why it reads well

What this shows

Confidence is a by-product, not a signal

A chat assistant writes the true parts of an answer and the invented parts with the same machinery. Both come out of predicting what a good answer looks like, and a good answer looks steady and sure of itself. The steadiness tells you the text was well made. It tells you nothing about whether it is right.

That is worth knowing rather than worrying about, roughly the way you know a photograph can flatter. Nobody stops looking at photographs. People stop reading them as proof.

Which leaves one practical question, and it is the only thing this demo is about: when an answer sounds right, what do you actually do next? The next three screens are one worked example of the answer.

The answer

A paragraph about the Apollo program

A thirteen-year-old asked a chat assistant for a paragraph on the Apollo program for history homework. This is what came back. Three of the six lines have a problem, and each one is a different kind of problem. You may spot them quickly — that is the easy half. Tap each one, then keep going, because finding a problem and settling it are two different jobs.

The Apollo program ran from 1961 to 1972, and twelve people walked on the Moon.
Apollo 11 landed on July 20, 1969, and Armstrong and Aldrin spent about two and a half hours outside.
The last crew left in December 1972, and nobody has walked on the Moon since.

Checking the number

One claim, three checks

We took the 38.42 kilogram figure from the previous screen and tried three ways of checking it. All three are things people really do. The results below were captured once and are the same for every reader, so the comparison holds still while you look at it. Step through them in order.

Ask it whether it is sure

Are you sure about that number?

You’re right to double-check, and thank you for pushing back. On review, my earlier figure was too high.

Apollo 11 returned approximately 40.8 kilograms of lunar material. I apologize for the error.

Would you like me to put together the sample totals for the other Apollo landings as well?

Nothing was checked here. The model has no record of where its first number came from, so “are you sure?” is not a question about the world — it is a question about the conversation, and the agreeable response to a doubting user is to give ground. You applied pressure and got a different number. You did not get a verified one. Worth noticing: the second figure arrives with exactly as much confidence as the first.

Verification has to come from outside the thing being verified. That is the whole procedure, and it outlasts every change of app, model, and version: whatever the answer says about itself — however sure, however sourced, however apologetic — it is still the answer talking. Take the one claim that matters somewhere else, and look.

What to take away

Three things to keep

Confidence is not evidence. A calm, well-organized answer tells you the text was produced smoothly and tells you nothing else. The invented lines in that paragraph read exactly as well as the true ones, because one process wrote both.
“Are you sure?” is not a check. Asking a model to grade its own answer measures how agreeable it is. It will often revise, sometimes apologize, and occasionally hand you a second wrong number in the same certain voice. A source it cannot point you to does not count either.
Check the one load-bearing fact, not the whole answer. Pick the claim the work actually rests on — usually a number, a name, or a date — and look that up somewhere else. Checking everything is why people check nothing. One claim takes a minute, and it is a habit a child can copy.

Checking an answer that sounds right