Module 09 · AI Fundamentals Lab
If it sounds right, is it right?
You will read six confident, fluent, professional-looking answers about cybersecurity and hunt for the claims that are simply made up. Then you will see why a model produces exactly that kind of error — and what actually reduces it.
Spot the fabrication
Each answer below mixes true statements with at least one invented one. Tap every sentence you think is made up, then check. Nothing here is live model output — every card is a written example, composed for this exercise so the answers can be marked with certainty.
Why a model makes things up
Nothing goes wrong when a model fabricates. It is doing the only thing it was built to do.
Three consequences of that loop
There is no lookup table
The model is not reading a database of facts and copying an answer out. Facts, when it has them, are smeared across billions of weights as statistical tendencies. Nothing is stored as a row you could go and check.
There is no “I do not know” state
At every step there is always a most-likely next token. The model never runs out of continuations, so silence is never the natural output. Saying “I am not sure” is itself just a sequence of likely words — a style it can imitate, not a measurement of its own knowledge.
Rare facts have thin, noisy signal
A common fact appears a million times in training and the pattern is strong. One obscure CVE number may appear a handful of times. The shape of the answer is still learned perfectly — CVE-YYYY-NNNNN, four digits, a dash. So the model produces a perfect-looking CVE number with invented digits.
The misconception worth killing now
“It is a bug. A future version will fix it.”
It is not a bug. Predicting a plausible continuation is the whole mechanism, and a plausible continuation is sometimes false. Newer models fabricate less often and hedge better, and grounding the model in real documents (module 08) cuts it further — but no released system has removed it, because removing it would mean removing prediction. Treat it as a permanent property to manage, like network latency, not a defect waiting on a patch.
What actually reduces it
Five things that measurably help, in rough order of how much they buy you. None of them is a fix.
-
Ground the answer in documents you control
Retrieval (module 08) puts the real text in front of the model and asks it to answer from that. The answer stops being recalled and starts being read. Insist on citations you can click through to the source passage — a citation you cannot open is just more fluent text. See module 08 →
-
Ask for sources, then actually check them
Asking for sources helps a little on its own, because it pushes the model toward material it has actually seen. It helps enormously if you open them. A fabricated citation dies the moment someone searches for it, and survives forever if nobody does.
-
Constrain the task
“Summarise this advisory I pasted” is far safer than “tell me about that advisory”. Transforming text you supplied — summarise, rewrite, translate, extract, classify — is low risk. Recalling a specific fact from memory is high risk. Move work into the first category whenever you can.
-
Ask twice and compare
Ask the same factual question in a fresh conversation. A fact the model holds firmly comes back the same. A thin, noisy one wobbles — a different CVE number, a different port, a different date. Instability is evidence of fabrication. Stability is not evidence of truth: a confidently wrong fact repeats just fine.
-
Keep a human on anything that has consequences
Lower the stakes rather than raise the trust. Draft with a model, decide with a person. Never let unverified output reach a firewall rule, a customer, a regulator, a court filing or a patch window.
Red flags — verify before you use it
A keeper. Screenshot this. If an answer trips any of these, it needs a second source before it goes anywhere.
- A specific number Port, key length, timeout, version, price, percentage, date. Specificity is cheap for a model to produce and expensive for you to be wrong about.
- A named citation CVE, RFC, standard clause, case name, paper, page number, URL. Open it. If it will not open, it does not exist.
- Anything recent Events, versions, prices, staff, org charts. The model's knowledge stops at a date, and it will answer past it without saying so.
- Private or internal detail How a company works inside, what a team decided, what is in a non-public system. There was no source. A confident answer here is invented by definition.
- Two similar tools in one answer Features get merged. Check which product actually has the menu, flag or tab being described.
- “Best practice is…” Guidance changes. Advice that was standard a decade ago is still all over the training data and comes back stated in the present tense.
- A tidy causal claim “Because we have X, Y cannot happen.” Neat causation reads well and is often the invented part of an otherwise sound answer.
- Anything you would act on The real trigger. If you would change a config, send it to a client, quote it to a regulator or spend money on it, verify it — regardless of how the answer sounds.
Guided experiments
Three short ones. Pick an answer, then reveal. Scenarios are written examples, not transcripts of a live model.
A model is optimised to sound right, not to be right — so verification is your job, not its.
Fluency and confidence tell you the prediction was easy. They tell you nothing at all about whether it was true.
My observations
Kept in this browser tab only, for this session.