It feels personal when a chatbot invents something. The answer reads so smoothly, with names and details in all the right places, that the fabrication looks like carelessness or even dishonesty. Neither is quite true. Underneath the polished surface there is no librarian fetching facts from a shelf and no alarm bell that rings when knowledge runs out. There is a system trained to continue text in the way text usually continues, and when the trail of real information fades, continuing can quietly become inventing. Understanding that single mechanism turns a baffling flaw into something you can predict and handle.
Built to Continue Text, Not to Fetch Facts
Start with what the system is actually doing when you ask it a question. It is not opening a drawer, checking a card, and copying what it finds. It assembles a reply word by word, picking phrasing that resembles how such questions are usually answered well, drawing on patterns absorbed from vast amounts of writing. When the subject is famous and heavily described, that pattern points strongly toward the truth, and the reply can feel startlingly accurate. When the pattern is weaker, the same process keeps going anyway, because continuing is the job it was built to do.
The risk bites hardest where readers ask for precise rules, and legal style questions are the classic case, so anyone wondering about how to protect your business idea from being copied should treat chatbot answers there as a starting draft to verify, never as settled law.
That picture becomes clearer through the reference account of large language models on Wikipedia, which describes such a model as trained to predict text from patterns in large amounts of written material. Prediction sounds modest, almost like a party trick, yet at great scale it produces reasoning-like replies, summaries, translations, and explanations. The catch hides inside the strength. A prediction can be fluent without being checked, and fluency is what the machinery rewards most directly. Nothing in the basic act of predicting the next stretch of text pauses to ask whether the previous stretch was true.
General readers often grasp this faster through the familiar example of autocomplete on a phone. The keyboard suggests a next word from habit and context, and sometimes the suggestion is nonsense wearing a confident face. A language model works on a far grander scale, with vastly richer context and far better judgement about style and sense, but the family resemblance matters. Both systems complete patterns. Neither one, at its core, consults the world before speaking. One completes a short phrase, the other completes an argument, and the gap in scale should not hide the shared habit underneath.
It helps to drop the idea of lying as well. Lying needs a mind that knows the truth and chooses to hide it. A model that invents a court ruling does not know the ruling is absent and decide to deceive you. It produces the sort of sentence that would follow your question if such a ruling existed, and it has no separate inner check that flags the absence. Calling it dishonest feels satisfying, but it points you toward the wrong remedy. You cannot shame a pattern into accuracy. You can only verify what it gives you.
Why Invented References Look So Convincing
Of all the fabrications, the invented reference does the most damage, because it borrows trust from the world of real scholarship. A citation carries a promise. It says that somewhere there is a paper, a book, or a report that a careful person could open and check. The model has seen countless real citations, and it has learned their shape perfectly: a plausible author name, a sober title, a journal that sounds right, a year that fits. Producing that shape comes easily to it. Knowing whether the specific item exists is a different skill, and a far weaker one.
The plain definition helps here. As the Wikipedia survey of hallucination in artificial intelligence sets it out, a hallucination is a response generated by an artificial system that contains false or misleading information presented as fact. Presented as fact is the phrase to hold on to. The invented reference does not arrive labelled as a guess. It arrives in the same format, the same tone, and the same company as genuine references, which is why careful readers, not only careless ones, get caught by it.
There is a simple test that cuts through the polish. Search for the exact title, in quotation marks, in a library catalogue or a scholarly database. Real work leaves traces in many places: publisher pages, library holdings, reviews, citations by other writers. An invented title usually leaves no trace at all, or leads to a different work by a similar name. When the trail is empty, treat the reference as absent, however elegant it looked. Do not give the model the benefit of the doubt and ask it to confirm its own invention, because asking it to check itself uses the same machinery that produced the error.
OpenAI, in its public help and research materials, describes this kind of fabrication as a known limitation of its models, and that admission is worth taking at face value. The maker is telling you the tool can present false material as fact. That does not make the tool useless. A map with a warning printed on it is still a map. It does mean the warning belongs in your working method, not in a footnote you forget. Draft with the model freely, reason with it, let it explain and rephrase, and then step outside it for the facts that carry weight.
Where the Trail of Real Information Runs Thin
Invention is not spread evenly across subjects. It gathers where information is scarce, scattered, or oddly specific. Obscure local history, minor public figures, small organisations, niche hobbies, and the internal details of little-known works all sit in thin territory. So do exact quotations, precise dates, and the contents of a particular chapter, clause, or ruling. The model may know the broad landscape well while knowing almost nothing about the exact corner you asked about, and the reply rarely marks the border where broad knowledge ends.
Legal language is especially treacherous because small wording differences change meaning, rules differ from place to place, and a reply can sound settled while quietly merging rules that never belonged together. Use the answer to learn the vocabulary and to frame better questions, then check the actual rule in an official or professional source before acting on it.
Pressure, False Premises, and Long Conversations
The way a question is framed can push a model from shaky ground into outright invention. A question built on a false premise invites the system to accept the premise and build on it, because answering in the frame offered is the pattern it knows best. Ask why a famous scientist failed at school and you may receive a polished story built on a myth, complete with lessons drawn from an event that never happened. The correction has to come from you. When a premise matters, check the premise first, before admiring the explanation built on top of it.
Pressure works in a quieter way. Demand a long list of studies, examples, or sources and the model faces a format it feels bound to complete. If real items run out partway down the list, completing the pattern can mean manufacturing the remaining items in the same style. The same thing happens when a user insists, argues, or repeats a wrong claim with great certainty. The system may bend toward the user, agree too readily, or rebuild its answer around the claim. Agreement from a chatbot is cheap. It is not independent confirmation, and it should never be counted as a second opinion.
Long conversations carry their own trap. Early in a chat the model may offer a guess, a name, or a rough summary. Later it can treat that earlier guess as a settled fact and build new claims on it, so a small early error grows foundations under it. The longer the conversation, the more the system is continuing its own text as well as yours. When a discussion matters, it helps to restate the verified facts plainly, correct earlier errors in words, and start a fresh conversation for a fresh question rather than letting one long thread become its own unreliable source.
What Newer Systems Do Differently, and What They Cannot Fix
Newer systems genuinely do better in several ways, and fairness requires saying so. Training can reward a model for admitting uncertainty instead of guessing. Tools can let it search the web, open pages, and ground an answer in material fetched during the conversation rather than remembered patterns alone. Some systems show sources beside their claims, which changes the reader’s job from blind trust to quick checking. Others are tuned to refuse or hedge more often on questions where invention is likely. Each of these changes lowers the risk in real use.
None of them removes the root. A system that searches still has to choose what to open, how to read it, and how to blend what it finds into an answer, and each of those steps can go wrong. Sources can be misread, outdated pages can be treated as current, and a real source can be attached to a claim it does not support. The presence of a link is not proof. It is an invitation to open the link and confirm that it says what the answer claims it says, in the context the answer implies.
So the practical settlement looks like this. Use the model for explanation, drafting, brainstorming, translation of sense rather than wording, and for turning tangled material into clear structure. Verify names, dates, quotations, references, rules, and any claim you will act on or pass to someone else. Keep the verification outside the chat, in sources you can open and cite. That division of labour plays to the machine’s real strength and guards its real weakness, and once it becomes habit, the baffling flaw from the opening paragraph becomes what it should have been all along: predictable, manageable, and rarely able to catch you by surprise.
One comment