How Generative AI Writes: Prediction, Not Understanding

The short answer

Generative AI does not understand what it writes in the way a person does. What it actually does is predict which word is likely to come next, add it, and repeat. Because it has absorbed patterns from an enormous amount of text, the result usually reads as coherent.

Once you see that mechanism, the line between what it is good at (drafting, rephrasing, summarising) and what it is not (guaranteeing facts, knowing recent events, exact arithmetic) becomes much easier to draw.

Text is handled as tokens

The model does not work with text one letter at a time. It splits text into small units called tokens: whole words, parts of words, punctuation. A sentence like "Please send the invoice" might become "Please", " send", " the", " in", "voice" — roughly, depending on the model.

Where the splits fall differs between models, and in languages such as Japanese one character does not reliably equal one token. That is why pricing and "how much it can read at once" are so often stated in tokens: the model counts text in these units.

An LLM predicts the next word

At the centre of text-generating AI is a large language model (LLM). A useful way to picture it is a program that has read vast amounts of text and learned, in very fine detail, which words tend to follow which.

Given "Thank you for taking the time", a likely continuation is "to meet with us" or "to read this". The model adds the predicted token to the end, then predicts again from the longer text, and keeps going. Even a long answer is built from this loop.

The key point is that the prediction is based on whether a sequence is plausible, not whether it is true. That is why the output reads naturally — and also why it can read naturally while being wrong.

Training and inference are separate stages

There are two broad stages.

• Training: the model is exposed to huge amounts of text and learns the patterns of language. This takes substantial time and computing resources, and it is not something users normally see.
• Inference: the trained model is given a question or instruction and produces a response. Typing into a chat window and getting a reply is inference.

Training uses text up to a certain point in time, so anything after that is not part of the model's own knowledge. Ask about something announced yesterday and it may not know, or may answer from older information. Some services feed search results into the conversation; that adds information at inference time, but the model itself has not relearned anything.

There is a limit to how much it can read at once

Each inference can only take in a limited amount of text. That range is called the context window. Your instructions, any documents you paste, the conversation so far and the reply being written all have to fit inside it.

In a long chat, the model sometimes stops following a condition you set at the start. Older parts may have dropped out of range, or there may simply be so much text that the important part gets less attention. Window sizes differ by model and change often, so no figures are given here.

A few habits help in practice.

1. Restate important conditions partway through a long conversation.
2. Pass only the relevant part of a long document.
3. Start a fresh conversation when the topic changes.

Why confident mistakes (hallucinations) happen

Generative AI sometimes states things that are untrue in a completely convincing tone. This is called hallucination: a book title that does not exist, the name of a scheme nobody created, a figure that merely looks right.

The cause follows from everything above. Because the model generates a plausible continuation, when it does not know the answer it may produce a well-formed sentence rather than stopping. It is not trying to deceive; checking whether something is true is simply not part of the prediction step.

How this plays out at work

Suppose you need to tell staff that the expense-claim cut-off date is changing next month. Ask generative AI for a clear notice and it will produce a readable draft in seconds. Structure and phrasing are exactly what it does well.

But if the draft says "the cut-off is the 25th of each month", a person has to confirm that date matches what was actually decided. The model has no idea what your company decided; it may simply have filled in a date that looks typical. There are ways to have AI answer from your own documents, and RAG, covered in the next article, is the best-known of them.

Summary

• Generative AI does not understand meaning; it predicts and appends the next word.
• Text is processed in tokens, and prices and limits are often expressed in them.
• Events after training are not part of the model's own knowledge.
• The context window is limited, so conditions can slip in long conversations.
• Assume plausible errors will happen, and check facts and figures against the source.