A Working Vocabulary: Tokens, Context Windows, and Temperature Explained for Data Teams

Most of the confusion I see on data teams new to language models traces back to three terms used loosely. Tokens, context windows, and temperature are not jargon for its own sake. Each one directly controls something you care about: what you pay, what the model can see, and whether you get the same answer twice. Get these three right and most of the mystery drains out of working with these tools.

Tokens: the unit of everything

A token is the chunk of text a model actually reads and writes. It is not a word and not a character. It sits in between. As a rough rule for English, one token is about four characters, and 100 tokens is roughly 75 words. The word "revenue" might be one token; an unusual product SKU or a long numeric ID might split into several. Whitespace and punctuation count too.

Why an analyst should care: tokens are the billing unit and the capacity unit. Every API priced per million tokens charges you for input tokens (your prompt, including all that schema context and those retrieved documents) and output tokens (what the model generates), usually at different rates, with output typically costing more. When someone asks why the RAG pipeline's bill jumped, the answer is almost always that the retrieval step started stuffing more documents into each prompt, inflating input tokens on every call.

Tokenization also explains a class of odd behavior. Models are famously shaky at character-level tasks, counting letters, reversing strings, precise arithmetic on long numbers, because they never see characters. They see tokens. A long account number split into three tokens is not a number to the model; it is three fragments. If your workflow depends on exact manipulation of IDs or digit-level precision, do that in SQL or Python, not in the model.

Practical habit: estimate tokens before you build. If you plan to send a 50-column schema plus ten retrieved records plus the user question on every query, add it up. Most providers publish tokenizers or token-counting endpoints. Knowing that a single call is 8,000 input tokens rather than 2,000 changes both your cost projection and your architecture.

Context window: the model's working memory

The context window is the maximum number of tokens the model can consider at once, input and output combined. In 2026 this ranges widely, from tens of thousands of tokens up to a million or more on the large-context models. It is the hard ceiling on how much you can show the model in a single call.

The mistake is treating the context window as free space to fill. It is not, for two reasons. First, you pay for every input token on every call, so a large window used carelessly is a large recurring bill. Second, and less obvious, models do not attend to a full window evenly. Information in the middle of a very long context is more likely to be overlooked than information at the start or end, a pattern often called "lost in the middle." Dumping your entire data dictionary into a million-token window and hoping the model finds the one relevant table is worse than retrieving the three tables that matter and presenting them cleanly.

This is why retrieval matters even when the window is huge. The context window sets what is possible; good retrieval and prompt construction set what actually works. For a data team, the practical implications are concrete. If you are analyzing a document longer than the window, you must chunk it and process pieces, then combine results. If a chatbot "forgets" what was said earlier in a long session, the conversation has exceeded the window and the oldest turns fell out. And when accuracy degrades as you add more context, the fix is usually less context, better chosen, not more.

One more distinction worth holding: the context window is not memory across sessions. The model retains nothing between separate API calls unless you resend it. Every call starts blank. Anything the model "remembers" is text your application chose to include again.

Temperature: the reproducibility dial

Temperature controls randomness in how the model picks its next token. At each step the model produces a probability distribution over possible next tokens. Temperature reshapes that distribution before a choice is made. Low temperature, near zero, makes the model almost always pick the highest-probability token, giving focused, consistent, largely repeatable output. Higher temperature flattens the distribution so lower-probability tokens get chosen more often, giving more varied and creative output.

For most analytics work, you want temperature low. If you are generating SQL, extracting structured fields from documents, classifying tickets, or anything where you want the same input to yield the same output, set it at or near zero. There is rarely an upside to creativity when you are asking the model to produce a JSON object matching a schema, and there is real downside: a query that varies run to run is a query you cannot trust or test.

Raise temperature deliberately, and only when variety is the goal, brainstorming hypotheses about why a metric moved, drafting several phrasings of a report summary, generating diverse test inputs. Even then, a moderate setting usually beats a high one, which tends toward incoherence.

Two honest caveats. First, low temperature reduces variability but does not guarantee identical output, because other factors in serving infrastructure can introduce small differences; if you need strict reproducibility, some providers expose a seed parameter, and you should still log inputs and outputs rather than assume determinism. Second, temperature does not make the model more accurate. A confidently wrong answer at temperature zero is still wrong. Lowering it makes the model consistent, not correct. Those are different properties, and conflating them is how teams end up trusting a repeatable mistake.

How the three connect

These are not separate settings; they interact in every call you make. Your prompt plus context consumes input tokens, bounded by the context window and billed per token. The model generates output tokens, and temperature governs how deterministic that generation is. A well-run pipeline keeps input tokens lean through good retrieval, stays comfortably inside the window with room for the response, and pins temperature low for anything that feeds a report. Master those three levers and you can reason about cost, reliability, and capacity before you write a line of integration code, which is exactly the position a data team wants to be in.