How Large Language Models Work: From Tokenization and Embeddings to Transformer Token-by-Token Generation

How large language models work depends on massive-data pretraining, prompt tokenization, embedding vectors, and Transformer processing. They generate content token by token, making them suitable for understanding text generation, translation, code generation mechanisms, and the basic principles of natural language processing tasks.

What Is a Large Language Model?

A large language model, also commonly written as LLM, is an extremely large deep learning language model trained or pretrained on large amounts of data. By learning the language structures, expressions, and content patterns in massive amounts of text, it can identify, interpret, and generate text.

Conceptually, a large language model can be understood both as a type of statistical language model and as something usually based on a deep learning architecture. The former emphasizes that it predicts subsequent content based on learned language patterns; the latter emphasizes that it relies on large-scale neural networks to complete representation, computation, and generation.

Key point: A large language model does not simply copy materials. After receiving a prompt, it gradually generates subsequent content based on patterns learned inside the model.

Understanding How Large Language Models Work with a Simple Analogy

A large language model can be understood as a language prediction system trained on massive corpora. It does not merely retrieve a ready-made answer according to fixed rules. Instead, it continues to predict the next token that is more likely to appear based on the user’s prompt, the context, and the content already generated.

Here, “prediction” does not mean truly understanding the world. More precisely, a large language model can identify, interpret, and generate text at the language level, and output text, code blocks, or other content driven by prompts.

For example, when a user enters “Please summarize this passage,” the model first processes the input content and then gradually generates a summary based on the context; when a user enters “Write a function example,” the model can also generate a code block based on the prompt.

The Training Foundation of Large Language Models Includes Massive Data and Unsupervised Learning

The reason large language models can handle a wide range of language tasks lies in large-scale data training. The model learns common grammatical structures, word combinations, expression styles, and content patterns from massive datasets, thereby gaining relatively general language capabilities.

Massive Data Lets the Model Learn Language Patterns

Through training on large amounts of data, the model can encounter different topics, different writing styles, and different language phenomena, and gradually form statistical representations of text structures. These representations are used in later tasks for text identification, interpretation, and generation.

Unsupervised Learning Reduces Dependence on Manual Labeling

Large language models can use unsupervised learning. The basic meaning of unsupervised learning is that the model can use unlabeled datasets to discover previously unknown patterns in data, without necessarily relying on every piece of data being manually labeled.

Pretraining Provides General Language Capabilities

Pretraining enables the model to acquire relatively general language processing capabilities before entering specific tasks. After that, when the model faces prompts such as writing, translation, explanation, or generating code blocks, it can generate corresponding outputs based on the language patterns it has already learned.

The Generation Workflow from Prompt to Answer Can Be Divided into Four Steps

After receiving a prompt during inference, a large language model usually goes through a computational process from text input to token-by-token output. To make it easier to understand, this process can be divided into four steps.

Step 1: Receive the Prompt and Split It into Tokens

The prompt entered by the user is first processed by the model into smaller units, which are usually called tokens. Tokens can be understood as the basic fragments the model actually uses when processing text.

Step 2: Convert Tokens into Embedding Vectors

Tokens themselves cannot be used directly for neural network computation, so they are converted into embedding vectors. An embedding vector is a numerical representation that enables the model to process tokens and their relationships in a computational space.

Step 3: Process the Context Through a Transformer

Next, the model uses a Transformer structure to process these representations. The Transformer combines contextual information and uses self-attention-related capabilities to model the relationships between tokens, thereby determining which parts of the current prompt are more important.

Step 4: Generate Output Token by Token

The model usually does not generate a complete answer all at once, but generates text token by token. Each time it generates a token, the model continues computing based on the prompt and the existing output until it forms complete text, a code block, or other content.

Workflow stageInput or intermediate resultMain function
Prompt inputText entered by the userClarify the task, question, or context
TokenizationTokensSplit text into units the model can process
EmbeddingEmbedding vectorsConvert tokens into computable representations
Transformer processingContextual representationsModel token relationships and contextual information
Token-by-token generationText or code blocksGenerate the final output content

The Role of the Transformer Structure in Large Language Models

Transformer, often corresponding to the English term Transformer, is one of the important underlying structures of many large language models. It helps the model process the input context and supports the model in generating output token by token based on the context.

The Position of Encoders, Decoders, and Self-Attention

In a high-level workflow, a Transformer can include encoder and decoder neural networks with self-attention functionality. Encoders, decoders, and self-attention are not isolated components. They jointly serve one goal: enabling the model to use contextual information when processing token sequences.

Self-attention can help the model focus on different positions in the input and model relationships between tokens. This article only explains the mechanism at the workflow level and does not expand into complex mathematical derivations.

Why Transformers Are Suitable for Language Generation

Language generation requires the model to continuously refer to context: it must consider both the user input and the content already generated. The Transformer structure can process contextual relationships in token sequences, so it is well suited to supporting the process of token-by-token generation.

What Natural Language Processing Tasks Can Large Language Models Complete?

The capabilities of large language models are mainly reflected in language- and text-related tasks. Based on prompts, they can generate, translate, identify, and interpret text, and can also generate code blocks.

Common tasks include:

  • Text generation: Generate article drafts, summaries, rewritten content, or question-and-answer responses based on prompts.
  • Text translation: Convert input text into expressions in another language.
  • Content generation: Generate explanations, outlines, or other text content around a given topic.
  • Text identification and interpretation: Understand, summarize, or explain input text.
  • Code block generation: Generate code snippets or auxiliary explanations based on prompts.

Key point: The typical capabilities of large language models revolve around natural language processing, but their output is not necessarily limited to ordinary text; it can also include structured content such as code blocks.

Summarizing How Large Language Models Are Used with Scenario Tables

The table below summarizes how large language models are used in common tasks from three dimensions: user need, solution method, and output result.

User needSolution methodOutput result
Write or rewrite textGenerate content based on topic, tone, and format promptsDrafts, summaries, rewrites, or Q&A responses
Translate textGenerate corresponding expressions based on the input text and target languageTranslation results or multi-version expressions
Assist developmentGenerate code blocks or explain coding ideas based on requirement promptsCode snippets, explanatory text, or example structures
Explain text contentIdentify, summarize, and explain the input textSummaries, key points, classifications, or explanations
Handle natural language tasksPerform generation, translation, identification, and other tasks based on promptsText output that matches the task objective

What these scenarios have in common is that the user expresses a goal through a prompt, the model converts the prompt into computable representations, and then gradually generates a result based on the context.

Several Issues That Are Easy to Confuse When Understanding Large Language Models

Training Stage and Inference Stage Are Different

The focus of the training stage is to let the model learn language patterns from large amounts of data. The focus of the inference stage is that the model receives a user prompt and generates results based on the patterns it has learned. One is the learning process, and the other is the usage process.

Prompts, Tokens, and Embedding Vectors Are Not the Same Thing

A prompt is the text the user inputs to the model; tokens are the basic units the model uses when processing text; embedding vectors are numerical representations of tokens. Understanding the differences among these three helps explain how large language models move from natural language input into neural network computation.

ConceptMeaningRole in the workflow
PromptThe task or question entered by the userSpecifies what the model should process
TokenThe basic fragment used by the model to process textSplits the input into processable units
Embedding vectorThe numerical representation of a tokenEnables the model to perform computation

Generating Text Is Not the Same as Directly Retrieving Materials

The core process of a large language model is to generate output token by token based on model computation, rather than simply copying answers from a fixed database. It can generate text that appears coherent, and it can also generate code blocks, but these outputs come from a prompt-driven generation process.

Being Able to Generate Code Blocks Does Not Mean It Is Only for Programming Tasks

Code block generation is just one expression of the prompt-driven generation capability of large language models. They can also be used for text generation, translation, summarization, explanation, and other natural language processing tasks.