Large language model

A large language model (LLM) is a neural network trained on a huge amount of text to do one thing: guess the next word, or more precisely the next token, a piece of a word. A model’s answer is many such guesses in a row. Large language models power chatbots, AI coding assistants and AI agents.

The technology’s biggest surprise is scale. The task, “continue the text”, has not changed since 2018, but the bigger the models, data and compute became, the more the models could do: solve tasks from a few examples, translate, write programs, and then reason and work on their own.¹² The debate over whether there is understanding behind this is still going on.

History

• 12.06.2017Google researchers publish the paper “Attention Is All You Need” on the Transformer architecture: it is built solely on the attention mechanism, without recurrent networks, and trains faster than earlier ones.³ ¶ csanuy4nd
• 11.06.2018OpenAI presents GPT: a transformer is first trained on a large corpus of text simply to predict what comes next, and then fine-tuned on small labelled datasets for a specific task.⁴ ¶ c9p9gfxz6
• 14.02.2019GPT-2 has 1.5 billion parameters and was “trained simply to predict the next word in 40GB of Internet text”. At first OpenAI does not release the full model for fear of misuse, and releases it in stages through 05.11.2019.⁵ ¶ cx8dznxpz
• 23.01.2020Jared Kaplan and co-authors at OpenAI describe scaling laws: a model’s loss falls as a power law as the number of parameters, the amount of data and compute grow.¹ ¶ czpw8t6ic
• 28.05.2020GPT-3 has 175 billion parameters. The model solves new tasks from a few examples given right in the prompt, without fine-tuning.² ¶ chru876b8
• 04.03.2022OpenAI describes InstructGPT: the model is fine-tuned on human ratings (RLHF), and people prefer the answers of its 1.3-billion-parameter version to those of GPT-3, which is 100 times larger.⁶ ¶ c585bcdf6
• 29.03.2022DeepMind presents Chinchilla: with the same compute budget, a 70-billion-parameter model trained on more text outperforms the 280-billion-parameter Gopher. The authors’ conclusion: model size and the number of training tokens should be scaled equally.⁷ ¶ cgg6g9xvw
• 30.11.2022OpenAI opens ChatGPT, a model from the GPT-3.5 series fine-tuned the same way as InstructGPT.⁸ ¶ cexyrfgim
• 15.12.2022Anthropic describes Constitutional AI, a way to train a harmless assistant without human labels of harmful outputs: the only human oversight is a list of rules and principles, a “constitution”. The model critiques and revises its own answers, and another model decides which of two answers is better.⁹ ¶ c6yd2bh89
• 01.02.2023According to UBS analysts, ChatGPT reached about 100 million monthly users two months after launch, faster than any consumer app before it.¹⁰ ⧗ ¶ c77uhf6xw
• 14.03.2023GPT-4: according to OpenAI, on a simulated bar exam the model scores around the top 10% of test takers, while GPT-3.5 scored around the bottom 10%.¹¹ ¶ cj6i7dziy
• 14.03.2023On the same day Anthropic opens Claude, an assistant based on the company’s research into “training helpful, honest, and harmless AI systems”, in two versions: Claude and the lighter, cheaper and faster Claude Instant; access is through a chat interface and an API.¹² ¶ c62ai5hfk
• 18.07.2023Meta releases Llama 2: “Llama 2 is free for research and commercial use.” The first LLaMA, in February, had been given only to researchers on request. Llama 2 starts a wave of open-weight models.¹³ ¶ cauga2wua
• 04.03.2024Anthropic releases the Claude 3 family, Haiku, Sonnet and Opus: models of different sizes to choose between by the balance of intelligence, speed and price. According to the company, Opus outperforms its peers on most common benchmarks; the context window is 200,000 tokens.¹⁴ ¶ cafmxgxnp
• 12.09.2024OpenAI releases o1, a reasoning model: reinforcement learning teaches it to “think” with a chain of thought before answering, and quality improves not only with training compute but also with time spent thinking.¹⁵ ¶ c2aep9gte
• 20.01.2025DeepSeek releases R1, an open-weight reasoning model under the MIT license; according to the company, it is on par with o1 in math, code and reasoning.¹⁶ ¶ cbq5443qj
• 07.10.2026OpenAI opens GPT-6 in ChatGPT to all users, including free ones.¹⁷ ¶ cb92g57km
• 07.10.2026On the same day Anthropic releases Claude Haiku 5.5: according to the company, it costs on average about 75% less to run than Haiku 4.5, and a million input tokens cost $0.10 instead of $1.¹⁸ ¶ csfz8waiq

How it works

Text is split into tokens, and from all the preceding tokens the model outputs probabilities for the next one; an answer is built from such steps, one token at a time. Training has two stages. In pre-training the model reads a huge corpus of text and learns to continue it; this is where it gains knowledge and skills. Fine-tuning turns a “text continuer” into an assistant: the model is taught to follow instructions and answer the way people prefer (RLHF), and reasoning models are also taught to solve problems with a long chain of thought in which the answer is checked (reinforcement learning).⁶¹⁵

The core ability comes from pre-training. In the GPT-4 report OpenAI writes that “the model’s capabilities seem to come primarily from the pre-training process—RLHF does not improve exam performance”.¹⁹ That is why the model race is above all a race of scale: scaling laws promise a predictable return on every dollar invested, and Chinchilla showed that data matter as much as model size.¹⁷

What happens inside a model long remained a black box. On 21.05.2024 Anthropic reported that it had found how millions of concepts are represented inside Claude Sonnet; in its words, this is “the first ever detailed look inside a modern, production-grade large language model”. For example, the same feature fires on mentions of the Golden Gate Bridge in English, Japanese, Chinese, Russian and other languages, and on images of it.²⁰ ¶ capmi3s8r

Data

Incidents

Mata v. Avianca: court decisions made up by ChatGPT, June 2023 ¶ ct8pjche9

Lawyers for the plaintiff in a case against the airline Avianca filed a document in federal court in New York citing non-existent court decisions with fabricated quotes, all made up by ChatGPT. On 22.06.2023 Judge P. Kevin Castel fined the two lawyers and their firm $5,000: they had not only submitted the fabrications but kept standing by them after the court questioned whether the decisions existed.²⁷

Criticism

In the paper “On the Dangers of Stochastic Parrots” (03.2021), Emily Bender, Timnit Gebru and co-authors write that “an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot”. The coherence of its text, they argue, exists only in the reader’s perception.²⁸ ¶ cx8v9ep8f

On 26.03.2023 Yann LeCun posted a slide reading “Auto-Regressive LLMs are doomed. … It’s not fixable (without a major redesign).”: each token has some probability of taking the answer off the correct path, the error compounds exponentially, and this cannot be fixed without a major change of architecture.²⁹ He is now the founder of AMI Labs, which builds “world models” that learn from real-world sensor data.³⁰ ¶ ccpxjcsqn

On 04.09.2025 OpenAI researchers explained model hallucinations this way: “the training and evaluation procedures reward guessing over acknowledging uncertainty”. Of 10 popular benchmarks, 9 give zero points for an “I don’t know” answer, so a model is better off guessing.³¹ ¶ cmt8ggmzu

Authors and publishers are challenging in court the very act of training on their texts. On 27.12.2023 The New York Times sued OpenAI and Microsoft; the case is now being heard together with other lawsuits against OpenAI, and there is no ruling yet.³² ¶ cur6y399x

In Bartz v. Anthropic, Judge William Alsup ruled on 23.06.2025 that training Claude on books was fair use, “exceedingly transformative”, but storing pirated copies of books was not.³³ Anthropic agreed to pay $1.5 billion, about $3,000 per work; on 20.07.2026 the court gave final approval to the settlement.³⁴ ¶ c3yvibi7r

Evolution

Training on models’ own texts does not replace human data: in a paper in Nature (24.07.2024), Ilia Shumailov and co-authors showed that when models are trained indiscriminately on generated text, the “tails of the original content distribution disappear” irreversibly: the rare and the unusual.³⁵ ¶ cbar47t97

On 13.12.2024 at the NeurIPS conference, Ilya Sutskever said that “pre-training as we know it will unquestionably end”: compute is growing but data is not, because “we have but one internet”; he called data “the fossil fuel of AI”. As next steps he named agents, synthetic data and inference-time compute.³⁶ ⧗ ¶ ci59hi8ky

That is how it turned out: since 2024 progress has come not only from pre-training but also from reasoning (reasoning models), and models increasingly work on their own, as agents, in agentic engineering and in mathematics. In June 2025 Andrej Karpathy said “the LLM is a new kind of a computer” and “LLMs are kind of like a new operating system”: “your prompts are now programs that program the LLM. And remarkably these prompts are written in English.”³⁷ ¶ ccan4rb26