--- title: Large language model url: https://ftrlg.com/en/wiki/large-language-model/ lang: en published: 2026-10-16 version: 1 summary: A neural network trained to guess the next word in a huge amount of text. As it scaled up, it learned to translate, write code and reason, and became the foundation of chatbots and AI agents. cite_as: "future.log, «Large language model», https://ftrlg.com/en/wiki/large-language-model/" license: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ --- # Large language model A **large language model** (LLM) is a neural network trained on a huge amount of text to do one thing: guess the next word, or more precisely the next token, a piece of a word. A model's answer is many such guesses in a row. Large language models power chatbots, AI coding assistants and [AI agents](https://ftrlg.com/en/wiki/autonomous-ai-agents/). The technology's biggest surprise is scale. The task, “continue the text”, has not changed since 2018, but the bigger the models, data and compute became, the more the models could do: solve tasks from a few examples, translate, write programs, and then reason and work on their own.[^4][^5] The debate over whether there is understanding behind this is still going on. ## History - **12.06.2017** Google researchers publish the paper “Attention Is All You Need” on the Transformer architecture: it is built solely on the attention mechanism, without recurrent networks, and trains faster than earlier ones.[^1] - **11.06.2018** [OpenAI](https://ftrlg.com/en/models/openai/) presents GPT: a transformer is first trained on a large corpus of text simply to predict what comes next, and then fine-tuned on small labelled datasets for a specific task.[^2] - **14.02.2019** GPT-2 has 1.5 billion parameters and was “trained simply to predict the next word in 40GB of Internet text”. At first OpenAI does not release the full model for fear of misuse, and releases it in stages through 05.11.2019.[^3] - **23.01.2020** Jared Kaplan and co-authors at OpenAI describe scaling laws: a model's loss falls as a power law as the number of parameters, the amount of data and compute grow.[^4] - **28.05.2020** GPT-3 has 175 billion parameters. The model solves new tasks from a few examples given right in the prompt, without fine-tuning.[^5] - **04.03.2022** OpenAI describes InstructGPT: the model is fine-tuned on human ratings (RLHF), and people prefer the answers of its 1.3-billion-parameter version to those of GPT-3, which is 100 times larger.[^6] - **29.03.2022** [DeepMind](https://ftrlg.com/en/models/google-deepmind/) presents Chinchilla: with the same compute budget, a 70-billion-parameter model trained on more text outperforms the 280-billion-parameter Gopher. The authors' conclusion: model size and the number of training tokens should be scaled equally.[^7] - **30.11.2022** OpenAI opens ChatGPT, a model from the GPT-3.5 series fine-tuned the same way as InstructGPT.[^8] - **15.12.2022** [Anthropic](https://ftrlg.com/en/models/anthropic/) describes Constitutional AI, a way to train a harmless assistant without human labels of harmful outputs: the only human oversight is a list of rules and principles, a “constitution”. The model critiques and revises its own answers, and another model decides which of two answers is better.[^33] - **01.02.2023** According to UBS analysts, ChatGPT reached about 100 million monthly users two months after launch, faster than any consumer app before it.[^9] ⧗ - **14.03.2023** GPT-4: according to OpenAI, on a simulated bar exam the model scores around the top 10% of test takers, while GPT-3.5 scored around the bottom 10%.[^10] - **14.03.2023** On the same day Anthropic opens [Claude](https://ftrlg.com/en/models/claude/), an assistant based on the company's research into “training helpful, honest, and harmless AI systems”, in two versions: Claude and the lighter, cheaper and faster Claude Instant; access is through a chat interface and an API.[^34] - **18.07.2023** [Meta](https://ftrlg.com/en/models/meta/) releases Llama 2: “Llama 2 is free for research and commercial use.” The first LLaMA, in February, had been given only to researchers on request. Llama 2 starts a wave of open-weight models.[^11] - **04.03.2024** Anthropic releases the Claude 3 family, Haiku, Sonnet and Opus: models of different sizes to choose between by the balance of intelligence, speed and price. According to the company, Opus outperforms its peers on most common benchmarks; the context window is 200,000 tokens.[^35] - **12.09.2024** OpenAI releases o1, a reasoning model: reinforcement learning teaches it to “think” with a chain of thought before answering, and quality improves not only with training compute but also with time spent thinking.[^12] - **20.01.2025** [DeepSeek](https://ftrlg.com/en/models/deepseek-ai/) releases R1, an open-weight reasoning model under the MIT license; according to the company, it is on par with o1 in math, code and reasoning.[^13] - **07.10.2026** OpenAI opens GPT-6 in ChatGPT to all users, including free ones.[^14] - **07.10.2026** On the same day Anthropic releases Claude Haiku 5.5: according to the company, it costs on average about 75% less to run than Haiku 4.5, and a million input tokens cost $0.10 instead of $1.[^36] ## How it works Text is split into tokens, and from all the preceding tokens the model outputs probabilities for the next one; an answer is built from such steps, one token at a time. Training has two stages. In **pre-training** the model reads a huge corpus of text and learns to continue it; this is where it gains knowledge and skills. **Fine-tuning** turns a “text continuer” into an assistant: the model is taught to follow instructions and answer the way people prefer (RLHF), and reasoning models are also taught to solve problems with a long chain of thought in which the answer is checked (reinforcement learning).[^6][^12] The core ability comes from pre-training. In the GPT-4 report OpenAI writes that “the model's capabilities seem to come primarily from the pre-training process—RLHF does not improve exam performance”.[^15] That is why the model race is above all a race of scale: scaling laws promise a predictable return on every dollar invested, and Chinchilla showed that data matter as much as model size.[^4][^7] What happens inside a model long remained a black box. On 21.05.2024 Anthropic reported that it had found how millions of concepts are represented inside Claude Sonnet; in its words, this is “the first ever detailed look inside a modern, production-grade large language model”. For example, the same feature fires on mentions of the Golden Gate Bridge in English, Japanese, Chinese, Russian and other languages, and on images of it.[^37] ## Data | study | what was measured | result | |---|---|---| | Epoch AI, 28.05.2024[^16] | training compute of frontier models since 2010; estimate | grows **4–5x per year** | | Stanford AI Index 2024, 04.2024[^17] | cost of training compute at cloud rental prices; Epoch AI estimate, as the companies do not disclose figures | GPT-4 about **$78 million**, Gemini Ultra about **$191 million** | | Stanford AI Index 2025, 04.2025[^18] | price of querying the cheapest model that scores at least as well as GPT-3.5 on MMLU | from $20 to **$0.07** per million tokens between 11.2022 and 10.2024, more than 280 times cheaper | | Epoch AI, 06.06.2024[^19] | stock of public human-generated text; estimate | about **300 trillion tokens**; at current rates it will be fully used between 2026 and 2032 | | NBER, 09.2025[^20] | how people use ChatGPT (consumer plans only); two of the authors are OpenAI employees | by 07.2025, **700 million** people a week, about 10% of the world's adults; 73% of messages are not related to work | | Stanford AI Index 2026, 04.2026[^21] | gap between the best US and Chinese models in the Arena ranking | as of 03.2026 the best US model, Claude Opus 4.6, leads the best Chinese one by **2.7%**; the report says “The U.S.-China AI model performance gap has effectively closed” | > **[EDITORS' CONCLUSION]** The data show two curves going in opposite directions. Training compute of frontier models grows 4–5x a year, and training a single model already costs hundreds of millions of dollars. Meanwhile the price of the same level of capability falls hundreds of times: a model that was frontier in 2022 costs almost nothing two years later. Hence mass adoption, hundreds of millions of people a week, and fast catch-up: Chinese models have nearly drawn level with the best American ones. The bottleneck is data: human-written text may last only until the end of the decade. Caveats: training costs and the stock of text are estimates from indirect data, and the usage study was carried out with OpenAI's involvement. ## Incidents ### Mata v. Avianca: court decisions made up by ChatGPT, June 2023 Lawyers for the plaintiff in a case against the airline Avianca filed a document in federal court in New York citing non-existent court decisions with fabricated quotes, all made up by ChatGPT. On 22.06.2023 Judge P. Kevin Castel fined the two lawyers and their firm $5,000: they had not only submitted the fabrications but kept standing by them after the court questioned whether the decisions existed.[^22] ## Criticism In the paper “On the Dangers of Stochastic Parrots” (03.2021), Emily Bender, Timnit Gebru and co-authors write that “an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot”. The coherence of its text, they argue, exists only in the reader's perception.[^23] On 26.03.2023 Yann LeCun posted a slide reading “Auto-Regressive LLMs are doomed. … It's not fixable (without a major redesign).”: each token has some probability of taking the answer off the correct path, the error compounds exponentially, and this cannot be fixed without a major change of architecture.[^24] He is now the founder of AMI Labs, which builds “world models” that learn from real-world sensor data.[^25] On 04.09.2025 OpenAI researchers explained model hallucinations this way: “the training and evaluation procedures reward guessing over acknowledging uncertainty”. Of 10 popular benchmarks, 9 give zero points for an “I don't know” answer, so a model is better off guessing.[^26] Authors and publishers are challenging in court the very act of training on their texts. On 27.12.2023 The New York Times sued OpenAI and Microsoft; the case is now being heard together with other lawsuits against OpenAI, and there is no ruling yet.[^27] In Bartz v. Anthropic, Judge William Alsup ruled on 23.06.2025 that training [Claude](https://ftrlg.com/en/models/claude/) on books was fair use, “exceedingly transformative”, but storing pirated copies of books was not.[^28] [Anthropic](https://ftrlg.com/en/models/anthropic/) agreed to pay $1.5 billion, about $3,000 per work; on 20.07.2026 the court gave final approval to the settlement.[^29] ## Evolution Training on models' own texts does not replace human data: in a paper in Nature (24.07.2024), Ilia Shumailov and co-authors showed that when models are trained indiscriminately on generated text, the “tails of the original content distribution disappear” irreversibly: the rare and the unusual.[^30] On 13.12.2024 at the NeurIPS conference, Ilya Sutskever said that “pre-training as we know it will unquestionably end”: compute is growing but data is not, because “we have but one internet”; he called data “the fossil fuel of AI”. As next steps he named agents, synthetic data and inference-time compute.[^31] ⧗ That is how it turned out: since 2024 progress has come not only from pre-training but also from reasoning (reasoning models), and models increasingly work on their own, as [agents](https://ftrlg.com/en/wiki/autonomous-ai-agents/), in [agentic engineering](https://ftrlg.com/en/wiki/agentic-engineering/) and in [mathematics](https://ftrlg.com/en/wiki/ai-mathematics/). In June 2025 [Andrej Karpathy](https://ftrlg.com/en/wiki/andrej-karpathy/) said “the LLM is a new kind of a computer” and “LLMs are kind of like a new operating system”: “your prompts are now programs that program the LLM. And remarkably these prompts are written in English.”[^32] ## Signals - 2026-10-15 — Nick Cammarata: «The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread … Neural networks are already clean» — interpretability researcher at Goodfire who started the automated neuron interpretability project at OpenAI, in the essay “Neural Networks Are Already Clean” (a working hypothesis, not peer-reviewed; a companion paper is promised); Neel Nanda and other researchers pushed back (https://ftrlg.com/en/signals/cgij8q2um/) - 2026-10-11 — Andrej Karpathy: «Around 6B people (~75% of the population) have barely come in contact with LLMs, while ~5,000 have access to frontier-grade systems internally» — on the widening gap in understanding AI capability between the public, professionals and the labs (https://ftrlg.com/en/signals/cc3qwtwnz/) ## See also Transformer · Chatbots · Reasoning models · Open weights · [Autonomous AI agents](https://ftrlg.com/en/wiki/autonomous-ai-agents/) · [Vibe coding](https://ftrlg.com/en/wiki/vibe-coding/) · [AI in mathematics](https://ftrlg.com/en/wiki/ai-mathematics/) ## References [^1]: A. Vaswani et al. Attention Is All You Need — arXiv, 12.06.2017. [arxiv.org](https://arxiv.org/abs/1706.03762) [^2]: Improving language understanding with unsupervised learning — OpenAI, 11.06.2018. [openai.com](https://openai.com/index/language-unsupervised/) [^3]: Better Language Models and Their Implications — OpenAI, 14.02.2019. [openai.com](https://openai.com/index/better-language-models/); GPT-2: 1.5B Release — OpenAI, 05.11.2019. [openai.com](https://openai.com/index/gpt-2-1-5b-release/) [^4]: J. Kaplan et al. Scaling Laws for Neural Language Models — arXiv, 23.01.2020. [arxiv.org](https://arxiv.org/abs/2001.08361) [^5]: T. Brown et al. Language Models are Few-Shot Learners — arXiv, 28.05.2020. [arxiv.org](https://arxiv.org/abs/2005.14165) [^6]: L. Ouyang et al. Training language models to follow instructions with human feedback — arXiv, 04.03.2022. [arxiv.org](https://arxiv.org/abs/2203.02155) [^7]: J. Hoffmann et al. Training Compute-Optimal Large Language Models — arXiv, 29.03.2022. [arxiv.org](https://arxiv.org/abs/2203.15556) [^8]: Introducing ChatGPT — OpenAI, 30.11.2022. [openai.com](https://openai.com/index/chatgpt/) [^9]: ChatGPT sets record for fastest-growing user base — Reuters, 02.02.2023 (reprint). [finance.yahoo.com](https://finance.yahoo.com/news/1-chatgpt-sets-record-fastest-051929384.html) ⧗ [^10]: GPT-4 — OpenAI, 14.03.2023. [openai.com](https://openai.com/index/gpt-4-research/) [^11]: Meta and Microsoft Introduce the Next Generation of Llama — Meta, 18.07.2023. [about.fb.com](https://about.fb.com/news/2023/07/llama-2/) [^12]: Learning to reason with LLMs — OpenAI, 12.09.2024. [openai.com](https://openai.com/index/learning-to-reason-with-llms/) [^13]: DeepSeek-R1 Release — DeepSeek, 20.01.2025. [api-docs.deepseek.com](https://api-docs.deepseek.com/news/news250120) [^14]: GPT-6 and Intelligent UI for everyone — OpenAI, 07.10.2026. [openai.com](https://openai.com/index/gpt-6-for-everyone/) [^15]: OpenAI. GPT-4 Technical Report — arXiv, 15.03.2023. [arxiv.org](https://arxiv.org/abs/2303.08774) [^16]: Training compute of frontier AI models grows by 4-5x per year — Epoch AI, 28.05.2024. [epoch.ai](https://epoch.ai/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year) [^17]: AI Index Report 2024 — Stanford HAI, 04.2024. [hai.stanford.edu](https://hai.stanford.edu/ai-index/2024-ai-index-report) [^18]: AI Index Report 2025 — Stanford HAI, 04.2025. [hai.stanford.edu](https://hai.stanford.edu/ai-index/2025-ai-index-report) [^19]: Will we run out of data? Limits of LLM scaling based on human-generated data — Epoch AI, 06.06.2024. [epoch.ai](https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data) [^20]: A. Chatterji et al. How People Use ChatGPT — NBER Working Paper 34255, 09.2025. [nber.org](https://www.nber.org/papers/w34255) [^21]: AI Index Report 2026 — Stanford HAI, 04.2026. [hai.stanford.edu](https://hai.stanford.edu/ai-index/2026-ai-index-report) [^22]: Mata v. Avianca, Inc., 22-cv-1461 (PKC): Opinion and Order on Sanctions — U.S. District Court, S.D.N.Y., 22.06.2023. [courtlistener.com](https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0_3.pdf) [^23]: E. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? — FAccT ’21, ACM, 03.2021. [doi.org](https://doi.org/10.1145/3442188.3445922) [^24]: Yann LeCun: “I have claimed that Auto-Regressive LLMs are exponentially diverging diffusion processes” — X, 26.03.2023. [x.com](https://x.com/ylecun/status/1640122342570336267) [^25]: AMI Labs. [amilabs.xyz](https://amilabs.xyz) [^26]: A. Kalai, O. Nachum, S. Vempala, E. Zhang. Why Language Models Hallucinate — arXiv, 04.09.2025. [arxiv.org](https://arxiv.org/abs/2509.04664) [^27]: The New York Times Company v. Microsoft Corporation et al., 1:23-cv-11195: complaint — U.S. District Court, S.D.N.Y., 27.12.2023. [nytco-assets.nytimes.com](https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf) [^28]: Bartz v. Anthropic PBC, 3:24-cv-05417: Order on Fair Use — U.S. District Court, N.D. Cal., 23.06.2025. [courtlistener.com](https://storage.courtlistener.com/recap/gov.uscourts.cand.434709/gov.uscourts.cand.434709.231.0_2.pdf) [^29]: Anthropic Copyright Settlement — official settlement website. [anthropiccopyrightsettlement.com](https://www.anthropiccopyrightsettlement.com/) [^30]: I. Shumailov et al. AI models collapse when trained on recursively generated data — Nature, 24.07.2024. [nature.com](https://www.nature.com/articles/s41586-024-07566-y) [^31]: Ilya Sutskever. Sequence to sequence learning with neural networks: what a decade — NeurIPS, 13.12.2024 (talk recording). [youtube.com](https://www.youtube.com/watch?v=1yvBqasHLZs) ⧗ [^32]: Andrej Karpathy: Software Is Changing (Again) — Y Combinator, 18.06.2025. [youtube.com](https://www.youtube.com/watch?v=LCEmiRjPEtQ) [^33]: Y. Bai et al. Constitutional AI: Harmlessness from AI Feedback — arXiv, 15.12.2022. [arxiv.org](https://arxiv.org/abs/2212.08073) [^34]: Introducing Claude — Anthropic, 14.03.2023. [anthropic.com](https://www.anthropic.com/news/introducing-claude) [^35]: Introducing the next generation of Claude — Anthropic, 04.03.2024. [anthropic.com](https://www.anthropic.com/news/claude-3-family) [^36]: Introducing Claude Haiku 5.5 — Anthropic, 07.10.2026. [anthropic.com](https://www.anthropic.com/claude-haiku-5-5) [^37]: Mapping the mind of a large language model — Anthropic, 21.05.2024. [anthropic.com](https://www.anthropic.com/research/mapping-mind-language-model) — future.log, «Large language model», https://ftrlg.com/en/wiki/large-language-model/ · CC BY 4.0