AÑEZ.digital
Research · How an LLM is built

How an LLM is built

And how the brain you already own does the same job with 20 watts. One long page: a brain you can take apart, a real language model that runs and trains right here, and the path to building our own and giving it away.

13 Nov 2026 Long form · about 25 minutes Live model inside 34 sources Analogies are loose
Neurons in your head86 billioneach one wired to thousands of others
Power your brain runs on~20 Wa dim light bulb
Numbers in this page's model2,975trained in 2.3 s at build time
Numbers in a large open model405 billionLlama 3.1, weights free to download
Text a large model reads15 trilliontokens, for one training run
Cheapest full chat model~$100four hours on rented GPUs

The whole thing in one picture

A large language model is a very big table of numbers that has been adjusted, billions of times, until it is good at one game: guess the next piece of text. Everything else you see it do (answer, write code, plan) falls out of playing that game extremely well.

The one sentence to keep. A brain and an LLM are both networks of simple units joined by adjustable connections, and both learn by changing the strength of those connections. The units, the wiring and the way they learn are very different. This page walks through both, side by side.

How to read this page

  1. Play with the two machines. Fig 1 is a brain you can light up and open. Fig 2 is a press that writes one word at a time using a real model.
  2. Hover everything. Stats, chart points and flowchart steps carry extra detail.
  3. Press Train in section 6. You will watch a language model go from noise to sentences in a few seconds.

1The brain you already own

Start with the original. About 1.3 kilograms, about 86 billion neurons [1], running on roughly the power of a dim bulb [3]. It is not one machine. It is a set of regions with different jobs, wired together, all running at once.

Below is a teaching model. Hover any region to light it up and see what it does, what the closest part of an LLM is, and where that comparison stops being true. Left alone, it runs a tour: a signal enters through the eye, crosses the cortex, then the lid comes off to show the deep parts.

The region that is firing nowCerebrum: four lobes on top, temporal lobe on the sideDeep parts, on the stem under the lidA simplified model, seen from the left. A signal comes in through the eye, runs along the optic nerve to the thalamus, then crosses the cortex from the back (seeing) to the side (meaning) to the front (speaking and deciding). Lift the cerebrum to see the parts that store memories, flag what matters and build habits.

Pick a region

What it does

Closest thing in an LLM

Where the comparison breaks

One thought, start to finish

You see the word "apple" on a page and say it out loud. That takes about half a second and crosses most of the brain. This is the classic textbook route, simplified:

Notice the shape: input, a stack of processing stages, output, with one part holding the goal the whole time. Keep that shape in mind. A language model has it too.

The twelve parts and their closest match in an LLM

#PartWhat it doesClosest thing in an LLM
01Prefrontal cortexKeeps the goal in mind, plans the steps, and stops the impulse that would break the plan. This is working memory and self-control.The context window and the system prompt. Whatever sits there steers every word that follows.
02Motor and speech cortexThe back of the frontal lobe. Broca's area lines up the words; the motor strip sends the command to the muscles.The output head: it scores every possible next token, and one is picked and written.
03Parietal lobeTouch, body position and space. It builds the map of where things are and helps point attention at one of them.Attention: for this step, which earlier pieces of the input matter most?
04Occipital lobeThe visual cortex. It turns signals from the eye into edges, then shapes, then objects, in a stack of layers.The tokenizer and the embedding layer: raw input turned into numbers the rest of the network can use.
05Temporal lobeHearing, recognising faces and objects, and word meaning. Wernicke's area, at the back of it, is where speech starts to make sense.The middle layers, where meaning lives as directions in a space of numbers. Researchers can find single features there, such as one for the Golden Gate Bridge.
06CerebellumSmooths and times movement, and learns from the error between what you meant and what happened. It holds most of the brain's neurons.The feed-forward (MLP) blocks, and supervised learning: predict, compare with the answer, adjust.
07BrainstemBreathing, heartbeat, sleep and waking. Every signal between the brain and the body passes through it.The runtime: the servers, power and cables that keep the model running and carry text in and out.
08ThalamusThe hub in the middle. Almost every sense passes through it on the way to the cortex, and it decides what gets through.The residual stream: the shared channel every layer reads from and writes to.
09HippocampusBinds a new experience into a memory within seconds, then replays it during sleep so the cortex can learn it slowly.Retrieval memory: notes and documents stored outside the model and looked up when needed. Training is the slow cortex.
10AmygdalaTags an event as a threat or a reward, fast, so that you react and remember it.The closest thing is a classifier that flags what needs care or priority.
11Basal gangliaChooses one action out of many and turns repeated choices into habits. Dopamine here signals better or worse than expected.Reinforcement learning: try an output, get a score, shift the weights toward what scored well.
12HypothalamusHunger, thirst, body temperature, hormones and the body clock. It keeps the body inside safe limits.Nothing. A model has no body, no needs and no drives of its own.
Read the last column as a hint, not a fact. Nobody designed an LLM by copying these regions. The matches are useful for remembering what each part of a model is for. Section 8 is the honest scorecard.

2One neuron, and why it zaps

Zoom all the way in. Every region above is made of the same basic unit: the neuron. It does one thing. It adds up what it hears and decides whether to fire.

A neuron you can drive

ABC dendrites: inputs cell body: adds them up axon: the output wire, wrapped in insulation synapses

The spike itself

This is the famous shape. Every spike from a given neuron looks the same, so the message is not in the size of the spike. It is in how often and when the neuron fires.

One spike · voltage inside the cell, in millivolts

+300-55-700 ms1 ms2 ms3 ms4 ms5 ms threshold resting
A schematic drawn from the textbook values [4]. Hover the six dots, left to right.

The copy we made in 1943

In 1943 Warren McCulloch and Walter Pitts wrote the neuron down as arithmetic [8]. Fifteen years later Frank Rosenblatt built one that could learn [9]. The artificial neuron keeps the core idea and throws away almost everything else:

Biological neuronArtificial neuron
InputsChemical nudges on thousands of dendrite branchesA list of numbers
Connection strengthSynapse strength, changed by useA weight: one number per input
Adding upVoltage builds in the cell body, and leaksMultiply each input by its weight, add
DecisionFire fully at about -55 mV, or not at allA simple curve, such as "keep it if positive, else zero"
OutputA train of identical spikes, timedOne number
SpeedA spike takes about 2 ms; signals travel metres per secondBillions of multiplications per second on a GPU
EnergyThe whole brain: about 20 WOne H100 GPU: up to 700 W [27]

The slider demo above is exactly an artificial neuron wearing a biological costume: three inputs, three weights, a sum, a threshold.

3From one neuron to a network

One neuron can only draw a single line between yes and no. Stack them in layers, feed each layer into the next, and the network can represent almost any pattern. The knowledge is not stored in any one neuron. It is spread across the weights of all the connections.

A small network · every line is one weight

inputshidden layerhidden layeroutputs
78 weights here. Thick lines are strong connections, dashed lines are negative ones (they hold the next unit back). The moving pulses show one input passing through.

Two ways to learn

The brain's way · local

Fire together, wire together

Each synapse changes based on what its own two neurons just did. Nothing tells it the big picture. Signals such as dopamine then say, broadly, "that went better than expected, keep those changes" [33]. It works while you live, one experience at a time.

The machine's way · global

Measure the error, pass the blame backwards

Run an example through. Compare the answer with the right one. Then work backwards through every layer and compute, for every single weight, how much it contributed to the error. Nudge each one the right way. This is backpropagation [10]. No one has found a mechanism in the brain that does exactly this, and whether the brain approximates it is an open question.

Eighty years in one line

1943The neuron as maths
1949Learning by use
1958Perceptron
1986Backpropagation
1997LSTM memory
2012GPUs and AlexNet
2013Words as vectors
2014Attention
2017The transformer
2020GPT-3
2022ChatGPT
2024Open giants
2025Yours for $100

4Turning words into numbers

A network only does arithmetic, so text has to become numbers first. That happens in two steps.

Step 1 · Tokens

A tokenizer cuts text into pieces and gives each piece an ID. Real models use pieces smaller than words, found by a method called byte pair encoding [24]: start from single characters, then keep merging the pair that appears together most often. Common words end up as one token; rare words are spelled out of parts. GPT-2 ended up with a vocabulary of 50,257 pieces [16]. For English, a common rule of thumb is about four characters per token.

The model on this page keeps it simple: one token per word, 31 words in total. Type using only its words and watch them turn into IDs. Anything it has never seen is dropped.

The tokenizer of this page's model

Its whole vocabulary:

Step 2 · Embeddings

An ID is just a label. So each token gets its own list of numbers, called an embedding: its coordinates in a space of meaning. Those coordinates start random and are learned during training, like every other weight. Our model uses 16 numbers per word. GPT-3 used 12,288 [17].

Nobody tells the model which words are alike. Below are the real coordinates our model learned, squeezed from 16 dimensions down to 2 so they fit on a screen.

What the model learned about its words · 16 dimensions flattened to 2

atebirdbonebranchcatchasedcheesedogdrankfishhidholeinmatmilkmouseonroserugsangsatseaseedskysleptsunswamthetreeworm.
Animals and the sun (who does it)Verbs (what they did)Things and places (where it ends)Glue words
Each dot is one word. Position comes only from training. Hover a dot for its numbers.
What the numbers say. Measured in the full 16 dimensions, the things and places sit 28% closer to each other than to other words. No one labelled them. Words that are used the same way get pushed to the same neighbourhood because that makes the next word easier to predict. This is the effect word2vec made famous in 2013 [13].

5The transformer: one step of thought

Since 2017 nearly every language model has used the same design, the transformer [15]. It is a stack of identical layers. Each layer does two things to every token.

  1. Attention: look around. Each token asks "which earlier tokens matter for me right now?" and pulls in information from them, weighted by how much they matter.
  2. MLP: think it over. A small feed-forward network processes what was gathered. Much of what a model "knows" is stored in these weights.

After the last layer, the model scores every token in its vocabulary as a candidate for the next one, turns the scores into probabilities, and picks. Then it adds that token to the text and does the whole thing again.

Here is that loop as a machine. It runs our real model: the lifters are its true attention weights and the bars are its true probabilities.

thecatsatonthemat100%rug0%branch0%sat0%tree0%
The part doing the work right nowLifters: how much the last token looks at each earlier oneBars: probability of each candidate next wordA toy model with 2,975 parameters, trained on 16 short sentences and running live in this page. Each cycle is one forward pass: the lifters show the real attention weights, the bars the real probabilities at the temperature on the knob. The pick samples one word, a slug with that word drops into the rail, and the pass runs again on the longer line until the model writes a full stop.

Attention, with real numbers

Attention is the idea that made all of this work. Two prompts that end in the same two words, "on the", need different endings. The only way to get it right is to look back at the animal. Here is where our trained model actually looks:

Where the last word looks before choosing the next one

the0%cat98%sat0%on2%the0%→mat100% surethe0%dog77%sat0%on22%the0%→rug100% surethe0%bird92%ate2%the6%→seed100% sure
Thicker arc, more attention. Percentages under each word are the real weights from this page's model.

Nobody programmed that. The model was only ever told "predict the next word". Looking back at the subject is the strategy it discovered because it lowers the error.

Run it yourself

The live model · 2,975 parameters

Bar under each word: how much the model is looking at it to choose the next one.

Click any candidate to choose it yourself. Slide the temperature down and the model plays it safe; slide it up and it takes chances.

What temperature does

The model never outputs a word. It outputs a probability for every word, and something has to choose. Temperature reshapes those probabilities before the choice. These are our model's real numbers for what follows the single word "the":

What comes after "the"? · the same scores at three temperatures

0%10%20%30%40%catdogbirdfishmousesunrug
0.5 · cautious1.0 · as trained2.0 · adventurous
Low temperature exaggerates the favourite. High temperature flattens the field, so unlikely words get picked more often.

How big the real ones are

Same design, more of everything. Note the scale is logarithmic: each step to the right is ten times more.

Learned numbers, from this page to a brain · log scale

This page2,975
Transformer, 201765 million
GPT-2 small, 2019124 million
nanochat, 2025561 million
GPT-3, 2020175 billion
Llama 3.1, 2024405 billion
Synapses in one brain~100 trillion
The scale is logarithmic: equal steps in length mean multiplying by the same amount. On a normal scale every bar except the last would be too short to see.
ModelYearParametersLayersNumbers per tokenContext
This page20262,9751168 tokens
Transformer, base [15]201765 million6 + 6512translation
GPT-2 small [16]2019124 million127681,024 tokens
GPT-3 [17]2020175 billion9612,2882,048 tokens
Llama 3.1 [19]2024405 billionopen weights, trained on over 15 trillion tokens

6Training: how it learns

Before training, every weight is random and the model outputs noise. Training is one short loop, repeated until the numbers are good.

Watch it happen

Press the button. Your browser will create a fresh model with random weights and train it on 16 sentences, right now. The chart is the loss. The three test prompts under it show what the model would say at that moment.

Train a language model in this tab

Random weights. Nothing learned yet.

1,500 steps · 89 examples per step

Three things to notice.

  1. It starts as noise. The first guesses are random words with low confidence.
  2. The easy patterns come first. Early on it learns that "the" is common. Only later does it learn which animal goes with which ending.
  3. It flattens out at 0.50, not zero. Some questions have no single right answer. After "the cat", four different words are equally correct, so a perfect model must stay uncertain there. The dashed line is the best any model can do on this text, and ours reaches it.

The build that shipped with this page

This is the real loss curve from the run that produced the weights you played with above: 1,500 steps, 2.3 seconds.

Loss during the build · lower is better

01230201003006001,0001,500 pure guessing among 31 words: 3.43 best possible on this text: 0.50
Steps along the bottom, stretched at the start because that is where everything happens. Loss up the side. Hover a dot. Final loss 0.4985; the floor is 0.4984.

The forward pass, in full

No library. This is the function that every number on this page comes from.

forward pass · plain JavaScript
function forward(w, ids) {
  const n = ids.length;
  const h = ids.map((id, i) => w.E[id].map((value, j) => value + w.Pos[i][j]));
  const last = h[n - 1];
  const q = mulVec(w.Wq, last);
  const keys = h.map((hi) => mulVec(w.Wk, hi));
  const values = h.map((hi) => mulVec(w.Wv, hi));
  const scores = keys.map((key) => key.reduce((sum, kj, j) => sum + kj * q[j], 0) / Math.sqrt(D));
  const attn = softmax(scores);
  const c = Array(D).fill(0);
  for (let i = 0; i < n; i++) for (let j = 0; j < D; j++) c[j] += attn[i] * values[i][j];
  const z = last.map((value, j) => value + c[j]);
  const pre = mulVec(w.W1, z).map((value, i) => value + w.b1[i]);
  const m = pre.map((value) => Math.max(0, value));
  const out = mulVec(w.W2, m);
  const z2 = z.map((value, j) => value + out[j]);
  const logits = mulVec(w.U, z2).map((value, i) => value + w.b[i]);
  return { h, last, q, keys, values, attn, c, z, pre, m, z2, logits };
}
Honest limits of the toy. Our model has 2,975 weights and saw 89 examples, so it has simply memorised 16 sentences. Real models are built the other way round, with far more text than weights. One well known study found the best balance at roughly 20 tokens of training text per parameter [22]. Memorising is not the goal. Generalising is.

7The factory: how a real one gets made

Scale that loop up by a factor of a hundred million and it stops being a script. It becomes a factory with several teams and a very large electricity bill.

The three stages of learning

Stage 1 · almost all of the cost

Pretraining: read everything

Predict the next token over trillions of tokens. This is where grammar, facts, reasoning patterns and style come from. The result can continue any text but has no idea it is supposed to be a helpful assistant.

Stage 2 · small and cheap by comparison

Fine-tuning: learn the job

Train a little longer on examples of the behaviour you want: question, then a good answer. The model keeps what it knows and learns the format, the tone and when to stop.

Stage 3 · shapes the character

Feedback: learn what people prefer

People compare answers, and the model is rewarded for the kind they prefer [25]. Some labs also train against a written set of principles so the model can critique and improve its own answers [26]. This is the nearest thing to the dopamine signal in section 3.

What it costs, from a laptop to a lab

BuildParametersHardwareTimeWhat you get
This page's model2,975One laptop core2.3 sUnderstanding
nanoGPT, GPT-2 size [20]124 million8 x A100about 4 daysWrites fluent English
nanochat [21]561 million8 x H100, rented3 h 51 min, about $92A small chatbot, end to end
Llama 3.1 405B [19]405 billionover 16,000 H100not publishedFrontier class, open weights

One number to feel the gap. An H100 rents for roughly $2 to $5 an hour [28, 29]. A cluster of 16,000 would be about $32,000 to $80,000 for every hour it runs. That is simple multiplication, not a published price, but it shows why only a handful of organisations pretrain at the frontier, and why everyone else builds on top of open weights.

8Brain vs LLM: the honest scorecard

The two are alike enough that each field keeps borrowing from the other, and different enough that calling an LLM "a brain" is wrong.

Where they really do meet

Shared ideaEvidence
Knowledge lives in connection strengthsTrue by design in a network, and the standard account of memory in the brain.
Prediction is the engineLanguage models trained only to predict the next word turn out to predict human brain activity during reading remarkably well. In one study the best models accounted for nearly all of the explainable variance in the recordings [32].
Layers build up from simple to abstractVision in the brain and in deep networks both go edges, shapes, objects [12].
Meaning is a direction in a spaceResearchers pulled millions of readable features out of one production model, including one for the Golden Gate Bridge [30].
Memory systems look relatedA transformer with a certain kind of position code behaves mathematically like models of the hippocampus [31].

Where they do not

Power draw · log scale

Your brainabout 20 W
One H100 GPUup to 700 W
16,000 H100 GPUsabout 11,200,000 W
The training cluster draws on the order of half a million times the power of the brain it is trying to imitate.
Your brainA large language model
LearnsAll the time, from single eventsOnly during training. After that the weights are frozen
MemoryHippocampus writes new memories in seconds [6]; sleep moves them into the cortex [34]The context window during a chat. Nothing is kept unless someone builds a memory around it
SignalTimed spikes, chemicals, slow wavesPlain numbers, one layer after another
WiringLoops everywhere; signals run both waysStrictly one direction per token
Data neededA child learns to speak from the talk around it, a tiny fraction of what a model readsTrillions of tokens [19]
EnergyAbout 20 WMegawatts to train
Body and needsHunger, pain, hormones, a life to protectNone. Its goals come from the person using it
ExperienceYou know there is something it is like to be youUnknown. Science has no test for this yet, and honest people disagree
The useful way to hold it. A language model is closest to one slice of a brain: a very large, very well read cortex, with no hippocampus, no body and no sleep. Which tells you exactly what to build around it. That is the next section.

9Building our own brain

You do not start by pretraining a giant. You start with the cortex someone else already paid for, and you build the missing parts of the brain around it. Then, step by step, you earn the right to train your own.

The brain around the model

Take the regions from Fig 1 and ask: which ones does a bare model lack? Each gap is a part you can build today with ordinary software.

Brain partWhat to buildHow
CortexA strong model, rented or openStart here · Rent a frontier model by the token, or run an open one on your own machine. Keep two, so you can switch.
Prefrontal cortexA standing brief in every chat: goals, plan, rulesStart here · One short text file, injected at the top of every conversation: who you are, what is on the plan, how you like work done.
HippocampusCapture, one fact per note, fast recallStart here · A folder of small dated notes plus a search over them and over past chats. Pull the few that match before the model answers.
CerebellumPractised routines the model can run without thinkingStart here · Write each repeated job down once as a step-by-step playbook the model loads when the job comes up. Two of them built this page.
Basal gangliaHabits: your corrections turned into defaultsNext · Every time you correct it, save the correction as a rule with the reason. Load the rules with the brief.
ThalamusA router that sends each job to the right workerNext · A small first step that reads the request and hands it to the right model, playbook or helper agent.
AmygdalaA flag for what needs a human nowNext · A rule that marks money, sends and anything hard to undo as "needs a person", and stops there until someone says yes.
SleepA nightly pass that replays the day into lasting memoryNext · A scheduled job that reads the day's chats, saves new facts, merges duplicates and fixes what went stale.
Our own weightsA small open model tuned on our own way of workingLater · Levels 1 to 4 below: train a small one to learn, then tune an open model on clean examples.

Five levels, each one small enough to finish

Level 0 · today · free

Understand one, end to end

You are here. The model on this page is complete: tokenizer, embeddings, attention, MLP, training loop, all in plain JavaScript inside this one page. Read its source, save a copy, change a sentence in the corpus, retrain, and see what changes.

read the whole model
view-source:https://anez.digital/llm
# search for makeTrainer: the model and its training loop

Level 1 · a weekend · about $100

Train a real small chat model

Rent one 8 x H100 machine for an afternoon and run nanochat from start to finish: about 561 million parameters, three hours and fifty-one minutes [21]. You end with a chat window talking to weights you trained. It will be charming and often wrong, and you will understand every stage of section 7.

get the code
git clone https://github.com/karpathy/nanochat
cd nanochat
# then follow the speedrun in its README

Level 2 · one to two weeks · one GPU

Tune an open model on how we work

Take a good open model and train LoRA adapters [23] on a few hundred to a few thousand clean examples: real tasks, the answer we would sign. Keep a held-out set so you can prove it got better, not just different.

Level 3 · ongoing

Finish the brain around it

Build the "next" rows in the table above, above all a nightly digest that turns the day into facts and a clear flag for what needs a human. This gives more than any amount of extra parameters, because it is the part a bare model does not have at all.

Level 4 · when levels 1 to 3 are boring

Release one

A small model with a clear purpose, trained only on text we have the right to use, published with its weights, its code, its test results and its known faults. See the next section.

Recommended next step

Level 1 this month: one afternoon, about $100, and the whole factory stops being a mystery.

10Giving it away, in a spirit of love

Everything on this page exists because people published what they knew: a 1943 paper, a 2017 paper, an engineer who put a whole training pipeline online for anyone with $100. Sharing is how this field moves. Doing it with care is how it stays good for people.

The release checklist

Why this order. Open and careful are not opposites. The safest thing to give away is something small, well understood, honestly labelled and trained on clean data. That is also the most useful thing for the next person who wants to learn.

11Twenty words, plainly

NeuronA cell that adds up its inputs and fires when they pass a threshold. In a model: one small sum and a simple curve.
Synapse · weightThe strength of one connection. Learning means changing these.
ParameterAny number the model learned. Weights are parameters. "405 billion parameters" means 405 billion learned numbers.
TokenA piece of text, often part of a word, with an ID number.
EmbeddingThe list of numbers that stands for a token: its position in a space of meaning.
Context windowHow many tokens the model can see at once. Its working memory.
AttentionThe step where each token gathers information from the other tokens, weighted by relevance.
MLPThe small feed-forward network inside each layer that processes what attention gathered.
Layer · blockOne round of attention plus MLP. Models stack dozens.
Residual streamThe running list of numbers for each token that every layer reads from and adds to.
LogitsThe raw scores for every possible next token, before they become probabilities.
SoftmaxThe formula that turns scores into probabilities that add up to 100%.
TemperatureA dial on the softmax. Low is safe and repetitive, high is varied and risky.
LossOne number for how wrong the model was. Training pushes it down.
BackpropagationThe method that works out each weight's share of the blame for the loss.
PretrainingThe long first phase: predict the next token over a huge amount of text.
Fine-tuningA short second phase on examples of the behaviour you want.
LoRAA cheap way to fine-tune: train a small add-on and leave the big model untouched.
RAG · retrievalLooking things up in an outside memory and putting them in the context before answering.
InferenceUsing a finished model. Weights frozen, one token at a time.

12Sources

Each one was opened and the number checked on 13 Nov 2026, except those marked classic reference, which are cited from the literature by their DOI.

  1. Herculano-Houzel S. et al., "The elephant brain in numbers", 2014. Gives 86 billion neurons in the human brain and 16.3 billion in the cortex, citing Azevedo 2009. pmc.ncbi.nlm.nih.gov/articles/PMC4053853
  2. Azevedo F. et al., "Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain", J. Comp. Neurol., 2009. About 69 billion neurons in the cerebellum. Classic reference. doi.org/10.1002/cne.21974
  3. Raichle M., Gusnard D., "Appraising the brain's energy budget", PNAS, 2002. About 2% of body weight, about 20% of energy. pmc.ncbi.nlm.nih.gov/articles/PMC124895
  4. OpenStax, Anatomy and Physiology, 12.4 "The Action Potential". Resting -70 mV, threshold -55 mV, peak +30 mV, about 2 ms. openstax.org
  5. Physics Today, "Sizing the synaptic cleft", 2007. A roughly 20 nanometre gap. physicstoday.aip.org
  6. Corkin S. et al., "H. M.'s Medial Temporal Lobe Lesion", J. Neurosci., 1997. The patient who could not form new memories. pmc.ncbi.nlm.nih.gov/articles/PMC6573687
  7. Language Log, "Firing and wiring", 2016. Traces "cells that fire together wire together" to Carla Shatz, 1992, not Hebb, 1949. languagelog.ldc.upenn.edu
  8. McCulloch W., Pitts W., "A Logical Calculus of the Ideas Immanent in Nervous Activity", 1943. mlanthology.org
  9. Rosenblatt F., "The perceptron: a probabilistic model for information storage and organization in the brain", Psychological Review, 1958. Classic reference. doi.org/10.1037/h0042519
  10. Rumelhart D., Hinton G., Williams R., "Learning Representations by Back-Propagating Errors", Nature, 1986. mlanthology.org
  11. Hochreiter S., Schmidhuber J., "Long Short-Term Memory", Neural Computation, 1997. Classic reference. doi.org/10.1162/neco.1997.9.8.1735
  12. Krizhevsky A., Sutskever I., Hinton G., "ImageNet Classification with Deep Convolutional Neural Networks", NIPS 2012. papers.nips.cc
  13. Mikolov T. et al., "Efficient Estimation of Word Representations in Vector Space", 2013. arxiv.org/abs/1301.3781
  14. Bahdanau D., Cho K., Bengio Y., "Neural Machine Translation by Jointly Learning to Align and Translate", 2014. arxiv.org/abs/1409.0473
  15. Vaswani A. et al., "Attention Is All You Need", 2017. Base model 65 million parameters, big model 213 million. arxiv.org/abs/1706.03762
  16. OpenAI, GPT-2 code and model release. github.com/openai/gpt-2
  17. Brown T. et al., "Language Models are Few-Shot Learners" (GPT-3), 2020. 175 billion parameters, 96 layers, 12,288 dimensions, 2,048 token context, about 300 billion training tokens. arxiv.org/abs/2005.14165
  18. Fortune, "ChatGPT is the fastest-growing app in history", 2 Feb 2023. The 100 million figure is a UBS estimate. fortune.com
  19. Meta, "Introducing Llama 3.1", July 2024. 8B, 70B and 405B; over 15 trillion tokens; over 16,000 H100 GPUs. ai.meta.com/blog/meta-llama-3-1
  20. Karpathy A., nanoGPT. Reproduces GPT-2 (124M) in about 4 days on 8 A100 GPUs. github.com/karpathy/nanoGPT
  21. Karpathy A., nanochat, "The best ChatGPT that $100 can buy", Oct 2025. 560,988,160 parameters, 3 h 51 min on 8 x H100. github.com/karpathy/nanochat/discussions/1
  22. Google DeepMind, "An empirical analysis of compute-optimal large language model training" (Chinchilla), 2022. A 70B model on 1.3 trillion tokens, about 19 tokens per parameter. deepmind.google
  23. Hu E. et al., "LoRA: Low-Rank Adaptation of Large Language Models", 2021. arxiv.org/abs/2106.09685
  24. Sennrich R., Haddow B., Birch A., "Neural Machine Translation of Rare Words with Subword Units" (byte pair encoding), 2016. arxiv.org/abs/1508.07909
  25. Ouyang L. et al., "Training language models to follow instructions with human feedback", 2022. arxiv.org/abs/2203.02155
  26. Bai Y. et al., "Constitutional AI: Harmlessness from AI Feedback", 2022. arxiv.org/abs/2212.08073
  27. NVIDIA, H100 product page. Up to 700 W, 80 GB. nvidia.com/en-us/data-center/h100
  28. Runpod, GPU cloud pricing (H100 from about $2 to $4.39 an hour, checked 13 Nov 2026). runpod.io/pricing
  29. Lambda, GPU cloud pricing (H100 $3.79 to $4.89 an hour, checked 13 Nov 2026). lambda.com/pricing
  30. Anthropic, "Mapping the mind of a large language model", May 2024. anthropic.com/news/mapping-mind-language-model
  31. Whittington J., Warren J., Behrens T., "Relating transformers to models and neural representations of the hippocampal formation", ICLR 2022. arxiv.org/abs/2112.04035
  32. Schrimpf M. et al., "The neural architecture of language: Integrative modeling converges on predictive processing", PNAS, 2021. pmc.ncbi.nlm.nih.gov/articles/PMC8694052
  33. Schultz W., Dayan P., Montague P. R., "A neural substrate of prediction and reward", Science, 1997. Classic reference. doi.org/10.1126/science.275.5306.1593
  34. McClelland J., McNaughton B., O'Reilly R., "Why there are complementary learning systems in the hippocampus and neocortex", Psychological Review, 1995. Classic reference. doi.org/10.1037/0033-295X.102.3.419