From CUDA to LLM

Chapter 1: A Thousand Hands - How a GPU Thinks

Chapter 1 illustration

The crate arrived on the back of the Tuesday vegetable truck, wedged between sacks of onions. A firm in the city had donated a computer to the gurukul, and the whole school carried it in from the road like a palanquin, Arjun directing, Meera warning everyone about the steps. Inside the grey box, visible through a vent, sat a second machine within the machine: a heavy slab of metal and fans, longer than Kabir's forearm. Meera read the sticker twice. "It's a graphics card. An NVIDIA GPU. My cousin has one. It's for games."

Guru-ji said nothing. He crossed to the shelf where he keeps the books nobody is allowed to touch with unwashed hands, and came back with the oldest one: The Complete Works of William Shakespeare, its spine held together by two generations of tape, bought in his college days for eleven rupees. He set it on the table beside the humming box.

"By the end of this year," he said, "this machine will write like this book."

Arjun wanted to know how fast the card was. Meera wanted to know how a toy for games could write plays, and whether anyone had verified this claim. But it was Kabir, sitting on the floor with his chin on his knees, who asked the question the whole year would hang on: "Guru-ji, why the graphics card? The computer already has a brain. Why does the extra one matter?"

Guru-ji sat down slowly, the way he does when the answer is long. "Because writing, it turns out, is arithmetic, beta. Terrible amounts of arithmetic. And this card has a thousand hands."

This chapter is the one lesson in the book with no code in it, and it may be the one you come back to most. Before you write a line of CUDA you need to understand how a GPU thinks, because a GPU does not think like the processor you have programmed all your life. It is not a faster CPU. It is a different answer to a different question, and every design choice you'll make in the next twenty-one chapters (how many threads to launch, how to lay out memory, why one kernel crawls and its twin flies) falls out of that answer.

By the end of this chapter you'll have the vocabulary the rest of the book is built on: latency versus throughput, streaming multiprocessors, warps, memory bandwidth, latency hiding. You'll know why neural networks and GPUs fit each other like hand and glove, and you'll have seen, from 10,000 feet, what the machine we're building actually is: a next-character guesser with 10.8 million adjustable dials, trained on Shakespeare until the guesses become verse. Everything we do, from Chapter 2's first kernel to the training run in Chapter 20, is a step toward one program, train_gpt.cu, and this chapter explains why that program wants a GPU at all.

There is a fair question to ask before we start: why learn this the hard way, kernel by kernel, when a few lines of Python can train a model this afternoon? Because the abstractions leak, and they always leak at the worst moment. When training crawls at half the speed it should, when the model won't fit in memory by a hundred megabytes, when the loss turns to NaN at three in the morning and the dashboard just stops - the person who can see through the framework down to the silicon is the one who fixes it, and everyone else waits for that person. The libraries the whole world runs on are themselves only CUDA, written by people who understood this chapter cold. By the last page you will have written that layer yourself, and you will never again have to wonder what actually happens when a language model "trains." You will know, because you will have built one from the first multiply to the last.

Two ways to be fast

Imagine two post offices.

The first is the village post office. It has one clerk, and she is astonishing. She knows every regulation by heart, anticipates your question before you finish asking it, and processes any letter, however strange, in seconds. If you walk in with one urgent envelope, there is no better place on earth. Her job is to minimize the time from your arrival to your departure. That time has a name: latency, the delay between asking for one thing and getting it.

Across town stands the second: the city's central sorting office. It employs five hundred sorters, and plainly, none of them is impressive. Each one does a single dull thing: read a pincode, toss the envelope in a bin. Any individual letter might sit around for an hour before someone touches it. But the building as a whole moves two million letters a day, a volume the brilliant village clerk could not handle in a year. Its job is to maximize the number of things finished per second: throughput.

Neither office is "faster." They are fast at different things, and this is exactly the split between a CPU and a GPU.

A CPU is the village clerk. Chip designers spend most of its silicon not on arithmetic but on making one thread of instructions finish as soon as possible: deep pipelines, branch predictors that guess which way your if goes before the data arrives, out-of-order engines that reshuffle your instructions on the fly, and megabytes of cache to avoid waiting for memory. A modern desktop CPU has perhaps 8 to 16 cores, each a small miracle of low-latency engineering. When you press a key and the editor responds instantly, thank the clerk.

The GPU is the sorting office. Its designers made the opposite bet: strip out the predictors, the reordering machinery, most of the cache, and spend nearly the entire chip on plain arithmetic units, thousands of them. Each individual unit is slow and simple. A single thread on a GPU runs worse than on a CPU, often much worse. But a big modern card has over ten thousand arithmetic units running at once, and if - this is the catch that the whole book teaches you to live with - if you can keep them all fed, the collective gets through work no CPU can touch.

Here is the trade drawn as silicon. The proportions are schematic, but they show the trade:

     CPU die (a few mighty cores)     GPU die (thousands of hands)
  +----------------------------+   +----------------------------+
  | CONTROL   | CONTROL  |     |   | ctrl| ALU ALU ALU ALU ALU  |
  | predict,  | predict, | big |   | ctrl| ALU ALU ALU ALU ALU  |
  | reorder   | reorder  |cache|   | ctrl| ALU ALU ALU ALU ALU  |
  |-----------+----------|     |   | ctrl| ALU ALU ALU ALU ALU  |
  |  ALU      |  ALU     |     |   | ctrl| ALU ALU ALU ALU ALU  |
  +----------------------------+   +-----+--- small caches -----+

On the left, the arithmetic units (ALUs, the circuits that actually add and multiply) are a minority shareholder in their own chip. On the right, they are the chip, with a sliver of control logic rationed out between them.

The question that decides everything, for any piece of work you might run, is: do I have one letter, or two million? Compiling a file, following pointer chains through a linked list, handling a keystroke: one letter each, and the clerk wins. Applying the same brightness adjustment to eight million pixels, or multiplying two large matrices: two million letters, and the sorting office wins by a landslide. Keep this question in your pocket. We will take it out again in a few pages when we meet the neural network, and the answer there is what justifies Guru-ji's whole plan.

The factory floor: SMs, warps, and lockstep

From here on, picture the GPU as a factory employing thousands of tiny workers. Each worker is a thread: one running instance of your program, with its own little set of variables. The workers are individually unimpressive, exactly like the sorters. The factory's genius is entirely in the organization, and the organization has two levels you must know.

The factory floor is divided into workshops. Each workshop is called a streaming multiprocessor, or SM, and it is the unit of real estate that matters on a GPU: a self-contained block with its own arithmetic units, its own pool of registers (the fastest storage there is, private to each thread), a small scratchpad memory shared by workers in the same crew (much more on this in Chapter 5), and a scheduler deciding who works next. A small laptop GPU might have 10 or 20 SMs; a big modern card has on the order of 100 to 150. When you hear that one card is bigger than another, SM count is usually what grew.

Inside each workshop, the workers are not free agents. They are organized into crews of exactly 32, and a crew is called a warp: 32 threads that execute in lockstep, meaning the hardware fetches one instruction and all 32 threads perform it at the same moment, each on its own data. The warp is the single most important idea in this chapter, so let me give it the picture it will carry for the rest of the book.

A warp is a rowing crew. Thirty-two rowers sit in one boat, and a coxswain calls the stroke: pull. All thirty-two oars enter the water together, sweep together, lift together. No rower decides her own rhythm; there is one call and one stroke, and the boat's speed comes precisely from that unanimity. What each rower does have is her own oar and her own patch of water - in our terms, her own registers and her own data. Same instruction, different data, one call.

The hardware people call this execution style SIMT: single instruction, multiple threads. One instruction fetch, one decode, and the cost of all that control logic is split 32 ways. That is how the GPU affords ten thousand arithmetic units: it doesn't give each one a brain, it gives each boat of 32 a single coxswain.

Here is the hierarchy in one picture. It's worth pausing on, because in Chapter 3 you'll be writing code that addresses every level of it:

GPU (the factory)
 |
 +-- SM 0 (a workshop; ~100+ per big card, typical)
 |    +-- warp: 32 threads rowing in lockstep
 |    +-- warp: 32 threads
 |    +-- ... dozens of warps resident at once
 |
 +-- SM 1
 |    +-- warps ...
 |
 +-- ...
 +-- SM 127

Lockstep has a price, and you should hear about it now even though we won't pay it until Chapter 6. Suppose your program contains a branch: if (x > 0) do_this(); else do_that();. In one warp, seventeen threads want the if side and fifteen want the else. A rowing crew cannot row in two directions at once. So the hardware runs the if side while the fifteen else rowers sit idle with their oars raised, then runs the else side while the seventeen sit idle. Both paths take turns; you pay for the sum. This is called warp divergence, and it means the GPU rewards programs where all 32 crew members genuinely want to do the same thing. Hold that thought; it becomes another reason neural networks fit so well.

One more piece of vocabulary while we're on the factory floor. You will read that a GPU has "10,000 CUDA cores." A CUDA core is just one arithmetic unit (one rower's arms, essentially), and the marketing number is simply how many exist across all SMs. It is not comparable to a CPU core, which is a whole clerk. I find it least confusing to ignore the word "core" on the GPU side entirely and count in warps and SMs, which is what the hardware actually schedules.

Memory bandwidth: the currency everything is priced in

So far this sounds like a simple story: more hands, more work. Arithmetic, though, is only half of any computation. The other half is carrying the numbers to the hands and carrying the results away, and this is where the real accounting of GPU programming lives.

Every number your program touches starts in the GPU's main memory: a few dozen gigabytes of it, sitting on the card next to the processor. The rate at which data can flow between that memory and the chip is called memory bandwidth, measured in bytes per second, and I want to put concrete numbers on it. These are typical round figures for current hardware, not benchmarks of any particular card:

A terabyte per second sounds like abundance. Watch how fast it turns into famine. A float, the 4-byte number that every value in our neural network will be, means 1 TB/s is about 250 billion floats per second arriving from memory. Meanwhile the same card's arithmetic units can execute on the order of 40 trillion floating-point operations per second (again, a typical round number for a current card). Divide one by the other: the hardware can perform about 160 operations in the time it takes to fetch one float.

Read that ratio again, because it is the economics of this entire book. If your program fetches a number, uses it once, and throws it away, your ten thousand hands will spend more than 99% of their time waiting for delivery. The arithmetic is effectively free; the fetching is what you pay for. Memory bandwidth is the currency, and every kernel we write from Chapter 4 onward will be priced in it: how many bytes did you move, and how much work did you wring out of each byte before letting it go?

This is not a hypothetical trap. In Chapter 4, Arjun will time a matrix addition and discover the mighty GPU beating the CPU by a factor of two instead of a hundred, and the reason is exactly this ratio: matrix addition does one operation per fetched number, so it runs at the speed of the memory, not the speed of the hands. And in Chapters 5 and 9 the single biggest optimization in the book - the one that takes our matrix multiplication from a crawl to within shouting distance of NVIDIA's own library - is nothing more than arranging for every fetched number to be reused dozens of times before it's dropped. Fetch once, share, reuse: that's where all the money is.

For now, one sentence to carry forward: a GPU is a memory-moving machine that does arithmetic on the side, and the programs that fly are the ones written by people who believe that.

Hiding latency: the trick that makes it all work

Bandwidth is how much the memory can deliver per second. There is a second, sneakier cost: latency, how long any single delivery takes. On a GPU, a request to main memory takes on the order of 400 clock cycles to come back (a typical figure; it varies, but hundreds is the right mental order). Four hundred cycles is an eternity. A warp that stops to wait for it is a rowing crew shipping oars mid-race.

A CPU attacks this problem with caches and prediction: the clerk keeps the popular forms in her top drawer. The GPU's answer is completely different, and it is the cleverest idea in the whole design.

It was at this point in the lesson, as it happens, that Arjun objected. "Four hundred cycles, Guru-ji! If every fetch stalls the crew for four hundred strokes, the thousand hands sit idle. The whole factory is a lie." Guru-ji looked delighted, the way he does when someone walks into the right question. "Beta, when your mother puts the dal on the fire, does she stand and watch it boil?" Arjun admitted she did not. "She chops onions for the next pot. And when those are done, she kneads the dough for the pot after that. The fire is as slow as ever. Tell me - standing in her kitchen, have you ever once noticed it being slow?"

That is the trick, exactly. The GPU doesn't try to make memory faster; it arranges to never be caught waiting for it. Each SM keeps far more warps resident than it can execute at any instant; this deliberate overbooking is called oversubscription. The moment warp 0 issues a memory request and stalls, the SM's scheduler switches to warp 1, which is ready to compute. When warp 1 stalls, warp 2 steps up. By the time the scheduler works its way back around, warp 0's data has arrived and it's ready to row again.

 time -->
 warp 0: [issue load]............(waiting)...........[compute]
 warp 1:    [issue load]............(waiting)..........[compute]
 warp 2:       [issue load]...........(waiting)..........[comp..
 warp 3:          [issue load]...........(waiting).........[co..
          ^
          the SM's arithmetic units: busy the whole time,
          because SOMEONE is always ready to row

The individual warp still waits 400 cycles: its latency didn't shrink by one cycle. But the hardware never idles, so the factory's throughput never dips. The slowness of the fire disappears because there is always another pot. This is called latency hiding, and it works for one reason worth underlining: switching between warps costs the SM nothing. On a CPU, switching threads means saving and restoring registers, which is expensive. On a GPU, every resident warp's registers stay physically parked in the SM's huge register file the whole time. Switching warps is just the scheduler pointing at a different row of an array. Zero cycles.

There is a consequence hiding in this design, and it explains a number that will look absurd when you first see it in Chapter 2. For latency hiding to work, the SM needs a deep bench: many warps resident and ready. Multiply that across 100+ SMs and the conclusion is that a GPU is only happy when you hand it tens of thousands to millions of threads at once. Launching a thousand threads on a GPU isn't parallelism; it's an insult. A CPU programmer's instinct says a million threads is madness. On this machine it is the design working as intended: you don't hire the sorting office to handle six letters.

Why neural networks fit like a glove

Take the one-letter-or-two-million question back out of your pocket, because it is time to say what a neural network actually does. You should brace for anticlimax.

A neural network is a function that turns a list of numbers into another list of numbers, built almost entirely out of one operation: multiply two numbers and add the result into a running total. That's the whole secret. Under every headline about machines that talk, there is a mountain of multiply-and-add. The network's behavior - everything it "knows" - lives in a large collection of stored numbers called parameters (I'll also call them dials, because turning them is what learning will mean). Running the network means combining your input with those parameters through millions upon millions of multiply-adds, arranged mostly into an operation called matrix multiplication, which gets all of Chapters 8 and 9 to itself.

Set that description beside everything this chapter has said about the machine, and watch the properties line up one by one.

The work is massive: our small-by-modern-standards model will do roughly ten million multiply-adds to guess a single character, and during training we'll ask for billions per second. Two million letters, and then some; the sorting office wins.

It is uniform, too: every one of those multiply-adds is the same tiny program applied to different data, which is the literal definition of what a warp does well. There are essentially no data-dependent branches inside the hot loops, so the rowing crews never split. All 32 oars want the same stroke, millions of strokes in a row.

And it is regular in memory: parameters and data sit in big contiguous arrays, walked in predictable order, exactly the pattern a 1 TB/s memory system is built to feed (Chapter 4 will show exactly how the warp must walk them). Better still, the arithmetic reuses data heavily: in a matrix multiplication each fetched number participates in many multiply-adds, which is precisely how you beat the 160-operations-per-float famine.

Embarrassing amounts of parallelism, no divergence, streaming access, high reuse. It is difficult to design, on purpose, a workload better shaped for this hardware. This is not a coincidence of history so much as a marriage of convenience that became a real marriage: neural networks got big because GPUs could run them, and GPUs grew in the direction neural networks pushed. The machine in the gurukul's classroom was built for rendering game worlds, but rendering was always the same story (one small program, millions of pixels), and Shakespeare, it turns out, arrives by the same road.

The two cards that started it all

The modern age of AI has a birth date, and it ran on gaming hardware. In the autumn of 2012 a neural network called AlexNet won the ImageNet image-recognition contest by a margin so wide it ended a decades-old argument about whether this whole approach could ever work. Its authors - Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto - trained it not on a supercomputer but on two NVIDIA GeForce GTX 580 cards, the kind sold to teenagers for smoother frame rates, plugged into an ordinary desktop. Each card had all of 3 GB of memory, so Krizhevsky spent months hand-writing CUDA to split the network across both of them. Nearly everything that followed - the models in the headlines, the companies racing to build them, the reason this book exists at all - traces back to those two gamers' cards kept fed with arithmetic. What you are about to build is a small, honest re-enactment of what they did.

What an LLM is, from 10,000 feet

We should also say what we're building, in plain words, before twenty-one chapters of detail. Here is the entire idea.

A large language model, or LLM, is a next-piece-of-text guesser. You show it some text; it outputs, for every possible next piece, a probability. In this book the pieces are single characters, from a vocabulary of exactly 65: the letters, digits, spaces, and punctuation that appear in Shakespeare (Kabir will count them on his fingers in Chapter 19). Show our finished model To be, or not to b and its job is to say: probability of e, very high; probability of q, vanishingly small.

That's it. That is the whole job description. Everything else follows from two moves.

The first move is training, which turns a random guesser into a good one. The model's guesses are computed from its parameters: for us, 10,770,816 of them, a number we will count by hand, dial by dial, in Chapter 14. At the start those dials are set randomly and the model's guesses are uniform noise. Training is a loop: show the model a slice of real Shakespeare, let it guess each next character, measure how surprised it was by the true ones (a single number called the loss; Chapter 12 builds it from scratch), then compute, for every one of the ten million dials, which direction to nudge it so the surprise shrinks, and nudge. The nudge-direction computation is backpropagation, the algorithm Kabir accidentally reinvents at a water tap in Chapter 16. Repeat a few thousand times, and the dials settle into values that encode the patterns of the text: that q is followed by u, that a line beginning ROMEO is followed by :, that clauses balance and scenes open with stage directions. Nobody programs those rules. They condense out of surprise-reduction, and watching it happen, loss ticking down on a screen before dawn, is Chapter 20.

Generation is the second move: once trained, ask the model for its next-character guess, pick a character from those probabilities, append it to the text, and ask again. One character at a time, the model writes. That loop, plus a few tricks for choosing well, is Chapter 21, the evening the children crowd around the screen while Guru-ji holds his book open to compare.

Two disclosures belong at 10,000 feet. First, the very large models in the news are this same design with the dials counted in billions instead of millions and the vocabulary made of word-pieces instead of characters; Chapter 22 walks the road from ours to theirs, and you will be surprised how short it is structurally. Second, everything in the training loop - the guessing, the surprise, the nudging of ten million dials - compiles down to exactly the workload of the last section: mountains of uniform multiply-adds over big regular arrays. The LLM is not merely suited to the GPU. The LLM as we know it exists because this machine got cheap enough to end up in a crate on a vegetable truck.

Meeting the machine

Enough theory; go say hello. On any machine with an NVIDIA card and its driver installed (Appendix B walks through renting one by the hour if the gurukul hasn't shipped you a crate), one command introduces you:

$ nvidia-smi
+-------------------------------------------------------------------+
| NVIDIA-SMI 550.54      Driver Version: 550.54     CUDA: 12.4      |
|--------------------------------+----------------------+-----------|
| GPU  Name                      | Memory-Usage         | GPU-Util  |
|================================+======================+===========|
|   0  NVIDIA GeForce RTX 4090   |    212MiB / 24564MiB |      0%   |
+-------------------------------------------------------------------+
| Processes:                                              GPU Memory |
|   No running processes found                                       |
+-------------------------------------------------------------------+

(Output abridged and illustrative; yours will differ in the particulars.) Three things on this little dashboard are worth reading with your new vocabulary. The memory line, 212MiB / 24564MiB, is the card's own on-board memory: the 24 GB that the 1 TB/s pipe feeds, entirely separate from your system's RAM; our model's parameters (10.8M floats, about 41 MB) will rattle around in it like a marble in a bucket, and Chapter 14 will budget precisely what training adds on top. GPU-Util is the fraction of time at least something is running on the SMs, the factory's are-the-lights-on gauge, and watching it pin at 100% during Chapter 20's training run is quietly satisfying. And the CUDA: 12.4 in the corner names the software platform this book teaches: CUDA, NVIDIA's system for writing ordinary C code that runs on those thousands of hands. From Chapter 2 on, it's where we live.

How a graphics card learned to think

For its first years a GPU could do exactly one thing: turn triangles into coloured pixels, very fast. NVIDIA coined the term "GPU" in 1999 with the GeForce 256, and for most of the decade that followed the chip was a magnificent idiot savant, useless for anything but drawing game worlds. What changed it was a question asked by a Stanford researcher named Ian Buck: if the card holds thousands of arithmetic units, why should they be allowed to touch only pixels? His research language, Brook, became the seed of CUDA, which NVIDIA shipped in 2007. CUDA is really just a permission slip - it lets you write ordinary C and aim it at the thousand hands, for any arithmetic you like. Everything in this book past the next chapter lives in the world that one permission slip opened.

The shape of the journey

The road ahead has three stretches, and it helps to see their shape once before we take the first step. (The table of contents has the chapter-by-chapter map; here I only want you to feel the arc.)

Part I, The Machine Awakens, teaches the machine's language: how to run code across those thousand hands, how to feed them without starving them, and how to measure honestly what they do. By its end the GPU will obey you, though it will not yet know anything. Part II, Forging the Parts, builds the organs of a mind one at a time and tests each in isolation before it joins the rest: the matrix multiply that is most of the arithmetic, the softmax that turns raw scores into choices, the normalizer that keeps a deep stack of signals from exploding, and attention, the part that lets the model weigh its own past. Then Part III, The Living Machine, wires those organs into a body, teaches it to learn from its own mistakes, feeds it Shakespeare, and sits with it through the night until, one evening, it writes a line of its own.

Here is the promise that holds the three parts together, and it is not the usual textbook bargain. Nothing you build is a toy that gets thrown away when the "real" version arrives. The softmax you sweat over in Part II is, line for line, the same softmax that picks the next character of Shakespeare on generation night; every piece you forge by hand is the exact piece the finished program runs. We are not going to study a language model. We are going to build one, and keep it.

Standing on two shoulders

This book is not the first to insist that a language model is small enough to understand in full. It follows a trail cut by Andrej Karpathy: first nanoGPT, which showed a working GPT in a few hundred readable lines, and then llm.c, which threw away the framework entirely and trained GPT-2 in nothing but C and CUDA. We walk that same road from zero, only slower and more patiently, stopping to build and test every organ by hand. Our model is deliberately small - ten million dials, not ten billion - but understand this one completely and the ten-billion version holds no mysteries, only bigger numbers. That is the whole promise of what follows: not a toy that resembles the real thing, but the real thing at a size a single person can hold in their head.

That evening, after the children had gone home, Guru-ji stayed a while in the classroom, listening to the machine's fans turn over in the dark. He took the Shakespeare down once more and set it on top of the grey box, dust jacket to sheet metal, as if introducing two old friends. "A thousand hands," he said to the empty room, "and not one thing to do yet. Well. Tomorrow we fix that."

What we built

No code yet: this chapter built the mental machine that all the real machines will be measured against. You now know how a GPU thinks, and you have the analogy vocabulary the whole book reuses.

Where we're headed. The machine is unpacked, the dream is stated, and so far the thousand hands have done nothing at all. Chapter 2 (First Light - Hello, CUDA) fixes that: we make the GPU print its first words, then put every one of those hands to work adding two arrays a million elements long. Time to make the machine do something. Anything.

Exercises

  1. Clerk or sorting office? For each task, decide whether it's latency-bound (wants the CPU) or throughput-bound (wants the GPU), and say why in one sentence: (a) responding to a keystroke in an editor; (b) applying the same color filter to every frame of an hour of video; (c) walking a linked list of 10,000 nodes; (d) computing the pairwise distances between 100,000 points; (e) parsing one JSON file.

  2. The famine, quantified. Using the chapter's typical numbers (1 TB/s of bandwidth, 40 TFLOP/s of arithmetic, 4-byte floats), verify the ~160 operations-per-float ratio. Then consider adding two arrays elementwise: each output element needs two floats loaded, one float stored, and one addition. What fraction of the card's arithmetic peak can this operation possibly use? Keep your answer; Chapter 4 measures it.

  3. The divided crew. A warp executes if (i % 2 == 0) path_a(); else path_b();, where i differs per thread and each path takes 100 cycles. Using the rowing-crew rule from this chapter, how long does the warp take? What if the condition were if (i < 32) and every thread in the warp had i < 32? What does this suggest about which branches are expensive?

  4. How deep a bench? Suppose a warp computes for 40 cycles, then issues a memory request that takes 400 cycles to return, then repeats. If the SM can execute one warp at a time, how many resident warps does it need so that its arithmetic units never sit idle? Sketch the timeline like the one in this chapter.

  5. Sizing the dream. Our model has 10,770,816 parameters, each a 4-byte float. How many megabytes is that? Training (you'll learn why in Chapters 16 and 18) needs roughly four times the parameter memory: parameters, their gradients, and two bookkeeping buffers of the same size. Does the whole training state fit in the 24 GB card from the nvidia-smi output, and by what margin? If you have access to any machine with an NVIDIA GPU, run nvidia-smi and find the same three numbers we read in this chapter.

Read Chapter 2 → Preview home