The Only Video You Need to Understand Generative AI

Every image model you've used starts from static. A picture of pure noise goes in, and step by step, a neural network cleans it up until something appears. For years, the network doing that cleanup was a U-Net. Today, almost every big image and video model uses a diffusion transformer instead, a DiT. So what actually changed, and why did the whole field switch?

Welcome back to the workshop. This one is for intermediate learners. You don't need any math beyond knowing that a neural network takes numbers in and gives numbers out. We'll do a one-minute refresher on diffusion and a quick tour of the U-Net, and then we'll spend most of our time taking a DiT apart, piece by piece.

First, the refresher. During training, we take a real image and add noise to it, a little or a lot, depending on a random timestep between zero and one thousand. At zero it's clean, and at one thousand it's pure static. The network sees the noisy image and the timestep, and it has one job: guess the noise that was added. We compare its guess with the real noise, and backpropagation nudges every weight so the next guess is a little better.

Generating an image runs that in reverse. We start from pure static. The network predicts the noise, we remove a little of it, and we repeat, often twenty to fifty times, until a clean image is left. The whole architecture question, U-Net or DiT, is only about that one box in the middle: the noise predictor.

One more piece before the architectures. Modern models don't denoise pixels directly. An autoencoder first squeezes the image into a latent, a small grid of numbers that keeps the meaning and drops the fine detail. A one thousand twenty-four pixel image becomes a grid of about one hundred twenty-eight by one hundred twenty-eight. It's a bit like my Game Boy screen: tiny, but packed with information. Both architectures work on that latent grid.

Now, the U-Net. Picture the letter U. On the way down, convolution layers shrink the grid step by step, trading detail for context, so the network can see the big picture. At the bottom, it's small and very abstract. On the way back up, it grows the grid again. And across the U, skip connections carry fine detail straight from the down side to the up side, so nothing sharp gets lost. The timestep is mixed into every block, and the text prompt comes in through cross-attention.

U-Nets were a great fit for images. A convolution only looks at a small neighborhood, which bakes in the assumption that nearby pixels belong together, and that helps it learn fast. It's what powered Stable Diffusion one point five and SDXL. But the design is hand-tuned: how many levels, where attention goes, how the channels grow. Making one bigger means redesigning it, without a clear recipe for which part to grow.

Meanwhile, transformers were taking over language, and they scaled so predictably that bigger reliably meant better. So in late twenty twenty-two, William Peebles and Saining Xie asked a simple question: what if we threw out the U and used a plain transformer as the noise predictor? Their paper, Scalable Diffusion Models with Transformers, introduced the DiT. Let's build one.

Step one is patchify. Transformers read sequences of tokens, not grids. So we cut the latent into little squares, usually two by two, and flatten each square into one token. The original DiT's thirty-two by thirty-two latent becomes two hundred fifty-six tokens. That's how my Game Boy draws its screen, out of tiles. Then each token gets a position signal, so the model still knows where every patch came from.

Step two is the transformer block, and its heart is self-attention. Every token looks at every other token and decides how much each one matters to it. A patch of sky can check in with a patch of ocean on the far side of the image, in a single layer. A U-Net's convolutions would need many layers to connect those two.

Mathematically, each token makes a query, a key and a value. You take a softmax of Q times K transposed, over the square root of the key size, multiply by V, then repeat for every head, in every layer, and... hang on. Let me just show you. Each token sends out a question. Every other token answers with how relevant it is. Then the token collects a blend of their values, weighted by those answers. That's attention.

Step three is conditioning. How does each block know the timestep, or what to draw? The DiT's best trick is called adaLN-Zero. The timestep, plus a label or a summary of the prompt, is turned into a scale, a shift and a gate for every block, which dial that block's work up or down. The gate starts at zero, so each block begins as a pass-through that changes nothing, and training ramps it up gently. That keeps big DiTs stable.

The full prompt comes in one of two ways. Some models, like PixArt and Wan, use cross-attention, where image tokens look up the text. Others, like Stable Diffusion three and Flux, use joint attention: text tokens and image tokens share one long sequence and attend to each other both ways. Step four is the exit. A final layer turns each token back into a patch, the patches are stitched into a grid, and out comes the prediction.

So why did the field switch? First, scaling. In the DiT paper, models that spent more compute per image made better images, very consistently, and the biggest one set a new state of the art on ImageNet. Second, it's the same machinery as large language models, so every speedup built for chatbots works here too. Third, flexibility. A wider image is just more tokens, and video is more tokens again. Sora and Wan cut clips into spacetime patches.

There's a catch, and it's the same one language models have. Attention compares every token with every other token, so doubling the tokens roughly quadruples that work. A one thousand twenty-four pixel image is about four thousand tokens. A few seconds of video can be tens of thousands. That's why so much current research goes into faster attention and smaller latents.

One bonus upgrade you'll find in most recent DiTs: they don't predict noise anymore. With flow matching, the network predicts a velocity, the direction to move from noise straight toward the image. Straighter paths usually mean fewer steps. Flow matching has its own episode on this channel, and it slots right into the same transformer.

Let's recap with a diff. The U-Net is a hand-designed U of convolutions, with skip connections and attention added in a few places. The DiT is a stack of identical transformer blocks working on patches, with conditioning dialed in by adaLN. Both answer the same question at every step. The DiT just scales better, handles any shape of image, and stretches to video.

So the next time a new model is described as a twelve billion parameter MMDiT, you'll know what that means. Patches become tokens, tokens pay attention to each other and to your prompt, and a stack of transformer blocks turns static into a picture, one step at a time.