Text Was Just the Opening Act
So far this course has been all about words. But generative AI never signed an exclusive deal with the alphabet. The same core idea (learn patterns from mountains of examples, then generate new things that fit those patterns) works on pixels, sound waves, and video frames too.
Instead of asking "what token comes next?", these models effectively ask "what image matches this description?", "what audio sounds like this voice?", or "what should the next frame of this video look like?" Different materials, same generative spirit.
Image Generation: Sculpting Pictures Out of Noise
Type "a corgi astronaut planting a flag on a planet made of cheese" and an image model paints it in seconds. The dominant technique, called diffusion, is delightfully weird: the model learns by watching images get progressively destroyed into static, and then learns to run that destruction backwards. At generation time it starts from pure noise and de-noises step by step, steered by your text prompt, until a picture emerges. Less like painting, more like sculpting: the corgi was "in the noise" the whole time, and the model chips away everything that isn't corgi.
Real-world sighting: product photos. An online shop can generate its kettle on a marble countertop, in a rustic cabin, at golden hour: no photographer, no plane tickets, no kettle-wrangling.
Audio Generation: Voices and Music
Audio models learn the patterns of sound, and they've gotten unnervingly good.
- Text-to-speech has gone from robotic GPS voice to warm, natural narration: pauses, emphasis, even a chuckle.
- Voice cloning can mimic a specific person's voice from a short sample. Amazing for an author narrating audiobooks in twelve languages; alarming for obvious reasons we'll hit hard in Lesson 10.
- Music generation produces full tracks (melody, instruments, vocals) from a text description like "upbeat lo-fi for studying, but make it slightly haunted."
Real-world sighting: voice assistants and customer-service lines that sound like people, and game studios generating thousands of lines of character dialogue without booking a recording studio.
Video Generation: The Hard Mode
Video is the boss level. A video is thousands of images that must stay consistent: the same dog, same lighting, same physics from frame to frame. Early attempts were pure nightmare fuel (six-fingered hands waving in melting rooms), but the technology is improving at an alarming rate: short clips from text prompts, animated product shots, whole scenes that flicker between "wow" and "why is the coffee pouring upward."
Real-world sighting: subtitles and dubbing. AI transcribes speech (audio in, text out), translates it, and can even re-voice the dubbed track: one video, effortlessly watchable in dozens of languages.
Multimodal: The Swiss Army Model
The newest frontier is models that handle several formats in one brain, the multimodal models. Show one a photo of your fridge contents and ask "what can I cook?" It sees the eggs and the sad half-pepper, reads your question, and writes a recipe. Snap a whiteboard diagram and get working code. Ask about a chart and get analysis.
| You provide | Model produces | Real-world example |
|---|---|---|
| Text | Image | Product shots, concept art |
| Text | Audio | Narration, music, voice assistants |
| Audio | Text | Transcription, subtitles |
| Image plus text | Text | "What's in this photo?", chart analysis |
| Text | Video | Short clips, animated ads |
How does one model juggle words AND pixels? Tokens, again. Lesson 4's big idea stretches further than you'd think: images and audio can also be chopped into token-like chunks, so different media become one shared language of sequences the model can predict.
Generated media still has tells. AI images historically fumbled hands, text on signs, and reflections; video adds physics glitches and objects that morph between frames. The tells shrink every year, though, so "it looks real" is officially retired as a verification strategy. Lesson 10 picks up that thread.
Want to peek behind the curtain? Even a fancy multimodal prompt is still just tokenized input under the hood:
๐ค Token visualizer
Loading tokenizerโฆ ๐งฎ
Checkpoint
The generative recipe (learn patterns, generate new examples) works on images, audio, and video too, and multimodal models combine several media in one brain.
Multimedia Quiz
๐ Quiz
Question 1 of 4How does a diffusion model generate an image?