CIS6220: Deep Learning at Scale

Topics & Readings

Course Topics & Readings

This is the week-by-week curriculum: topics, readings, and key concepts. For the actual calendar — dates, holidays, midterms, and project deadlines — see the Schedule.

This content is tentative and may be adjusted based on class progress and interests. Readings should be completed before the listed class session.

The readings listed under each week are the assigned reading — read these before class. Deeper/more current/background material used in preparing each lecture lives separately in the instructor’s own lecture notes, not here, so this list doesn’t grow into more than students can reasonably read.

Week 0: Course Introduction, Deep Learning Review & Attention

This week sets the shared vocabulary and mental models — architectures, losses, optimizers, regularization, and attention — that every later week assumes. Deeper question: what are the basic building blocks of a neural network, and why do these particular choices (not others) tend to work?

Tuesday: Course Overview; How to Read a Paper; Deep Learning Review

Thursday: Attention

Readings:

Key Concepts:

Week 1: NLP and Attention Mechanisms

Week 0 already covered how attention works — self-attention, multi-head attention, positional encoding, masking, the core “Attention Is All You Need” mechanics. This week goes deeper on what breaks when you actually try to run that mechanism at scale: how do you make attention cheaper over long documents, and does the resulting model actually use everything in a long context well? Deeper question: attention’s basic mechanics don’t change with scale — what does?

Tuesday: Long Context — Sparse Attention and Whether It’s Used

Thursday: FlashAttention — Making Exact Attention Fast

Ongoing Theme: Evaluation (introduced here; revisited for reasoning models in Week 9 and for agents in Week 11)

Readings:

Key Concepts:

Not covered this week (moved out of the old plan — see lecture_notes/supplemental_papers.md for where this content is parked, pending a decision on which week it belongs to): RoPE/YaRN positional-encoding extrapolation, Kaplan/Chinchilla scaling laws, and the MHA→MQA→GQA→MLA KV-cache-compression lineage. Flagged for Lyle — these were the prior plan’s Week 1 content but don’t match what’s actually been assigned to students; they need a new home in the schedule.

Week 2: Computer Vision and Segmentation

Object detection and segmentation are about getting a model to answer “what is where.” YOLO and U-Net solved this efficiently for a fixed, closed vocabulary of object classes; SAM asked a harder question — can one model segment anything, given just a point or a box, with no retraining for new categories? Cool application: SAM-family models now power real-time video editing, robotics perception, and medical-image annotation pipelines.

Tuesday: Object Detection

Thursday: Segmentation

Readings:

Key Concepts:

Week 3: VLMs, multimodal models, diffusion models

How do you get a single model to connect a caption to a picture? CLIP’s contrastive trick — pull matching image/text pairs together in a shared embedding space, push mismatched pairs apart — became the substrate for nearly every text-to-image system since. Deeper question: how do you use language to control image generation, and where does “meaning” actually live in that shared embedding space? Aside: most of the current image-generation frontier (FLUX.2, Stable Diffusion 3.5) is still built on the same latent-diffusion + text-conditioning recipe taught here — the advances since 2022 are mostly in fidelity and control, not a new paradigm. Complication: Google’s “Nano Banana” line briefly tested the alternative — the original Nano Banana (Gemini 2.5 Flash Image, Aug 2025) generated images autoregressively, as LLM-style tokens, before Nano Banana 2 reverted to a diffusion decoder — a real, if short-lived, run at a genuinely different paradigm.

Tuesday: Multimodal Architectures

Thursday: Diffusion Models

Readings:

Key Concepts:

Week 4: MCP, Harnesses; Agentic AI

How should an AI system get access to knowledge and capabilities it wasn’t trained on? RAG answers this for knowledge — retrieve, then generate. MCP and tool use answer it for capabilities — a model that can act, not just recite. Deeper question: what should live inside the model’s weights vs. outside it in the surrounding system, and who controls that boundary?

Tuesday: RAG, Tool Use, and Harnesses

Thursday: Agentic AI — Single Agents and Multi-Agent Systems

Readings:

RAG is covered in lecture (retrieve-then-generate architecture, vector DBs/chunking, alternatives) but has no assigned reading this year.

See also (optional, background):

Key Concepts:

Week 5: Midterm 1

Only session this week is Midterm 1, Tuesday (covers everything through Week 4). Thursday is Fall Term Break — no class. No new content this week.

Week 6: Mechanistic Interpretability and Steering

Once a model can do something, how do you make it do a specific thing reliably — and how do you know what’s happening inside when you try? Deeper question: how should AI be steered — by changing its weights (fine-tuning), by changing its input (prompting/in-context learning), or by directly editing its internal activations?

Tuesday: Five Ways to Steer, and Looking Inside

Thursday: Steering with Activations

Readings:

See also (optional, background):

Key Concepts:

Week 7: Distillation; Efficient Architectures: MoE and Quantization

Distillation asks: can a small model learn to imitate a big one’s behavior, capturing most of its capability at a fraction of the cost? Cool current example: DeepSeek’s R1-distilled models — small models trained purely by fine-tuning on reasoning traces generated by R1, with no RL of their own — beat other similarly-sized models on math benchmarks. It’s a direct, current illustration of the distillation idea taught here, and a callback to Week 8’s reasoning models.

Distillation makes a big model tractable after training, for deployment. The other half of “making big models tractable” happens before training even starts, and comes up Thursday alongside MoE: how do you train a model too large to fit on one device in the first place? Two complementary strategies: shard the model itself across devices (tensor/model parallelism), or shard the optimizer state/gradients instead of replicating them everywhere (ZeRO/FSDP-style memory sharding). This is the technical foundation behind several of the “Training Efficiency” and “Memory Optimization” final project ideas.

As models get too big to run cheaply, two more questions emerge: how do you activate only part of a giant model per token instead of all of it (MoE), and how do you represent its weights with fewer bits without breaking it (quantization)? Cool application: DeepSeek-V3’s combination of MoE and low-precision training is a big part of why it could be trained for a fraction of the compute of comparably capable dense models.

Tuesday: Knowledge Distillation

Thursday: Mixture of Experts and Quantization

Readings:

See also (optional, background):

Key Concepts:

Week 8: Reinforcement Learning - Policy Optimization

How do you get a model to do the right thing when “right” can only be judged after the fact, not labeled in advance? RLHF and its descendants (DPO, GRPO) turn human or automated preference judgments into a training signal — and the same machinery, pointed at verifiable rewards instead of human preferences, is what trains today’s reasoning models.

Tuesday: Imitation Learning and PPO

Thursday: Preference Optimization and RL for Reasoning

Readings:

See also (optional, background):

Key Concepts:

Week 9: Reinforcement Learning - Off-Policy RL and Self-Play

Q-learning asks a basic RL question with a neural net standing in for the lookup table: can an agent learn a value for every action in every state purely from trial-and-error reward, and act greedily on it? Because it learns off-policy, it can reuse old experience, which PPO cannot. Soft Actor-Critic carries the same idea to continuous actions such as robot control. Self-play pushes trial-and-error further — no human demonstrations, no labeled data, just an agent improving against itself. That mechanism isn’t only for games: Week 8’s reasoning models already train on verifiable rewards, and Absolute Zero closes the loop, using self-play to generate a model’s own training curriculum with zero external data at all.

Tuesday: Off-Policy Value Learning — from DQN to Soft Actor-Critic

Thursday: Self-Play, Test-Time Compute, and RL for Agents

Readings:

See also (optional, background):

Key Concepts:

Week 10: Speech and Audio Processing

Speech recognition is “just” sequence-to-sequence transduction, but Whisper’s real lesson was that weak supervision at massive scale beat carefully-curated, strongly-supervised pipelines — a pattern that recurs throughout this course (more noisy data + scale often beats cleverness). Aside: by 2026 several open models (Qwen3-ASR, NVIDIA Canary) have overtaken Whisper on raw accuracy leaderboards, but the architectural lesson taught here — weak supervision at scale — is the point, not who currently tops a leaderboard.

Tuesday: Speech Recognition — From Audio to Text

Thursday: Speech Generation and Spoken Dialogue

Readings:

Key Concepts:

Week 11: LLMs for Research; Agentic AI — Orchestration & Evaluation

If an AI can read papers and write code, can it also do science — generate hypotheses, run experiments, and write them up? Deeper question: as AI-assisted (or AI-generated) papers flood the literature, how do we evaluate contributions when authorship and originality themselves become harder to pin down? This closes the loop on the course’s evaluation theme (Weeks 1, 8) at the level of science itself.

On the agentic side, the deeper question shifts once one agent can call tools: who decides what to delegate to which sub-agent, and how do you know the resulting swarm actually did the right thing?

Tuesday: AI for Scientific Research

Thursday: Agentic AI — Orchestration, Harnesses, and Evaluation (a current-state-of-the-art follow-up to Week 4’s RAG/MCP/agents material)

Readings:

Key Concepts:

Week 12: Video Understanding and Generation; Course Wrap-up

Video generation is diffusion plus one more axis of complexity: time. Deeper question: how do you keep a scene physically and character-consistent across hundreds of frames when you’re generating pixels, not simulating physics? Aside: Sora itself has already been superseded (Sora 2, then deprecated in 2026); current leaders (Veo 3.1, Seedance 2.0) are judged as much on audio/dialogue sync as visual fidelity — the frontier has moved from “can it generate video” to “can it generate a consistent audiovisual scene.” Closing the course here: this is the last new technical content before presentations, followed by a recap of the course’s major themes.

Tuesday: Video Understanding and Generation (merged into one session — the diffusion fundamentals were already covered in Week 3, so this is an application/extension, not a new foundation)

Course Wrap-up: recap of major course themes (architecture/data/loss choices, evaluation, scale) — see Key Takeaways below.

Readings:

Key Concepts:

Key Takeaways