Course Overview

This course covers the core architectures and training methods used in contemporary deep learning. Topics include transformers, vision, diffusion, multimodal models (VLM), deep reinforcement learning, mechanistic interpretability and cutting-edge research in areas such as speech and video generation, RAG, MCP and agents, and scientific paper writing. Students will read and critically discuss the seminal papers across deep learning and contribute to that literature.

Learning Goals

• Understand the major concepts used in deep learning, including LLM, VLM, audio and video architectures, training, fine tuning, reinforcement learning, and interpretability
• Be able to select appropriate deep learning architecture, training data, and loss function for a given problem
• Be able to critically read current deep learning papers
• Formulate, conduct and write up a research contribution in deep learning

Course Structure

Two lectures per week (mandatory attendance), weekly homework (vibe-coded pytorch) and paper critiques, two midterms, and a group final project.

Prerequisites

• Graduate machine learning: CIS5190, CIS5200 or equivalent

Grading

Component	Weight
Attendance	8%
Weekly Quizzes	2%
Homework	20%
Paper Critiques	20%
Final Project	30%
Midterms	20%

Major Course Themes

• Attention and Transformers: The backbone of modern deep learning
• Vision and Segmentation: YOLO, U-Net, SAM, diffusion models
• Multimodal Learning: CLIP, ViLBERT, cross-modal representations
• Fine-Tuning: LoRA, efficient adaptation
• Agentic AI and Harnesses: Tool use, external memory, Toolformer
• Reinforcement Learning: Policy optimization (DPO, PPO), Q-learning
• MoE, Distillation, and Quantization: Sparse models, knowledge transfer, low-precision inference
• Audio/Video: Conformers, Sora, speech models
• Interpretability and Steering: Probes, steering

Recurring questions will include: “Why this architecture and loss function?”, “How was the training data collected, cleaned and augmented?”, “What are the contributions of pretraining and fine tuning/post training?”, “What is internal to the network; what external; why?”, “How does this scale?”

Course Logistics

• Time: Tuesday/Thursday, 10:15–11:44 AM
• Location: 3401 Walnut St, Room 401B
• Office Hours: TBD

Course Policies

• Technology in class: No laptops. Students who need an accommodation should contact the instructor directly.

See the Schedule for the calendar of dates, midterms, and project deadlines, Topics & Readings for weekly content and readings, Resources for course infrastructure and materials, and Homework for homework guidelines.