Efficient AI

Project

Semester-long model compression recipe: given a model and target domain, compress while preserving performance. This year’s setup uses Qwen-3.5-4B on the math domain. Checkpoints on Sep 15 and Nov 15, 2026.

Description: project/main.pdf; Project updated instructions
Starter code: skylight-org/efficient_ai_CS6013_2026Fall
Leaderboard: Weekly project leaderboard

Topic Sheet

Spreadsheet: Seminar topic sheet; Seminar Group Allocation Sheet

Course schedule

Lecture materials will be linked here as they become available.

Date	Lecture	Topic	Slides	Notes
7/29/2026	1	Course Logistics, DL Overview	Logistics · DL review	—
7/31/2026	2	Pruning (OBS, OBD, IMP, Results from Han.et.al)	Pruning I	pruning_a_linear_model pruning_a_model
8/5/2026	3	Pruning (OBS, OBD complexity, Structured Pruning, Hardware Support)	Pruning II	hessian_complexity
8/7/2026	4	Pruning (SparseGPT, Wanda, N:M)	Pruning III	—
8/12/2026	5	Tail Bounds, GPU programming model	GPU programming model	tailbounds
8/14/2026	6	Visiting Lecture: Piyush Sawarkar, BharatGen	RL: From Imitation to Reasoning to Agentic
8/19/2026	7	Bonsai, Sampling(I)	—	bonsai sampling
8/21/2026	8	Sampling(II)	Sampling	sampling
8/26/2026	9	Holiday(Id-E-Milad)	—	—
8/28/2026	10	Seminar: Sparse Attention(Decode)	Sparse Attention Seminar	—
9/2/2026	11	WRS, Quantization Basics, Spill Over Seminar	Sampling (WRS) Quantization Basics	—
9/4/2026	12	Seminar: KV Quantization(I)	KV Quantization Seminar	—
9/9/2026	13	Rotation, Stochastic rounding, Granularity	Quantization Basics	—
9/11/2026	14	Seminar: KV Quantization(II)	KV Quantization Seminar	—
9/16/2026	15	No Class (Exam Week)	—	—
9/18/2026	16	No Class (Exam Week)	—	—
9/23/2026	17	MidSem paper cribs	—	—
9/25/2026	18	Seminar: Sparse Attention (Prefill/ Native) I	Sparse Attention (Prefill)	—
9/30/2026	19	Seminar: Sparse Attention (Prefill / Native) II	Sparse Attention (Prefill)	—
10/2/2026	20	Holiday(Gandhi Jayanti)	—	—
10/7/2026	21	Quantization in LLMs	—	Quantization, Part II
10/9/2026	22	Seminar: Speculative decoding	Speculative decoding Seminar	—
10/14/2026	23	Projections (JL Lemma, Subspace Embeddings, CountSketch)	—	Subspace Embeddings
10/16/2026	24	Seminar: Projection learning	Projection learning Seminar	—
10/21/2026	25		—	—
10/23/2026	26	Seminar: Linear Attention	Linear Attention Seminar	—
10/28/2026	27		—	—
10/30/2026	28	Seminar	—	—
11/4/2026	29		—	—
11/6/2026	30	Seminar	—	—

References

Selected readings linked by topic.

Topic	References
Deep Learning Review	Vutukuru and Rathi. “Overview of Deep Learning Concepts.” Chapter from Systems for Machine Learning. Torralba, Isola, and Freeman. “Foundations of Computer Vision.” MIT Press, 2024. Zhang et al. “Dive into Deep Learning.” Goodfellow, Bengio, and Courville. “Deep Learning.” MIT Press, 2016.
Tail Bounds and GPU programming model	Mitzenmacher and Upfal. “Probability and Computing.” Chapters 3–4. NVIDIA. “CUDA Programming Guide.” Chapter 1.
Pruning	Cerebras. “Harnessing the Power of Sparsity for Large GPT AI Models.” Cerebras. “Introducing Sparse Llama: 70% Smaller, 3x Faster, Full Accuracy.” Frankle. “The Lottery Ticket Hypothesis: On Sparse, Trainable Neural Networks.” PhD diss., MIT, 2023. Frantar and Alistarh. “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot.” arXiv:2301.00774, 2023. Gale, Elsen, and Hooker. “The State of Sparsity in Deep Neural Networks.” arXiv:1902.09574, 2019. Han, Mao, and Dally. “Deep Compression.” arXiv:1510.00149, 2016. Han et al. “Learning Both Weights and Connections for Efficient Neural Networks.” arXiv:1506.02626, 2015. Hassibi and Stork. “Second Order Derivatives for Network Pruning: Optimal Brain Surgeon.” NeurIPS 1992. He et al. “AMC: AutoML for Model Compression and Acceleration on Mobile Devices.” ECCV 2018. Kang et al. Automatic Pruning Discovery for Large Language Models. Kolawole et al. “Everybody Prune Now: Structured Pruning of LLMs with Only Forward Passes.” arXiv:2402.05406, 2026. Kuo. “vLLM or Ollama on Blackwell? Benchmarks, Landmines, and What Agents Actually Need.” 2026. Lanczos. “An Iteration Method for the Solution of the Eigenvalue Problem of Linear Differential and Integral Operators.” J. Res. Natl. Bur. Stand., 1950. LeCun, Denker, and Solla. “Optimal Brain Damage.” NeurIPS 1989. Liu et al. “Ministral 3.” arXiv:2601.08584, 2026. Luo et al. “Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis.” arXiv:2501.12084, 2025. Ma, Fang, and Wang. “LLM-Pruner: On the Structural Pruning of Large Language Models.” NeurIPS 2023. NVIDIA. “Exploiting NVIDIA Ampere Structured Sparsity with cuSPARSELt.” 2020. Patel and Chen. “NVIDIA Tensor Core Evolution: From Volta To Blackwell.” SemiAnalysis. Pearlmutter. “Fast Exact Multiplication by the Hessian.” Neural Computation, 1994. Rumelhart, Hinton, and Williams. “Learning Representations by Back-Propagating Errors.” Nature, 1986. Sun et al. “A Simple and Effective Pruning Approach for Large Language Models” (Wanda). 2024. Wen et al. “Learning Structured Sparsity in Deep Neural Networks.” arXiv:1608.03665, 2016. Yang et al. “Let LLM Tell What to Prune and How Much to Prune.” ICML 2025. Zheng et al. “Enabling Unstructured Sparse Fine-Tuning and Inference for Foundation Models on Wafer-Scale Engine.” SC Workshops 2025.
Quantization	Dettmers et al. “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.” arXiv:2208.07339, 2022. Han, Mao, and Dally. “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding.” arXiv:1510.00149, 2016. Frantar et al. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.” arXiv:2210.17323, 2023. Dong et al. “HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision.” arXiv:1905.03696, 2019. Lin et al. “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.” arXiv:2306.00978, 2023. Xiao et al. “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.” arXiv:2211.10438, 2022. Zadeh et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” arXiv:2404.00456, 2024. Ben-Basat et al. “Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven.” arXiv:2605.06014, 2026. Jégou, Douze, and Schmid. “Product Quantization for Nearest Neighbor Search.” IEEE TPAMI, 2011. Grootendorst. “A Visual Guide to Quantization.” ggml-org. “GGUF documentation.” GitHub. NVIDIA. “Model Quantization: Concepts, Methods, and Why It Matters.” Technical Blog, 2025. NVIDIA. “Floating-Point 8: An Introduction to Efficient, Lower-Precision AI Training.” Technical Blog, 2025. NVIDIA. “Per-Tensor and Per-Block Scaling Strategies for Effective FP8 Training.” Technical Blog, 2025. NVIDIA. “NVIDIA Hopper Architecture In-Depth.” Technical Blog, 2022. NVIDIA. “Using FP8 and FP4 with Transformer Engine.” Transformer Engine documentation. NVIDIA. “MXFP8.” Transformer Engine documentation. PyTorch. “HadaCore: Tensor Core Accelerated Hadamard Transform Kernel.” PyTorch. “Quantization-Aware Training for Large Language Models with PyTorch.”
Sparse Attention (decode)	Desai et al. “HashAttention: Semantic Sparsity for Faster Inference.” arXiv:2412.14468, 2024. Chen et al. “MagicPIG: LSH Sampling for Efficient LLM Generation.” arXiv:2410.16179, 2024. Zhang et al. “PQCache: Product Quantization-based KVCache for Long Context LLM Inference.” arXiv:2407.12820, 2024. Desai et al. “vAttention: Verified Sparse Attention.” arXiv:2510.05688, 2025. Hooper et al. “Squeezed Attention: Accelerating Long Context Length LLM Inference.” arXiv:2411.09688, 2024. Joshi et al. “Inference Time Context Sparsity: Illusion or Opportunity?” arXiv:2605.24168, 2026.

Efficient AI

Course details

Instructor

Teaching assistants

Project

Topic Sheet

Course schedule

References