Efficient AI

Course Logistics

Aditya Desai · July 29th 2026

Many thanks to all the resources listed in References on the main website. Almost all figures are taken from these resources; some were generated with Gemini. Some images are taken from Google Images and still need proper citation (in progress).

Size of AI is huge and it is only growing!

Model Sizes Over Time

Training models requires more and more compute

Training Compute Needs

Cost of training is already exorbitant

  • Cost of inference is generally much higher!
Cost of training is already exorbitant

What kind of questions are we going to ask?

  • Can I run this model in less memory?
  • Can I run this model faster?
  • Can I deploy this model with higher throughput?
  • Can I train a model in lower memory?
  • Can I train a model in less time?

In general:

  • Can I do "X" using less resource "Y"
  • X = "infer / train / etc"
  • Y = memory / compute / communication

Systems Solutions

  • Assumes that computational workload is fixed
  • Under this constraint, try to improve resource utilization
  • e.g. memory reduction -- offloading etc
  • e.g. lower latency -- parallelization, scheduling etc

We ...

  • Computational workload is NOT sacrosanct
  • We can change the workload while maintaining the "quality"
  • e.g. memory reduction -- we are allowed to change model parameters
  • e.g. lower latency -- we are allowed to change architecture or execution of model

We ...

  • Try to solve the fundamental bottlenecks that systems does not address
  • e.g O(N^2) attention computation -- system will not try to reduce the complexity
  • but we can!
  • Important: The bottlenecks that we should solve are to be informed by the systems research

Pre-requisites

No official prerequisites are listed, but you are expected to be familiar with:

  1. Machine learning in general — models, computational graphs, forward pass, backward pass
  2. Transformer architecture — FFN, MoE, self-attention layers
  3. Linear algebra, Calculus basics
  4. Random variable analysis — expectation, variance, MSE of a RV

Course Elements

  • Part 1: Lectures
  • Part 2: Seminar
  • Part 3: Project
  • Part 4: Midterm and Final Paper Pen Exam

Grading

  • Project: 60%
  • Seminar: 20%
  • End sem: 10%
  • Mid sem: 10%

Lectures will try to cover topics:

(we will see how much we can cover)
  • Pruning
  • Quantization
  • Low-rank
  • Projection-based learning
  • Knowledge Distillation
  • Parameter Efficient Fine-Tuning
  • Execution efficiency

What This Course Is Not About

  • System optimizations (parallelism, scheduling, etc.)
  • CUDA programming
  • LLM inference Engines
  • For those topics, take Systems for ML (Prof. Mythili)

Seminars

Efficiency topics in LLMs

  • Sparse attention (decode)
  • Sparse attention (prefill)
  • KV cache compression
  • Speculative decoding
  • Attention alternatives
  • Semantic Caching / Model Routing / Miscellaneous topics
  • and more.

Seminar Logistics

  1. Sign up for topics (FCFS) — Link to Sheet
  2. Up to 5 students per lecture — everyone reads all the papers and compiles a joint presentation / lecture.
  3. Seminars start after a few lectures — you have ample time to read on the topic.
  4. Let D = day of presentation:
    • D−7: meet to discuss / whiteboard the papers
    • D−3: slide deck review
  5. Evaluated subjectively on participation in (D−7) discussions, (D−3) slide deck review, and the day-D presentation.

Project: Description & Resources

  • Semester-long: develop a model compression recipe — given a model and target domain, compress while keeping performance.
  • Model: Qwen-3.5-4B
  • Domain: Math
  • Starter Code: https://github.com/skylight-org/efficient_ai_CS6013_2026Fall
    • sets up some basic starter code for eval.
    • Also, has a basic gptq quantization code to begin with.

Resources

  • Reuse papers / GitHubs as starting points; iterate as we cover techniques in class.
  • Resources: Kaggle, Google Colab, Molab, etc. for development.

Project: Goals

  • Track 1: evaluate by quality at different memory footprints alone (checkpoint size). (no CUDA compatibility required)
  • Track 2 : Same as above but techniques must be CUDA-speedup-capable in principle. Explain how; no kernels required. (while submitting, you can mark your submission for track 2)
  • Track 2+ (bonus, optional): CUDA kernels + measured submodule speedups at scale. Open to mid-term top performers; A100/H100 access; used to push grades up.

Leaderboard

  • We will maintain a regularly updated leaderboard of the best performing submissions for each track.
  • This will also give you visibility of the best scores achieved.

Project: Evaluation & Timelines

  • Two checkpoints: Sep 15 (mid-term) and Nov 15 (end-term), 2026.
  • Each eval: (1) leaderboard position (2) Report
  • Mid-term top performers → eligible for Track 2+.
  • We will select few submissions and have the students present their ideas at mid-term, which can be used by others.
  • Slack Days: No Slack days.

Project: Weightage

Project = 60% of course grade. Track 2+: up to +5% over the total grade.

Eval 1 — 50% Eval 2 — 50%
Leaderboard Report Leaderboard Report
60% 40% 60% 40%
  • Reports: PDF, ≤3 pages of text (figures/tables after). Over-length → not evaluated.
  • (Note: Ideas in report can become important differentiators, since it might get difficult to beat the SOTA and it is quite possible that all scores will concentrate around that)

Project: Submission Details

  • Max 1 submission / week;
  • Code in Github repo (Use starter as template and maintain private repo with TAs and instructor)
  • Submission = huggingface compressed model + compress/decompress code in github.
  • Keep pyproject.toml up to date for dependencies.

Project Details

In a couple of weeks, TAs will show you:

  1. How to make submissions
  2. How the leaderboard will look , where to find it etc.

If you have difficult time setting up / using GPU resources, please use TA hours to get help.

Project: AI Policy

  • Code / ideation: any-AI — use any AI tools. Interesting usage is encouraged and rewarded when reported.
  • Report writing: strict NO-AI — not even for grammar or spell-check.
  • Any AI use in the report → grade of 0 for the entire project.

Project: Honor Code

Sign the honor-code pledge on every submission (end of the submission form):

“I hereby declare that this submission is my own work and I have not shared my code or artifacts with any other student. I understand that any violation of the honor code will result in a grade of 0 for the project.”

All details in: Project PDF

Office Hours

  • Instructor: Wednesday (after class) 12:30–1:30
  • TAs: to be added

All discussions regarding the course are expected to take place during course office hours.