🐦 Follow me on Twitter • ➡️ Jump to LLMs! 📧 Feedback

I struggled to grind for ML/AI interviews so I went back to the basics and created a list after careful research.

TorchLeet is broken into two sets of questions:

1. Question Set: A collection of PyTorch practice problems, ranging from basic to hard, designed to enhance your skills in deep learning and PyTorch.
2. LLM Set: A new set of questions focused on understanding and implementing Large Language Models (LLMs) from scratch, including attention mechanisms, embeddings, and more.

Note

Avoid using GPT. Try to solve these problems on your own. The goal is to learn and understand PyTorch concepts deeply.

Note

Yes. I used GPT to help write the code and I ended up testing it out myself as practise. I found the strategy to be super useful

Table of Contents

• Question Set
  • 🔵Basic
  • 🟢Easy
  • 🟡Medium
  • 🔴Hard
• LLM Set
• Advanced ML Systems Set (v3)
  • Classical ML from Scratch
  • LLM Decoding
  • LLM Inference & Systems
  • Modern Architectures
  • Alignment & Training
  • GPU Systems & Kernels
  • Company Quick-Reference
• Getting Started
  • 1. Install Dependencies
  • 2. Structure
  • 3. How to Use
• Contribution

Question Set

🔵Basic

Mostly for beginners to get started with PyTorch.

1. Implement linear regression (Solution)
2. Write a custom Dataset and Dataloader to load from a CSV file (Solution)
3. Write a custom activation function (Simple) (Solution)
4. Implement Custom Loss Function (Huber Loss) (Solution)
5. Implement a Deep Neural Network (Solution)
6. Visualize Training Progress with TensorBoard in PyTorch (Solution)
7. Save and Load Your PyTorch Model (Solution)
8. Implement Softmax function from scratch

---

🟢Easy

Recommended for those who have a basic understanding of PyTorch and want to practice their skills.

1. Implement a CNN on CIFAR-10 (Solution)
2. Implement an RNN from Scratch (Solution)
3. Use torchvision.transforms to apply data augmentation (Solution)
4. Add a benchmark to your PyTorch code (Solution)
5. Train an autoencoder for anomaly detection (Solution)
6. Quantize your language model (Solution)
7. Implement Mixed Precision Training using torch.cuda.amp (Solution)

---

🟡Medium

These problems are designed to challenge your understanding of PyTorch and deep learning concepts. They require you to implement things from scratch or apply advanced techniques.

1. Implement parameter initialization for a CNN (Solution)
2. Implement a CNN from Scratch (Solution)
3. Implement an LSTM from Scratch (Solution)
4. Implement AlexNet from scratch
5. Build a Dense Retrieval System using PyTorch
6. Implement KNN from scratch in PyTorch
7. Train a 3D CNN network for segmenting CT images Solution

---

🔴Hard

These problems are for advanced users who want to push their PyTorch skills to the limit. They involve complex architectures, custom layers, and advanced techniques.

1. Write a custom Autograd function for activation (SILU) (Solution)
2. Write a Neural Style Transfer
3. Build a Graph Neural Network (GNN) from scratch
4. Build a Graph Convolutional Network (GCN) from scratch
5. Write a Transformer (Solution)
6. Write a GAN (Solution)
7. Write Sequence-to-Sequence with Attention (Solution)
8. [Enable distributed training in pytorch (DistributedDataParallel)]
9. [Work with Sparse Tensors]
10. Add GradCam/SHAP to explain the model. (Solution)
11. Linear Probe on CLIP Features
12. Add Cross Modal Embedding Visualization to CLIP (t-SNE/UMAP)
13. Implement a Vision Transformer
14. Implement a Variational Autoencoder

---

LLM Set

An all new set of questions to help you understand and implement Large Language Models from scratch.

Each question is designed to take you one step closer to building your own LLM.

1. Implement KL Divergence Loss
2. Implement RMS Norm
3. Implement Byte Pair Encoding from Scratch (Solution)
4. Create a RAG Search of Embeddings from a set of Reviews
5. Implement Predictive Prefill with Speculative Decoding
6. Implement Attention from Scratch (Solution)
7. Implement Multi-Head Attention from Scratch (Solution)
8. Implement Grouped Query Attention from Scratch (Solution)
9. Implement KV Cache in Multi-Head Attention from Scratch
10. Implement Sinusoidal Embeddings (Solution)
11. Implement ROPE Embeddings (Solution)
12. Implement SmolLM from Scratch (Solution)
13. Implement Quantization of Models
    1. GPTQ
14. Implement Beam Search atop LLM for decoding
15. Implement Top K Sampling atop LLM for decoding
16. Implement Top p Sampling atop LLM for decoding
17. Implement Temperature Sampling atop LLM for decoding
18. Implement LoRA on a layer of an LLM
    1. QLoRA
19. Mix two models to create a mixture of Experts
20. Apply SFT on SmolLM
21. Apply RLHF on SmolLM
22. Implement DPO based RLHF
23. Add continuous batching to your LLM
24. Chunk Textual Data for Dense Passage Retrieval
25. Implement Large scale Training => 5D Parallelism

What's cool? 🚀

• Diverse Questions: Covers beginner to advanced PyTorch concepts (e.g., tensors, autograd, CNNs, GANs, and more).
• Guided Learning: Includes incomplete code blocks (... and #TODO) for hands-on practice along with Answers

---

Advanced ML Systems Set (v3)

A research-backed set of 30 questions covering what top AI companies actually ask in 2024-2025 interviews. Each question is tagged with the companies known to test that topic.

Note

These questions were compiled from real interview reports across Glassdoor, Blind, Reddit, and first-person accounts from candidates who interviewed at 15+ frontier AI labs. Questions are organized by interview role rather than just difficulty.

Classical ML from Scratch

Still asked at traditional FAANG companies (Google, Meta, Amazon, Uber, LinkedIn).

1. Implement Softmax from Scratch (numerically stable) (Solution) 🟢 Easy — Apple Meta Google Amazon
2. Implement K-Means Clustering in PyTorch (Solution) 🟢 Easy — Uber LinkedIn Google Amazon
3. Implement KNN in PyTorch (Solution) 🟢 Easy — Uber LinkedIn Meta
4. Implement Logistic Regression with Gradient Descent (Solution) 🟢 Easy — Google Meta Amazon

---

LLM Decoding

Core questions at frontier AI labs — Anthropic, OpenAI, DeepMind, Cohere, Perplexity.

5. Implement Contrastive Loss (InfoNCE) + CLIP Training Loop (Solution) 🟡 Medium — OpenAI Anthropic DeepMind Midjourney Apple
6. Implement 2D Positional Embeddings (Solution) 🟡 Medium — Anthropic DeepMind Midjourney Runway
7. Implement Top-p (Nucleus) Sampling (Solution) 🟡 Medium — Anthropic OpenAI DeepMind Perplexity Cohere
8. Implement Top-k Sampling (Solution) 🟡 Medium — Anthropic OpenAI DeepMind Cohere
9. Implement Beam Search for LLM Decoding (Solution) 🟡 Medium — Google DeepMind Meta Apple
10. Implement Temperature Sampling (Solution) 🟢 Easy — OpenAI Anthropic Cohere Perplexity

---

LLM Inference & Systems

Hot topic for LLM infrastructure roles at Perplexity, Together AI, Anyscale, Meta.

11. Implement LoRA on a Linear Layer (Solution) 🟡 Medium — Meta Google Anthropic OpenAI Databricks
12. Implement KV Cache for Autoregressive Generation (Solution) 🟡 Medium — Anthropic OpenAI Meta Perplexity Together AI
13. Implement Sliding Window Attention (Solution) 🟡 Medium — Mistral Anthropic Google DeepMind
14. Implement DPO Loss from Scratch (Solution) 🔴 Hard — Anthropic OpenAI DeepMind Meta
15. Implement PPO for RLHF (Solution) 🔴 Hard — Anthropic OpenAI DeepMind Meta
16. Implement Gradient Checkpointing (Solution) 🔴 Hard — Meta Google NVIDIA Tesla
17. Implement Mixture of Experts Layer (Solution) 🔴 Hard — Google DeepMind Mistral Databricks xAI
18. Implement Speculative Decoding (Solution) 🔴 Hard — Google DeepMind Anthropic Apple
19. Implement Continuous Batching for LLM Inference (Solution) 🔴 Hard — Perplexity Together AI Anyscale Meta

---

Modern Architectures

Cutting-edge topics at image-gen companies, research labs, and autonomous driving.

20. Implement DDPM (Denoising Diffusion) from Scratch (Solution) 🔴 Hard — Midjourney Runway Stability AI Adobe Google
21. Implement DDIM Sampling + Classifier-Free Guidance (Solution) 🔴 Hard — Midjourney Runway Stability AI Adobe
22. Implement Selective State Space Model (Mamba Block) (Solution) 🔴 Hard — DeepMind Google Anthropic
23. Implement Vision Transformer + MAE Pretraining (Solution) 🔴 Hard — Meta Google Apple Tesla Waymo
24. Implement Knowledge Distillation (Solution) 🟡 Medium — Google Apple Meta Qualcomm Tesla

---

GPU Systems & Kernels

For ML infrastructure and systems roles at NVIDIA, Meta, xAI, and frontier labs.

24. Write a Fused Softmax Kernel in Triton (Solution) 🟣 Expert — NVIDIA Meta Google xAI Tesla
25. Implement FlashAttention-2 in Triton (Solution) 🟣 Expert — NVIDIA Meta Together AI xAI
26. Implement FSDP (Fully Sharded Data Parallel) from Scratch (Solution) 🟣 Expert — Meta Google NVIDIA Anthropic xAI
27. Implement GRPO (DeepSeek-R1 Algorithm) (Solution) 🟣 Expert — DeepMind Anthropic OpenAI
28. Build a Complete LLM Inference Engine (Solution) 🟣 Expert — Perplexity Together AI Anyscale Fireworks AI
29. Implement Ring Attention for Long Contexts (Solution) 🟣 Expert — Anthropic Google Meta xAI

---

Company Quick-Reference

"If I'm interviewing at X, which v3 questions should I prioritize?"

Company	Priority Questions
Anthropic	5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 22, 26, 27, 30
OpenAI	5, 7, 8, 10, 11, 12, 14, 15, 27
DeepMind	5, 6, 7, 8, 9, 13, 14, 15, 17, 18, 22, 27
Meta	1, 2, 3, 4, 9, 11, 12, 14, 15, 16, 19, 23, 24, 25, 26, 30
Google	1, 2, 4, 9, 11, 13, 16, 17, 18, 20, 22, 23, 24, 26, 29, 30
Apple	1, 5, 9, 18, 23, 29
NVIDIA	16, 24, 25, 26
Midjourney / Runway / Stability AI	5, 6, 20, 21
Perplexity / Together AI / Anyscale	7, 10, 12, 19, 25, 28
Tesla / Waymo	16, 23, 24, 29
xAI	17, 24, 25, 26, 30
Mistral / Cohere	7, 8, 10, 13, 17

---

Getting Started

1. Install Dependencies

• Install pytorch: Install pytorch locally
• Some problems need other packages. Install as needed.

2. Structure

• <E/M/H><ID>/: Easy/Medium/Hard along with the question ID.
• <E/M/H><ID>/qname.ipynb: The question file with incomplete code blocks.
• <E/M/H><ID>/qname_SOLN.ipynb: The corresponding solution file.

3. How to Use

• Navigate to questions/ and pick a problem
• Fill in the missing code blocks (...) and address the #TODO comments.
• Test your solution and compare it with the corresponding file in solutions/.

Happy Learning! 🚀

Contribution

Feel free to contribute by adding new questions or improving existing ones. Ensure that new problems are well-documented and follow the project structure. Submit a PR and tag the authors.

Authors

Chandrahas Aroori 💻 AI/ML Dev

Caslow Chien 💻 Developer

Stargazers over time