Material for the class on debugging deep learning models at Data Science Retreat.

Useful resources for debugging

• Debug a deep learning network: A nicely written blog post on how to debug a deep learning model.
• Recipe for training neural networks: An Andrej Karpathy's blog post on how to train neural nets.
• Troubleshooting deep learning models: A great video to watch where many debugging steps are summarized in a comprehensive way.
• Machine learning yearning: A practical manual written by Andrew Ng, which gives a full overview how one should structure a deep learning project.
• Bayesian optimization: Bayesian optimization for hyperparameter search.
• Dive into deep learning: An amazing website with theory of deep learning, code examples, exercises. It starts from basics and covers the most advanced topics in DL.
• Deep learning: The Bible of deep learning written by Ian Goodfellow and Yoshua Bengio and Aaron Courville. If you wanna go deep, this book is a must to read.

Some other resources, explaining notebook materials

• Understanding LSTM networks: A clear written post on LSTMs. Good for a quick overview and recalling some basics.
• Visualizing What Batch Normalization Is and Its Advantages: A blog post explaining batch normalization in detail.
• Gradient clipping: An article on gradient clipping in RNNs.
• Teaching Large Language Models to Self-Debug: A paper about Rubber Duck self-debugging with LLMs.
• LangChain Docs: LangChain documentation page.

Local installation manual

• Clone the repo:

git clone https://github.com/stanislav-chekmenev/debugging-dl-models

• Install VS Code:

  • Follow this link for the installation details.
  • If you are on Linux Ubuntu starting from 16.04, then please use the following command:

   sudo snap install --classic code

• Create a new project and a virtual environment for it.

  • Start VS Code and choose the debugging-dl-models/project directory as the root directory for the VS code project!

  • Create and activate a new virtual environment with python 3.12. Feel free to use any virtual environment of your choice. I prefer python3-venv for its simplicity. It also works good with Linux.

    Conda:

     conda create --name <name> python=3.12
     conda activate <name>

    Virtual environment. For Ubuntu 24.04:

     python3 --version # make sure you have python 3.12 installed
     sudo apt install python3-venv
     python3 -m venv <path/to/venv>
     source <path/to/venv>/bin/activate

• Upgrade pip, it might be of an old version.

   pip install pip --upgrade

• Install requirements.

   cd debugging-dl-models/project
   pip install -r requirements.txt
   pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu # for Linux & Windows
   # pip install torch==2.8.0 # MacOS

• Test your environment installation and check if the requirements are installed correctly, as well. Run the test_env.py from your project root directory that must be already set to debugging-dl-models/project. The script has to be run in the activated environment, so do not forget to activate your env!

   cd debugging-dl-models/project
   python test_env.py

• If everything is correctly installed and the env is activated, the script should return this text:

   Torch installed successfully. Torch version: 2.8.0+cpu
   Scikit-learn installed successfully. Scikit-learn version: 1.7.0
  

Congrats, everything is ready for our class!

Class material

• Colab 0: Introduction to debugging deep learning models
• Colab 1: Implementation recipe of a bug free model
• Colab 2: Most common bugs