Status: Draft


Differentiable Programming


  • Training Neural Networks depends on Gradient Descent
  • Gradient Descent depends on differentiation
  • Designing network architectures would require manual calculation of partial derivatives (tedious, error prone)
  • Automatic differentiation needed
  • First numerical solutions? (find evidence in literature!)
  • Recent solutions forward-backward paradigm? (find evidence in literature!)



  • Part 1: Working with existing frameworks
  • Part 2: How to implement such a framework yourself
  • part 3: Best practices in deep learning

Working with existing Frameworks

  • TODO: Name some frameworks here, including PyTorch, TensorFlow.


The following exercises also exist in the course Introduction to Machine Learning. They teach the core concepts of neural networks, including linear regression, logistic regression, cost functions and gradient descent. While teaching these core concepts, they are also suited to get familiar with PyTorch as deep learning framework for automatic differentiation. The following PyTorch exercises are suitable for beginners with machine learnign as well as beginners with PyTorch, though if you are not familiar with the concepts mentioned, we suggest to first complete the course Introduction to Machine Learning, where you will find the same exercise and their pendants using numpy only.

The exercise below is almost the same as the PyTorch exercise above but includes the implementation of activation and cost functions in numpy aswell as their derivatives and writing code to build the computational graph. This exercise is a first approach onto automatic differentation, but yet still tailored for the use case of fully connected feed forward networks only.


How to Implement a Deep Learning Framework

If you ever wondered how the deep learning frameworks realize automatic differentiation, this chapter is for you.

Best practices in deep learning

TODO: Add remaining exercises: