Software I like: Neural networks

This page of personal notes was started because I had to look up ‘loss function’.

Software

Frameworks

PyTorch: open-source, started by Facebook; is based on Torch, a scientific computing framework written in C/CUDA
TensorFlow: open-source, started by Google; TensorFlow 2.0 plus the Keras interface for Python has features adopted from PyTorch; includes the TensorBoard visualization toolkit

Comparisons: PyTorch vs TensorFlow (2024 May 8)

Making network diagrams

Alexander Lenail’s NN-SVG (on GitHub): Web-based creation of SVG for publication-ready NN-architecture schematics in 3 different styles
TensorBoard can produce graph diagrams. It comes with TensorFlow and can be used with PyTorch.

Tips

model.summary(): the None values in the Output Shapes of the layers are a reminder that the model expects the input to have a batch size as the outermost dimension (ref)

Terminology

Loss function	Error	Used as a cost function in minimization
ReLU (rectified linear unit)	Rectifier	faster than sigmoid and tanh (Nair & Hinton, 2010; Krizhevsky et al., 2012) and also fixes the ‘vanishing gradient problem’ (https://www.baeldung.com/cs/sigmoid-vs-tanh-functions)

Architectures

Encoder-decoder vs. U-net

Last modified: 2025-01-08 12:53:38