Articles

• ---

  Weighted Skip Connections are Not Harmful for Deep Nets

  
  January 11, 2025
  

  The paper Identity Mappings in Deep Residual Networks has design mistakes leading to incorrect conclusions about training deep networks with gated skip connections. You should try gated/weighted skip connections yourself and see if they improve results on your problems.

• ---

  Bayesian Flow Networks (A Twitter Overview)

  
  August 16, 2023
  

  by Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson and Faustino Gomez

  This post is a compilation of a Twitter thread introducing our paper on Bayesian Flow Networks. It gives a very high-level summary of the system in the paper.

• ---

  ClipUp: A Simple and Powerful Optimizer for Distribution-based Policy Evolution

  
  December 8, 2020
  

  by Nihat Engin Toklu, Paweł Liskowski and Rupesh Kumar Srivastava

  ClipUp is a simple adaptive optimizer that makes it easier to solve certain optimization problems in practice.