Learning Discrete Data with Harmoniums: Part I, The Essentials
From the Archives: Generative AI in the ‘00sI want to take you back to the last generative AI episode, in the early ’00s. During this time, Geoff Hinton, one of the founding fathers of deep learning, published an influential paper detailing the contrastive divergence algorithm . This discovery allowed Smolensky’s harmonium — which Hinton called the restricted Boltzmann machine — to be trained efficiently. It was soon realised that this model could be used for all sorts of purposes: initialising a feed-forward neural net…