The SuperVision team and the ImageNet win
The team led by Geoffrey Hinton, alongside Alex Krizhevsky and Ilya Sutskever, won the 2012 ImageNet Large Scale Visual Recognition Challenge with AlexNet. This deep convolutional neural network achieved a top-5 error rate of 15.3%, which beat the runner-up by 10.8 percentage points. The model classified images into 1,000 distinct object categories. The training set contained 1.2 million images from the ImageNet dataset. Fei-Fei Li and her collaborators created this dataset starting in 2007 to advance visual recognition through large-scale data. The dataset originally contained over 14 million labeled images across 22,000 categories. Hinton, who earned his PhD in artificial intelligence in 1978, previously worked with David Rumelhart and Ronald J. Williams to develop the backpropagation algorithm in 1986. This research group at Carnegie Mellon University favored the connectionist approach, which suggests that neural networks can learn logic and grammar directly from data. This position stood in opposition to symbolists who advocated for explicitly programming knowledge and rules into AI systems. Hinton’s supervisor, Christopher Longuet-Higgins, favored the symbolic AI approach over neural networks. The SuperVision team submitted an ensemble of seven AlexNets to the contest, including five standard models and two variant models trained on the ImageNet Fall 2011 release.
Architecture and the GPU shift
The SuperVision team trained AlexNet using two Nvidia GTX 580 GPUs in Krizhevsky’s bedroom. Because the entire network structure did not fit the 3GB VRAM of a single Nvidia GTX 580, the team split the architecture into two copies to run on two separate GPUs. This setup required significant computational expense. The model contains 60 million parameters and 650,000 neurons. The architecture includes five convolutional layers and three fully connected layers. The convolutional filters utilize sizes of 11×11, 5×5, and 3×3. The team used ReLU activation and dropout regularization with a probability of 0.5 to improve performance. They also used local response normalization and data augmentation. The training process involved scaling, cropping, RGB shifting, and horizontal reflection. This method increased the training set size 2048-fold.
| Component | Specification |
|---|---|
| Total Parameters | 60 million |
| Total Neurons | 650,000 |
| Training Epochs | 90 |
| GPU Model | Nvidia GTX 580 |
| Dropout Probability | 0.5 |
| Training Duration | 5 to 6 days |
You know the name AlexNet, but the technical setup in a bedroom changed everything. The team used stochastic gradient descent with a momentum of 0.9 and weight decay of 0.0005. They manually decreased the learning rate three times during training. Weights were initialized as gaussians with 0 mean and 0.01 standard deviation, while biases in convolutional layers 2, 4, 5, and all fully-connected layers were set to constant 1 to avoid the dying ReLU problem. Notably, convolutional layers 3, 4, and 5 connected to one another without any pooling or normalization. Google acquired the team’s company, DNNresearch Inc., for $44 million in 2013.
Neural networks and the Nobel Prize
Geoffrey Hinton received the 2024 Nobel Prize in Physics for foundational discoveries that enable machine learning with artificial neural networks. This follows his 2018 Turing Award. Hinton, a professor emeritus at the University of Toronto, left Google in May 2023 to speak freely about the risks of AI technology. He expressed concern about malicious actors and the existential risk from artificial general intelligence. He estimates a 10% to 20% chance that AI might wipe out humanity. He proposed building maternal instincts into AI models so that these systems care for people. Fei-Fei Li, who co-founded World Labs, disagrees with this framing. She advocates for human-centered AI that preserves human dignity and agency. Hinton has also contributed to research on Boltzmann machines and distributed representations. His earlier research included the wake-sleep algorithm and the development of the t-SNE visualization method. He also served as the chief scientific advisor of the Vector Institute in Toronto. How can researchers technically build maternal instincts into superintelligent systems?
