The scale of the labeling effort
Fei-Fei Li began her work on ImageNet in 2006. She met Christiane Fellbaum in 2007 to discuss building the database from WordNet nouns. She drew inspiration from a 1987 estimate that the average person recognizes 30,000 different kinds of objects. The project utilized 49,000 workers from 167 countries through Amazon Mechanical Turk to filter and label over 160 million candidate images between July 2008 and April 2010, ensuring that each of the 14,197,122 images received three separate annotations. The team had enough budget to have every one of the 14,197,122 images labeled three times. I consider the scale of the labeling effort more important than the specific architectures that followed. The categories fall into 9 levels, from level 1 like mammal to level 9 like German shepherd. The team scraped images from online searches using synonyms in multiple languages to find the 21,841 synsets.
The ILSVRC era
The ImageNet Large Scale Visual Recognition Challenge began in 2010. The 2010 winning team used a linear support vector machine with 52.9% classification accuracy and 71.8% top-5 accuracy, training on three 8-core machines for four days. In 2011, the XRCE team reached 74.2% top-5 accuracy using quantized Fisher vectors. In 2012, AlexNet achieved a top-5 error of 15.3%, which was 10.8 percentage points lower than the runner-up. This model used graphics processing units during training. You already know that deep learning changed everything. In 2013, the winners were OverFeat for localization and an ensemble of CNNs by Clarifai for classification. In 2014, GoogLeNet won classification while VGGNet won localization. In 2015, ResNet exceeded human performance. In 2016, CUImage won with an ensemble of six networks. In 2017, the SENet architecture reduced the top-5 error to 2.251%. The 2017 organizers stated the benchmark was solved and planned to move to 3D images, though this never materialized.
| ImageNet Metric | Value |
|---|---|
| Total images | 14,197,122 |
| Total synsets | 21,841 |
| Bounding box images | 1,034,908 |
| 2012 AlexNet error | 15.3% |
Andrej Karpathy estimated in 2014 that he could reach a 5.1% error rate with concentrated effort, while humans could reach 2.4% with maximal effort.
Dataset errors and the move to 3D
The dataset contains errors. Over 6% of labels in the validation set are wrong, and 10% of ImageNet-1k contains ambiguous or erroneous labels. I see the problems in the person subtree, where 1,593 out of 2,832 synsets were labeled potentially offensive. After filtering for non-visual synsets, only 158 remained. In the ImageNet-1k version, 17% of the 1,431,093 images contain faces that the team blurred in 2021. This blurring involved 562,626 faces. What happens when these 2D benchmarks reach their limit? Li began a two-year sabbatical in 2024. She now works on World Labs to develop Large World Models that understand 3D environments. The company raised close to $100 million in two funding rounds from investors like Andreessen Horowitz and Radical Ventures. Li won the 2023 Intel Lifetime Achievements Innovation Award and was named in the Time 100 AI Most Influential People list.
