Predicting progress for Large Langue Models

[argosopentech]

Maybe, the software and datasets are definitely getting better quickly too.

Algorithmic progress vs Compute progress for ML

We investigate algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision. We estimate a model, informed by work on neural scaling laws, and infer a decomposition of progress into the scaling of compute, data, and algorithms. Using Shapley values to attribute performance improvements, we find that algorithmic improvements have been roughly as important as the scaling of compute for progress computer vision. Our estimates indicate that al…