The Limits of Scaling: A New AI Shell Game
The recent fervor surrounding AI, particularly large language models (LLMs), has been largely fueled by the promise of scaling laws. These laws suggested that by simply increasing the amount of data, parameters, and compute power, we could achieve significant improvements in AI performance. However, a growing number of experts, including AI pioneer Gary Marcus, are questioning the validity of these scaling laws. In a recent Substack post, Marcus argues that the focus on scaling has led to a neglect of fundamental innovation. The Shifting Paradigm of AI Scaling Microsoft CEO Satya Nadella recently introduced a new dimension to scaling laws: "inference time compute." This suggests that by increasing the amount of time an AI model spends on a specific task, its performance can be improved. While this approach may yield some benefits, it raises concerns about efficiency and cost-effectiveness. Marcus points out that the original promise of scaling laws was to predict AI performance …
