Really think the speculation about an “end of scaling” seems very premature given (1) AI lab insiders are mostly universally bullish on scaling and (2) scaling inference works & has barely been exploited… but also a more linear improvement path for AI would still be disruptive
— Ethan Mollick (@emollick) November 13, 2024
The leading artificial intelligence companies OpenAI, Google, and Anthropic are facing significant challenges in their efforts to develop more advanced AI models. Despite investing heavily in computing resources and training data, these companies are encountering diminishing returns in terms of performance improvements. OpenAI’s latest model, known as Orion, has reportedly fallen short of the company’s expectations, particularly in handling coding tasks.
The model lacks significant improvements over existing systems when compared to the gains made by its predecessor, GPT-4.
I don't wanna say "I told you so", but I told you so.
Quote: "Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training – the phase of training an AI model that uses a vast amount of unlabeled…
— Yann LeCun (@ylecun) November 13, 2024
Similarly, Google is facing obstacles with its upcoming Gemini software, while Anthropic has delayed the release of its anticipated Claude 3.5 Opus model. Industry experts attribute these challenges to the increasing difficulty in finding new, untapped sources of high-quality, human-made training data and the enormous costs associated with developing and operating new models concurrently with existing ones.
The age of scaling up AI models infinitely by throwing money, computing power and data at them is over.https://t.co/ANwmVrUF5G
Bigger LLMs don't necessarily offer proportional benefits or outcomes.. Scaling and tweaking the "right" moving parts is more important.. This was our…
— CP Gurnani (@C_P_Gurnani) November 12, 2024
Diminishing returns in AI advancement
The belief that more computing power, data, and larger models will inevitably lead to better performance and ultimately artificial general intelligence (AGI) could be based on false assumptions. As a result, companies are now exploring alternative approaches, such as further post-training, which incorporates human feedback to improve responses and refine tone, and developing AI tools called agents that can perform targeted tasks, like booking flights or sending emails on a user’s behalf.
Is AI reaching the ceiling? I hope not! I still think there’s a lot that AI can do and I’m excited to see what companies like OpenAI come up with! https://t.co/td6C8kCSE0
— John Legere (@JohnLegere) November 13, 2024
Margaret Mitchell, chief ethics scientist at AI startup Hugging Face, suggests that “different training approaches” may be needed to make AI models work well on a variety of tasks. Other experts echo Mitchell’s sentiment, indicating that the industry’s faith in “scaling laws”—the belief that increased computing power and more data will yield exponentially better AI—may not hold true. The challenges faced by major AI companies pursuing breakthrough general-purpose AI models could ultimately validate more conservative strategies, such as Apple’s approach of developing specific AI features that enhance the user experience while prioritizing privacy.
As costs soar and expectations intensify, experts warn against expecting sustained rapid progress in AI development. The initial excitement surrounding these technologies might give way to a more tempered pace of advancement as companies navigate the complex landscape of AI research and development.
Rashan is a seasoned technology journalist and visionary leader serving as the Editor-in-Chief of DevX.com, a leading online publication focused on software development, programming languages, and emerging technologies. With his deep expertise in the tech industry and her passion for empowering developers, Rashan has transformed DevX.com into a vibrant hub of knowledge and innovation. Reach out to Rashan at [email protected]
