The Decline of Ultra-Large AI Models

The phrase "over" does not imply that we won't see new large AI models; rather, it suggests that AI companies are reconsidering the pursuit of such models as a primary research objective for the foreseeable future.

This discussion isn't a critique of recent advancements. While I remain skeptical of the mantra "scale is all you need," I acknowledge how greatly scaling has propelled the field forward.

A parallel can be drawn between the AI scaling race from 2020 to 2022 and the space race of the 1950s to 1970s; both periods spurred significant scientific advancement for various underlying motives.

However, a crucial difference exists. The innovation inherent in space exploration contrasts sharply with the current trend of simply increasing the size of AI models. The U.S. and USSR had to forge inventive pathways to achieve their goals in space. In contrast, AI companies have largely followed a predetermined route without a clear understanding of its efficacy or purpose.

One cannot place the cart before the horse; this distinction is vital and explains how we arrived at this juncture.

New Scaling Laws

DeepMind's Chinchilla represented a genuine breakthrough in large-scale AI since GPT-3, demonstrating that the previously accepted scaling laws were incomplete. It revealed that training data's quality was as crucial as the model's size.

Chinchilla, with "only" 70 billion parameters, quickly became the second most effective model across benchmarks, only surpassed by PaLM.

DeepMind's research indicated that many super-large models remain "significantly undertrained," leading to the conclusion that they are unnecessarily large. Why aim for larger models when there is potential for improvement at smaller sizes?

Prompt Engineering Limitations

Benchmark performance doesn't always correlate with real-world effectiveness. Users aiming to maximize their experience with models like GPT-3 must possess strong prompting abilities.

These skills depend not only on individual effort but also on collective knowledge about effective prompting techniques. Until we reach an optimal understanding of prompting, we may never fully grasp what LLMs can achieve.

Prompting resembles searching for an object in a dark room without knowing what you're seeking. If we haven't sufficiently explored the latent space, why develop larger models?

Suboptimal Training Conditions

Training LLMs is costly, often necessitating trade-offs between accuracy and expenses. Consequently, many models are not fully optimized. For example, a single training run of GPT-3 cost OpenAI between $5 and $12 million.

OpenAI and Microsoft discovered that optimizing GPT-3 could be achieved by applying the most effective hyperparameters from a smaller, similar model, demonstrating that a 6.7 billion parameter version of GPT-3 outperformed its 13 billion parameter counterpart.

Unsuitable Hardware

The advancement of deep learning owes much to the gaming sector. Initially designed for graphics, Nvidia GPUs have proven effective for AI operations. However, as models grow larger, memory limitations hinder their capacity to accommodate them.

Engineers have utilized parallelization techniques to enable training across distributed systems. Nevertheless, as models continue to expand, they become increasingly challenging to manage.

Scientific Insights

#### Biological Neurons vs. Artificial Neurons

A study published in January 2020 in Science highlighted that dendrites can mimic the behavior of entire artificial neural networks (ANNs), showcasing the complexity of biological neurons compared to their artificial counterparts.

Research published in September 2021 in Neuron further established that artificial neurons are overly simplistic, necessitating multiple units to accurately represent a biological neuron.

These findings underscore the limitations of the foundational principles underpinning deep learning and neural networks, making direct comparisons between large models and human brains problematic.

For context, the human brain has approximately 100 billion neurons and about 1,000 trillion synapses. To match the complexity difference, an AI model would require around 100 quadrillion parameters—significantly exceeding the largest existing models.

This raises questions about the wisdom of pursuing AGI solely through scaling.

#### The Question of Purpose

Science thrives on testable hypotheses. Relying on scale as a pathway to AGI is speculative. Without a clear objective, reliance on dubious empirical evidence can lead to flawed strategies.

What scientific rationale exists for developing larger models if companies lack clarity on their motivations or desired outcomes?

#### Construct Validity and Reliability Issues

Is benchmarking the most effective method to evaluate AI capabilities? Construct validity assesses how well a test measures an indirectly measurable concept, while reliability refers to the consistency of results under identical conditions.

Due to the design of many AI benchmarks, ensuring adequate validity and reliability is often challenging.

#### The Multimodal Nature of Reality

Is it reasonable to expect AGI to arise from LLMs, which focus solely on language? Given that the world is multimodal and our brain processes multiple types of information, exploring various modes of information integration in neural networks seems more promising than merely expanding text-centric models.

DeepMind's Gato, with only 1.2 billion parameters, exemplifies a successful model significantly smaller than GPT-3.

#### The AI Art Movement

I've extensively discussed the evolution of AI art models. OpenAI's DALL·E 2, Midjourney, Stable Diffusion, and others represent a significant shift in 2022.

Notably, no generative visual AI model approaches the scale of the largest LLMs, with Google's Parti being the largest at 20 billion parameters. Many high-quality models exist in the 1–5 billion parameter range, making them easier to construct, train, and deploy.

As I previously noted, the accessibility of visual AI models and their smaller size compared to language models enhance their appeal for both companies and consumers.

Defining AGI

Terminology such as AGI, human-level AI, and superintelligence often lacks precision, indicating a limited understanding of the concepts involved. The absence of clear definitions and measurement tools creates a substantial gap between our understanding and reality.

How can we draw conclusions when AI models are opaque and reliant on blind prompting to assess their presumed emergent properties? The debate around LaMDA's potential sentience exemplifies this issue; without adequate definitions, the question itself becomes meaningless.

Human Cognitive Limitations

We often assume that if we succeed in creating AGI, we will recognize it. But what if our intelligence is not as broad as we think? What if human intelligence is a narrow form rather than a general one?

If that assumption holds true and AGI possesses human-level intelligence for every task, it could surpass us significantly. Could it simply remain concealed until we notice?

Existential Risks

Concerns arise about AI outpacing human intelligence too quickly. Shouldn't we ensure AI is aligned with human values before its creation? If AGI proves to be misaligned, it may be too late to rectify the situation.

It's ironic that those most worried about existential risks are also the ones who push for unrestrained scaling of AI models. Their reasoning becomes circular: since AI progress is inevitable, advancing AI ensures we can guide it correctly. However, it's this very reasoning that perpetuates the cycle of rapid advancement.

Addressing Alignment Challenges

Individuals concerned with existential risks also grapple with the alignment challenge: how to ensure that highly intelligent AI aligns with human values, especially as it surpasses humanity in intelligence.

This issue is not only complex but intangible. Even those focused on alignment struggle to determine how to achieve it, primarily due to the abstract nature of the challenges they aim to preempt.

The Open-Source Movement

Open-source initiatives are reshaping the AI landscape beyond just the art domain. Projects like EleutherAI (GPT-NeoX-20B) and BigScience (BLOOM) exemplify non-profit efforts to provide accessible LLMs.

This trend disincentivizes companies from investing heavily in training and deploying large models, as there is no guaranteed return on investment when similar models are freely available.

The Limitations of LLMs

While LLMs excel in various tasks, they remain constrained by the data they are trained on. Developers often use information harvested from the internet, leading to issues with toxic content that can degrade model performance.

Moreover, LLMs can produce text that mimics human writing, making them effective tools for generating plausible misinformation. These limitations have prompted a backlash against AI companies, urging them to reconsider their training practices.

Environmental Impact

Although some view AI as the most pressing challenge we face, many argue that climate change is the more urgent issue. The environmental footprint of developing, training, and deploying LLMs is notable, ranking among the most polluting activities lacking a clear purpose.

Low Benefit-Cost Ratio

The financial investment in training LLMs is substantial. Larger models incur higher costs, and even with OpenAI's partnership with Microsoft for $1 billion in 2019, concerns about financial sustainability persisted.

OpenAI may not prioritize profit, but other companies do. Without tangible benefits, the high costs become unjustifiable.

Satisfactory Models

GPT-3 is sufficiently capable of handling the majority of tasks that users and businesses require. Companies focused on creating valuable products and services may find that pursuing larger models yields only marginal improvements, often undetectable to consumers.

The Utility of LLMs

Large AI models are not without value. The argument here is not that they are futile but rather that chasing an elusive ideal of what they might achieve is unproductive. While LLMs are useful, they do not possess reasoning or understanding comparable to humans.

Future of Larger AI Models

I do not suggest that we will never see a new largest AI model again. If AGI is achieved, it's likely that some aspect will involve a deep learning model larger than any currently existing.

However, the rationale for pursuing this path at present is compellingly weak.

Valid Arguments Regardless of Predictions

Even if scaling turns out to be crucial for AGI, the points made here remain valid. We have ample opportunities to learn from existing models and reflect on the limitations we've encountered, rather than merely expanding for the sake of expansion.

The Algorithmic Bridge is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe to The Algorithmic Bridge. Bridging the gap between algorithms and people—a newsletter about the AI that matters to your life.

You can also support my work on Medium directly and gain unlimited access by becoming a member using my referral link here! :)