Sam Altman asserts that the research approach responsible for creating ChatGPT has reached its limits, and future advancements in artificial intelligence will necessitate innovative concepts.
The extraordinary abilities of ChatGPT, OpenAI’s chatbot, have ignited a renewed interest and investment in artificial intelligence. However, OpenAI’s CEO recently indicated that the research approach responsible for creating the chatbot may have reached its limits, leaving the source of future advancements uncertain.
Over the years, OpenAI has achieved remarkable progress in AI language processing by scaling existing machine learning algorithms to unprecedented levels. GPT-4, the most recent project, likely involved trillions of text words and thousands of powerful computer chips in its training, costing over $100 million.
However, CEO Sam Altman believes that further advancements will not result from simply enlarging models. Instead, he envisions improvements in other aspects of the models. This revelation implies a surprising turn in the race to develop and implement new AI algorithms.
Since ChatGPT’s launch, Microsoft has integrated the underlying technology into its Bing search engine, and Google has released a competing chatbot named Bard. Numerous individuals have eagerly experimented with these chatbots for work or personal tasks. Additionally, well-funded startups like Anthropic, AI21, Cohere, and Character.AI are investing heavily in developing larger algorithms to compete with OpenAI’s technology.
Altman’s statement implies that GPT-4 may be the last significant breakthrough stemming from OpenAI’s strategy of enlarging models and providing them with more data. The future of research strategies and techniques remains unclear. In the GPT-4 paper, OpenAI acknowledges diminishing returns on model size scaling. Altman also highlights the physical limitations on data center construction and the speed at which they can be built.
Cohere cofounder Nick Frosst, who previously worked on AI at Google, agrees with Altman’s belief that simply increasing size will not suffice indefinitely. Frosst asserts that progress in transformers, the machine learning model central to GPT-4 and its competitors, depends on factors beyond scaling. He points to new AI model designs, architectures, and further tuning based on human feedback as promising research directions.
OpenAI’s influential series of language algorithms consists of artificial neural networks that predict the words following a given text string. Starting with GPT-2 in 2019, which had 1.5 billion parameters, the company continued to scale up with GPT-3 in 2020, boasting 175 billion parameters. These models inspired other companies and research institutions to pursue similar and larger AI models.
After ChatGPT’s release, many anticipated that GPT-4 would be an immense and complex model. However, when OpenAI announced the new AI model, they did not reveal its size, suggesting that size is no longer the primary focus. Altman confirmed that GPT-4’s training cost exceeded $100 million.
Though the specifics of GPT-4 remain undisclosed, it is likely that some of its intelligence derives from factors beyond scale, such as reinforcement learning with human feedback. This method, used to improve ChatGPT, guides the model toward providing higher quality responses based on human evaluation.
GPT-4’s astounding capabilities have amazed experts and prompted discussions about AI’s potential impact on the economy, disinformation, and job displacement. Recently, some AI experts, tech entrepreneurs, and scientists penned an open letter advocating for a six-month moratorium on developing anything more powerful than GPT-4. Altman confirmed at MIT that OpenAI is not currently working on GPT-5 and will not be for some time.
Follow my posts on LinkedIN
