Meta, which develops one of many largest foundational open-source massive language fashions, Llama, believes it would want considerably extra computing energy to coach fashions sooner or later.
Mark Zuckerberg mentioned on Meta’s second-quarter earnings name on Tuesday that to coach Llama 4 the corporate will want 10x extra compute than what was wanted to coach Llama 3. However he nonetheless needs Meta to construct capability to coach fashions quite than fall behind its opponents.
“The quantity of computing wanted to coach Llama 4 will seemingly be nearly 10 occasions greater than what we used to coach Llama 3, and future fashions will proceed to develop past that,” Zuckerberg mentioned.
“It’s laborious to foretell how this may development a number of generations out into the longer term. However at this level, I’d quite danger constructing capability earlier than it’s wanted quite than too late, given the lengthy lead occasions for spinning up new inference initiatives.”
Meta launched Llama 3 with 80 billion parameters in April. The corporate final week launched an upgraded model of the mannequin, known as Llama 3.1 405B, which had 405 billion parameters, making it Meta’s largest open-source mannequin.
Meta’s CFO, Susan Li, additionally mentioned the corporate is considering completely different information heart initiatives and constructing capability to coach future AI fashions. She mentioned Meta expects this funding to extend capital expenditures in 2025.
Coaching massive language fashions could be a pricey enterprise. Meta’s capital expenditures rose almost 33% to $8.5 billion in Q2 2024, from $6.4 billion a 12 months earlier, pushed by investments in servers, information facilities and community infrastructure.
In keeping with a report from The Data, OpenAI spends $3 billion on coaching fashions and an extra $4 billion on renting servers at a reduction price from Microsoft.
“As we scale generative AI coaching capability to advance our basis fashions, we’ll proceed to construct our infrastructure in a means that gives us with flexibility in how we use it over time. This can enable us to direct coaching capability to gen AI inference or to our core rating and suggestion work, once we count on that doing so could be extra precious,” Li mentioned in the course of the name.
In the course of the name, Meta additionally talked about its consumer-facing Meta AI’s utilization and mentioned India is the biggest market of its chatbot. However Li famous that the corporate doesn’t count on Gen AI merchandise to contribute to income in a major means.
