Apparently, getting LLMs to run locally on PCs is forcing new computer hardware designs that sound to me like Apple Silicon’s SoC architecture:
2 Likes
It’s an interesting article for sure. I’ll admit the Mac fan in me was a bit put off when they essentially spend 4 pages talking about how Intel and Nvidia and MS are all looking forward to doing what Apple has been doing for the past 5 years (which they barely acknowledge) but then follow that up with this broadside
However, Apple’s GPUs aren’t as capable as the best ones used in PCs, and its AI tools for developers are less widely adopted.
There is not a single PC notebook out that can hold a candle to what an M4 Max MBP overall can do while maintaining the weight of a MBP. There are “portable” bricks. And there are serious non-portable workstations (I just got a brand new massive 19" rack-mount Linux box here myself), but if it’s supposed to be truly portable and potent, Apple is still where it’s at. They can be knocked for several things, but Apple silicon and their MBPs aren’t among them. Those are still hands down best in class.
2 Likes
But people doing major AI work on PCs aren’t doing it on their laptops. They’re buying desktop systems in tower cases, using the latest and greatest PCIe GPUs from NVIDIA, and drawing kW of power in the process.
These absolutely do blow the doors off of Apple Silicon. And you will pay a hefty premium, both in dollars and power consumption for that capability.
2 Likes
Lots of innovation coming from all around.
I interpret the news in the article Nello linked to confirm Apple’s lead in hardware development to remain on the order of a couple of years, as it has been for thirty or more years.
Meanwhile over at Apple, it seems the limit on ongoing growth will be the manufacturing rate for M-series processors.
macOS Tahoe 26.2 will give M5 Macs a giant machine learning speed boost
The cluster described is limited by the rate of data transfer between nodes, using Thunderbolt 5 in the example. As noted in the article, this is a prototype, and Apple has promised more data in the near future.
One limit seemingly eliminated by this clustering is the size of available RAM. The example cluster uses 1.5 TB of RAM, available for processing by the CPUs, GPUs and Neural Net processors of all the Macs in the cluster.
As speculated in a comment, if this cluster is limited by the data transfer rate of TB5, what if Apple put M-series processors on PCIe cards, which were be linked through the PCIe bus of a Mac Pro?
Demonstrations of prototypes are available, such as
Apple JUST Dropped a Game-Changer
While the described clustering technology will eliminate, or at least modify, Apple’s inability to process models requiring large RAM arrays, it also changes the supplier from which the RAM must be sourced. The terabyte or two of RAM used by a four unit cluster will be included in the SoC units which form the cluster. Mac clusters will not be affected by the RAM shortages and prices plaguing the rest of the industry.
Clustered Macs retain Apple Silicon energy efficiency. The prototype cluster in the linked example runs at about 500 Watts. Apple Computer Data Centers (ACDC) running Mac clusters will impose much less load on the energy supply grid. Facility cooling loads will also be much less. Adoption of ACDC will advance Apple’s environmental goals. Easy to see the market niche Apple has targeted for ACDC.
Low initial yields manufacturing state of the art variants of M5 and M6 processors may limit initial ACDC availability. Low availability could still supply in-house use, providing Apple reduced server capital outlays as Apple adapts existing AI software to its hardware. Lower capital costs, and lower operating expense of ACDC could increase the business operating margin as Apple moves into large scale AI. (Reminiscent of the cell phone market, where Apple has a relatively small part of unit sales, but almost all profits.)
As M5 and M6 processor manufacturing yields improve and costs drop, in-house use of ACDC will have trained AI models which can be run in inferential mode on (at that future time New!) Mac Pros which implement Apple AI on their internal clusters of Apple Silicon processors. Such putative “New Mac Pros” will probably cost too much for individual users, but easily fit the budget of organizations such as hospitals, businesses, studios, and research organizations seeking to benefit from AI. Which is to say, much less expensive and more available than the earth- and business-destroying monstrosities created with Nvidia and similar technology. New Mac Pros will enable organizations to apply AI to their local specialized data sets, as well as provide the “edge computing” needed to interface with server AI such as from ACDC. This is the second market which recent announcements suggest Apple is targeting.
In the article Nello linked, Apple’s competitors describe the inadequacy of their existing technology to meet the needs envisioned. In the articles I linked, the adequacy of Apple technology technology is described, and prototypes illustrating its use, are shown. Windows reviewers’ ignorance of Apple technology is an old, old, story.
I think we will see before too long why Tim Cook smiles when interviewers ask what Apple will do to recover from being so far behind in AI. I expect hospitals and other health organizations will be among the first beneficiaries of Apple AI, ACDC, and so on. The world’s written manuscripts are probably not the most productive trove of data asking, needing, to be mined for AI insight. As Tim has been saying for years.
I have no special inside information. I just read the writing on the wall, then try to assemble the bits into a coherent story. Sometimes a framework, even if only speculative, can make the picture more apparent.
3 Likes