That's what HBM is actually. The memory dies are directly next to the GPU die, on the same substrate. The main difference between Apple SoC and GPUs is the former use regular LPDDR while GPUs use HBM.
One of the key points of HBM is that dies are stacked up with many, MANY, more signals and channels. That's how NVIDIA has a memory bandwidth an order of magnitude higher than M4: 550GB/s for the M4 Max, 4.6TB/s for H200. And yes, that's bytes per second, not bits per second.
some GPUs use HBM. Most use GDDR. AMD and Nvidia still extract huge bandwidth from GDDR via high bus speeds + wide buses (like the 1.79 TB/s on the 5090)
Indeed! I implied "AI GPUs" since that’s where HBM is commonly used (despite AMD pioneering it on some consumer cards). And yeah, thousand-bit wide busses get close in performance.
One of the key points of HBM is that dies are stacked up with many, MANY, more signals and channels. That's how NVIDIA has a memory bandwidth an order of magnitude higher than M4: 550GB/s for the M4 Max, 4.6TB/s for H200. And yes, that's bytes per second, not bits per second.