Distributed computing for data science not only benefits when we go big, but also when we need go small with independent nodes.
e.g. Cluster of scalable ARM SBC nodes or even multi-architectural nodes could be highly efficient.
I've been exploring this kind of setup for a while now, frameworks like Dask, Ray[1], Modin can do this with Python to some extent. But they are still finicky and Dask seemed more stable than other frameworks for this setup.
I wanted to try out language level distributed computing setup, but Julia required same setup (OS/Arch/Path) replicated on all their nodes last time I visited it.
I feel, distributed computing as such hasn't got much love as it deserves in the consumer market, especially since many have several computing units in their houses now(PC, tablet, smartphone, Watch, TV, Game console). If interoperable distributed computing layer was fundamentally baked in with all modern operating systems minimising the latency with Network, Storage, Memory; Then we could leverage huge compute power on demand, which not possible without investing lot money in single compute unit nowadays.
Then again, Compute power is a major strategic advantage for the manufacturers. Why would Apple share its A13X for compute with a Snapdragon or Intel?
e.g. Cluster of scalable ARM SBC nodes or even multi-architectural nodes could be highly efficient.
I've been exploring this kind of setup for a while now, frameworks like Dask, Ray[1], Modin can do this with Python to some extent. But they are still finicky and Dask seemed more stable than other frameworks for this setup.
I wanted to try out language level distributed computing setup, but Julia required same setup (OS/Arch/Path) replicated on all their nodes last time I visited it.
I feel, distributed computing as such hasn't got much love as it deserves in the consumer market, especially since many have several computing units in their houses now(PC, tablet, smartphone, Watch, TV, Game console). If interoperable distributed computing layer was fundamentally baked in with all modern operating systems minimising the latency with Network, Storage, Memory; Then we could leverage huge compute power on demand, which not possible without investing lot money in single compute unit nowadays.
Then again, Compute power is a major strategic advantage for the manufacturers. Why would Apple share its A13X for compute with a Snapdragon or Intel?
[1]https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b...