I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubernetes isn't that much more complicated either.
A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.
Service Fabric needs to die. It is an abomination on software.
Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in.
No thanks.
Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in.
> Windows doesn’t have “distributions”, so there’s less need for containers in that space.
I don't follow. What does distributions have to do with need for containers?
> What does distributions have to do with need for containers?
A lot (but not everything).
It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc...
In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters.
Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself.
Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable.
[1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is.
[2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc
Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powerful. It also enables you to do it outside of Azure/Service Fabric on other clouds or metal.
You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues
Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios.
DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work.
There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also.
Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines.
Not so much, no. Linux is the kernel, and all Linux distros, every single one, employs Linux as its kernel, or else it would not be Linux. Stepping back from my pedantry, I believe what you must mean is that different Linux distributions will customize the kernel a bit. That doesn't change what it is. Just because you have automatic windows and I have cranks, and you have leather interior and I have plush doesn't mean our vehicles aren't the exact same year, make and model.
Sure, I can imagine, but that is merely a bug, or perhaps the compatibility issue stems from misuse of a general but inaccurate claim of "Linux compatible." My expectation is specific compatibility is stated plainly in the documentation and on the box, if there is one, without ambiguity.
Note that because of Linus's strict policies on userspace compatibility, A>B is basically guaranteed to always be safe, and only A<B will ever cause problems, unless distros add patches that break things.
I e heard this before, but the only case I’ve heard of people running into issues is if you hit a kernel bug or depend on a really new feature that’s not present in the kernel yet.
Mixed workloads. Not everything inside MS is Windows, in fact, more than people probably realize is running on Linux. We have our own internal Linux distribution (Mariner) and own Kinvolk (which maintained CoreOS/Container Linux/Flatcar Linux).
That famous tech called cgroup was actually a Google contribution. But I agree that k8s is essentially Google's step to make themselves relavant in Cloud. They have missed the initial opportunity by promoting their PaaS AppEngine instead of something IaaS like ec2 in the beginning of the cloud competition, so Google just play the open-source game and keep releasing stuffs that can be used in all three clouds to lure people to use GCP. But then k8s is a very nice piece of tech that allows one to manage large clusters without vendor lock-in.
Cgroups, namespaces, apparmor/selinux, overlay filesystems, there is much more to containers than just cgroups.
The no vendor lock is looks great on paper but you are locked in day one. (Eg on aws you probably use IAM, LB, ASG for K8 Nodes - you can maybe move it to another cloud but the effort is going to be significant). Cloud agnosticism is a lie.
Effort will be significant for any global changes for non-trivial software. Significant effort is fine. Can you compare moving something from AWS to GCP for Kubernetes and for something like Lambda+Fargate?
I'd say not only is it a lie but is actively a not very good strategy to pursue right now, at least not all out as though you were pursuing some kind of multi-cloud end game.
At the K8s level, it seems like the introduction of the Gateway API is probably a good level of abstraction to work towards that will keep things about as flexible as possible without all of the insanity that comes with going beyond that to keep everything 100% vendor neutral.
It's very strange some of the phrasing of this post. I don't know AWS and Azure that well so I might be off here but every time I look over into those ecosystems it's hard to not come to the conclusion that GCP is literally several years ahead in terms of what they are offering.
Even just sticking strictly to the K8s space, there is nothing even close to matching GKE standalone, let alone GKE autopilot that I am aware of. In the world of serverless the gap looks to be even bigger.
I couldn't imagine a situation where I would even consider something else in a greenfield project / company honestly.
