> Note, we did get a little more complicated and create a SystemD script to launch it at startup.
There's a good chance that most of what you want is in Systemd.
Restart a process if it fails health checks? Systemd. Do you want to isolate a process in a container? Systemd. Open privileged sockets? Systemd. You can do a lot with Systemd.
systemd is great. I used it successfully for operations on many projects that I ran. It provides the standardized way to reliably run and manage services on a single machine. I also agree with opponents that shell was the de facto standard way to orchestrate things, allowing you to go from dev to prod without significant changes. I was a fan of runit as a process manager before systemd took over the space.
That said there is a problem when you want to run things on multiple machines. What then? Unix and systemd were not designed as an OS for machine clusters, even though it won't be hard to imagine the concepts to be stretched even to this realm.
The docker/kubernetes crowd has been targeting the multi-machine OS scene much sooner and better, even though the solution of let's pack all dependencies, the entire machine and then some to a fat package and then have it run by a scheduler complex as f seems not like the ultimate future of the space.
I took a look at Nomad as a replacement for Kubernetes, which is much much simpler and you can get up and running with a single binary within 2 hours using this tutorial [0]. Nomad supports not only docker as a backend, but you can run raw_exec backend as well - that is any binary, such as shell script, that you wish to run without packaging the entire world with it. Now imagine if only you could schedule to run systemd units using Nomad. You'd scale from a single machine OS to cluster OS just like that.
I predict that Kubernetes will NOT be the ultimate answer as many are hoping for, betting their careers on it. It is too complex and not supporting other than docker runtimes makes it much inferior product than Nomad. Simple solutions usually win ultimately, so Nomad or systemd extended for multi machine services or even just vanilla linux process management extended to cluster level seem like much saner approaches.
> Now imagine if only you could schedule to run systemd units using Nomad
With some tweaks to your unit files this is actually possible. Nomad has the concept of custom task drivers you can implement to make it schedule any kind of workload you like. I am maintaining a task driver which allows you to run systemd-nspawn containers with Nomad [1].
Using this task driver you can deploy your systemd units running inside a systemd-nspawn container into a Nomad cluster. If this sounds interesting to you I have written a how-to blog-post about this [2]
When you say that Kubernetes only supports the Docker runtime I guess you meant only containers as it supports anything that implements CRI (Container Runtime Interface) like containerd.
Think I just saw from your article that Nomad supports straight up Java binaries but don't know how flexible that is
What would be the point of using Nomad to orchestrate systemd units instead of just using Nomad to raw_exec stuff? Nomad does health checks and all that, so systemd isn't really needed.
I've gotten incredibly far with packer-generated amis, a few build scripts and systemd to keep everything humming along. The only part I'm not a fan of is re-building amis due to the amount of time it takes to bake them.
I have a bunch of platforms that’re just jarfile monoliths (with external db/redis) that we deploy by provisioning the latest Amazon Linux onto EC2, then running a
Ansible to configure and add our local dependencies and the jarfile, then run the start script. While our Ansible step runs quickly enough, this is a nice way to not worry too much about managing our own AMIs.
(Two things really help here, we have Windows/macOS/Linux devs all developing and running the code locally so they’re pretty good at not accidentally becoming dependant on stuff outside the JVM, and the healthcheck endpoint the load balancers use is very comprehensive and over the years has been updated to include every non-detected failure or performance degradation.)
The only argument I have heard for Docker that makes any sense to me lately is that when your developers are using macOS and your servers run Linux, it’s nice to have dev/prod parity. Practically I think this speaks to either using better development environments, or writing code that is much more portable than what people try to get away with.
Get something running in any environment really quickly because it is already a separate environment
Nobody has to deal with package managers messing up their system, or have to deal with package manager managers that lets them switch between different versions of a package manager just so they can run some obscure dependency before getting to the project
I'm not sure why systemD is needed for restarting a process what about a simple loop with an if statement? Process isolation to run a go binary on a presumably dedicated web server? Privileged sockets, not even sure what those might be.
The point of this post is to not complicate things, but what I hear is even more complicated be it via a docker alternative.
> I'm not sure why systemD is needed for restarting a process what about a simple loop with an if statement?
It's not needed, but that's why a lot of older apps have their very custom solution to deal with that issue. (For example MySQL) Each with their custom exit condition handling, cleanup, backoff, etc. A simple loop also won't cleanup processes forked from the main server.
if you go down the Restart=always route, please take a long look at StartLimitIntervalSec, RestartSec, StartLimitBurst. in some failure modes always isn't quite always always.
That's the whole point, you're not mysql. You don't need the sophistication of that magnitude. You need to run a standalone go binary to listen on a port. Simple. Why bother with the blackbox of systemD when you can perfectly get away with a basic shell script?
The first link you sent is 1024 lines long, do you genuinely believe using something like this is preferred to a simple shell script + nohup? I'm amazed at the hoops some devs jump through just to get to work with their favorite tool.
That script is never going to be simple in a few years. You're going to keep adding features to it with time until it badly implementsa tiny fraction of what you already had available in your init system. The question could be: why spend time rewriting a broken restart script if you already have a good version available behind a config switch.
If you don't like how long that man page is, you should check the size of bash+nohub man page which you'd otherwise need :-)
> I'm not sure why systemD is needed for restarting a process
You need a systemd unit file that starts your daemon properly when the server boots anyways (most common distros use systemd for init), and it's only a few more lines to make sure it gets restarted if it dies.
Systemd config files are fairly simple and straightforward. It gives a simple interface to start/stop/restart processes and lets you do things like automatically start a process on server boot.
> Note, we did get a little more complicated and create a SystemD script to launch it at startup.
There's a good chance that most of what you want is in Systemd.
Restart a process if it fails health checks? Systemd. Do you want to isolate a process in a container? Systemd. Open privileged sockets? Systemd. You can do a lot with Systemd.