Network cards aren't simply just cards anymore. Often you get multiple interfaces depending on each other, e.g. bridges depending on multiple ethernet interfaces which wait for STP delay to be over until they are really forwarding packets. VLAN interfaces that are dependent on their parent ethernet interface. VPN and other tunnels. So you get stuff like "start wpa_suplicant for 802.1x, take the ethernet interfaces up. start the bridge interface. check if DNS works for the VPN. then start the VPN daemon and the VPN interface. then start network services listening on the VPN interface". That would be a typical setup for a server in a remote office somewhere.

Another fun situation is a server with LVM, RAID and encrypted disks (also typical for a remote office somewhere). SAS disks will come online (after spinup time), generate an event. LVM and raid detection will run, generating more events, usually dependent on one another. Device naming is in some cases dependent on the event order, so you also have to rely on systemd-udev not fucking up by-id or by-name links. Then you have cryptsetup, which decrypts the disk, generating another event, but only upon an automatic or manual all-clear and password from the (hopefully working) network. Then you arrive at the actual filesystems in fstab, after a chain of udev events, each of which (and especially the LVM and cryptsetup ones) sometimes fails. Same "fun" for SAN devices.

Oh, and another fun problem is systemd exhausting randomness for too many UUIDs, so cryptsetup hangs for swap devices because those get a key from /dev/random.

Of course those bugs get often fixed over time, but only until the next RedHat release arrives with yet another systemd-somethingd, networkmanager, systemd-udev-bug that introduces another new problem in the above. Usability has gone up since RH7, but reliability is definitely down.

This is a long list of complexities, but I'm not sure how they relate to udev/systemd. Yes, setting up aggregate interfaces, VPNs and fancy drive configurations is complex and can fail due to various delays. That's not a problem with udev reliability if an underlying system fails. We had the same issues before systemd/udev was a thing.

> cryptsetup hangs for swap devices because those get a key from /dev/random

You can choose urandom (third column of crypttab). Systemd doesn't enforce anything here.

You wanted to know how many udev events and therefore dynamic behaviour a server could have at boot. I told you my experience.

And of course the underlying system may fail, but actually it is more common to have udev, lvm and systemd fail. Just google cryptsetup, udev, boot, lvm, raid and problem in arbitrary combinations. Most of what you will find are bugs in udev, lvm or systemd.

urandom is a workaround for stupid behaviour from systemd. Of course /dev/random isn't necessary, but that is beside the point, systemd again broke something by being boneheaded.

Being able to hotswap more components in the data center without power cycling the server is becoming more useful again (I say again because this was a long standing benefit of mainframes). More than just hard disks, it's useful for SSDs, and now GPUs and other types of accelerators. PCIe supports hot plug, but it's rarely actually used.

Not just mainframes, there was and is even RAM and CPU hotplug x86 hardware (but rare and expensive). Sun hardware (enterpricey, but not mainframe per se) had CPU boards you could pull and replace while running. Network cards and HBAs are other more typical examples of things you could hotswap. But the trend has favored software based solutions providing redundancy on multiple cheaper servers. Imho rightly so.