This is interesting. I would be curious to try a setup where I keep a local hybrid cache and transition blocks to deep storage for long-term archival via S3 rules.

Some napkin math suggests this could be a few dollars a month to keep a few TB of precious data nearline.

Restore costs are pricy but hopefully this is something that's only hit in case of true disaster. Are there any techniques for reducing egress on restore?

The easy way to achieve that, is using ZeroFS NBD server with ZFS L2ARC (L2ARC with local storage and “main” pool on ZeroFS).

Interesting! What would be a typical use-case of ZeroFS? could I use this to store my Immich and Jellyfin data on S3 so I don't need disk?

If you don't mind paying about a dollar to stream one of your own movies as well as a couple of bucks per year to store it.

You don’t have to use AWS S3, any compatible implementation will work.

That should work!

The Jellyfin metadata would certainly be a fit but what about streaming video content i.e. sequential reads of large files with random access?

If you have the network that matches, it should be perfectly fine.