r/AlmaLinux • u/RetroGrid_io • 14d ago
ZFS on root?
I love AlmaLinux! I love ZFS! The idea of these working together sounds heavenly:
- Do an update, have it "go south" and just roll-back the root filesystem and reboot.
- Backup everything in a bootable state, in real time with snapshots send|receive.
- Clone the / filesystem into a disk image that you can boot immediately, on the same machine, in a VM to test software installs and such, without wasting disk space, and without taking the host machine offline, and without risking the host machine's stability.
- I'm sure there's more
However, ZFS has always seemed a second-class-citizen on anything RHEL based, which seems absurd to me because ZFS is (IMHO) SO MUCH PREFERABLE to XFS on LVM.
However, I've never run ZFS on root, simply because it complicates recovery if there's ever a problem. I can't just boot off an Alma ISO and clean it up. Instead, what I do is run the O/S on simple partition RAID1 and then use ZFS for everything else. I made my mind to try ZFS on root when it demonstrably improved stability which is already a really high standard.
Are you running ZFS-on-root under Alma? Can you describe your set up, and what you specifically benefit from it?
5
u/jonspw AlmaLinux Team 14d ago
A lot of distros would pick it up if Oracle would loosen their grip on the licensing. That's the problem.
OpenZFS works great on top of AlmaLinux.
4
u/RetroGrid_io 14d ago edited 14d ago
This is a legal question, and here's Ubuntu's public statement about this. Ubuntu supports ZFS "out of the box".
I agree that it would be fantastic if Oracle clarified / revised its license.
1
u/tcpipguy 12d ago
Oracle is in bad shape financially - think teetering on insolvency. They way over leveraged their position on AI. If Oracle dies, I expect some interesting things to happen.
3
u/etbe 11d ago
While interesting things would certainly happen that doesn't mean ZFS license gets resolved. More likely chapter 11 for Oracle would mean no resources to develop ZFS or to respond to legal issues or questions - no response from official people means individuals can use it for free but companies (Red Hat/IBM, Canonical, etc) are limited.
If Oracle actually becomes insolvent and is broken up then the result would probably be uncertainty about who owns the ZFS rights (because the Oracle database and other big assets are the valuable things which will be clearly spelled out in contracts and other things get forgotten) which again is bad for corporations wanting to use it unless they want to pay serious money - not just license fees but paying lawyers to sort out the mess and deal with anyone who objects.
1
u/tcpipguy 11d ago
I mean you're right ofc. I just want to see ZFS fully opened up.
2
u/etbe 11d ago
Me too. But at the moment BTRFS has been doing better for what I need. The inability to shrink ZFS is a major problem for home use and all the serious work storage has gone to the cloud.
2
u/tcpipguy 11d ago
Btrfs is something I keep meaning to check out. I've heard good things about it.
1
u/etbe 11d ago
If you have 1 or 2 devices it basically does what ZFS does with less effort and confusion.
If you have a home setup where you keep buying new devices and replacing old ones then it's much better than ZFS.
If you want RAID-5 then it works in my tests with DATA BUT NOT METADATA. Do not try BTRFS RAID-5 metadata for anything other than test setups where you are happy to lose data at this time. You probably don't want to use BTRFS RAID-5 for data either.
For BTRFS RAID-6 (RAID-Z2) I haven't even tested that after the excitement of testing RAID-5.
My home server has an Intel 6.2TB NVMe device, a couple of 3.8TB DC grade SATA SSDs, and a coupel of 1.6TB DC grade SATA SSDs. BTRFS runs RAID-1 data and RAID-1C3 metadata on all of that. No problems of mixing sizes so I just buy any DC grade storage I see going cheap.
2
u/RetroGrid_io 10d ago
If you want RAID-5 then it works in my tests with DATA BUT NOT METADATA. Do not try BTRFS RAID-5 metadata for anything other than test setups where you are happy to lose data at this time. You probably don't want to use BTRFS RAID-5 for data either.
Statements like this utterly decimate my interest in BTRFS because keeping days at scale (100s of TB to PB scale) basically require RAID-like setups, eg RAIDZ1/2/3.
My original beef with BTRFS was in such a scenario, and would be my primary user today. What specifically did you find in your testing?
1
u/etbe 10d ago
For many years BTRFS RAID-1 has worked without problems for me. If all you need is RAID-1 and RAID-1C3 then BTRFS will make you happy.
For PB scale on BTRFS you could have hardware RAID-5 or RAID-6 arrays to cover disk failure, BTRFS SINGLE for data, and BTRFS RAID-1 for metadata. That would work reasonably well.
But really for PB scale ZFS is your only option.
3
u/Revolutionary_Click2 14d ago
I pretty much never run Alma on bare metal anyway. Alma is for VMs / guests; my hosts all run Proxmox, which is really just Debian and has excellent first-class ZFS support. Alma guests use XFS, which also avoids ZFS-on-ZFS write amplification issues, and because the underlying storage pool is RAIDz, I still get most of the benefits and protections of ZFS. Regular ZFS snapshots of guests from the host level, including before every system update managed by Ansible from my Semaphore controller, as well as nightly PBS backups help fill in the remaining gaps.
2
u/vetinari 13d ago
Don't.
Only on systems, that do ship ZFS with their vendor kernels (i.e. Ubuntu, Proxmox, anything else? [TrueNAS too, but there's not a much choice in choosing the filesystem]).
The chances of an update breaking your system is when, not if. The breakage increases it's chances when the update helpfully rebuilds initrds for your existing, working kernels and replaces the old, working zfs with the new, not-necessarily-broken-but-incompatible version. You need that working USB stick (or ILO image, or what have you) handy.
Or alternatively, if you really want root on ZFS, use another distribution. If you need just to mount a pool, with root elsewhere (and with nice working system when troubleshooting, not just emergency console), using RH-derived distro is fine.
(I've used Centos 7 with zfs dkms. Won't do it again. More trouble than benefits).
2
u/RetroGrid_io 13d ago
Mostly, that's the take I've been on for years. I hoped it had changed or matured somewhat.
It just seems really odd to have the "Enterprise" distro ignore the most "Enterprise" file system around, as numerous other distros eat their lunch. (shrug)
1
u/Both_Cup8417 14d ago
I know btrfs can do at least one of those, I think.
3
u/RetroGrid_io 14d ago
When I first heard of btrfs I was so excited!! However, in implementation, it just hasn't quite met muster for me. Once you replace a device in a btrfs pool, if that replacement device then fails, you lose your pool.
I found this many years ago in my reliability testing and it still was not fixed years after it was acknowledged. The problem is architectural and I haven't checked in a few years but with known issues like this, btrfs is a non-starter for me.
1
u/shadeland 12d ago
Is there still that write hole for RAID5/6?
In today's age of expensive media, I reallllly need that parity storage.
1
1
1
u/VagrantGato 13d ago
Zfsbootmenu has instructions for running almalinux on root. I did it for a while but then switched to ubuntu
7
u/Chester-Berkeley 14d ago
Everything Red Hat doesn't control is treated as a second-class citizen ¯\_(ツ)_/¯