Setup

Debian 13, kernel 6.12.101, OpenZFS. Single pool tank, one mirror vdev, 2× WD Red 4TB. Non-ECC RAM. HBA is a 10Gtek LSI SAS3008, firmware 16.00.10.00, currently in IR mode (not IT).

• sda — WD40EFRX-68N32N0, 57,191 hours, SMART completely clean (0 reallocated, 0 pending, 0 UDMA CRC, 0 ICRC, 0 R_ERR)
• sdb — WD40EFZX-68AWUN0

Timeline

July 2026: ZFS starts throwing errors, one drive drops out of the mirror. Errors reproduce reliably under sustained large-file writes, with frequent SATA link resets and COMRESET. Over several days I swapped, one variable at a time: PicoPSU replaced with a full ATX supply (errors identical), all cables, both ports, and finally the onboard AHCI controller replaced with the SAS3008 HBA. Symptoms stopped after the HBA swap.

The suspect drive was pulled and tested on a separate machine — 40 GB sustained dd write at 208–215 MB/s, zero errors. It went back into the mirror.

After that scrub completes. scrub repaired 0B in 06:57:52 with 148 errors. 296 CKSUM on each drive — identical counts on both independent members, and repaired 0B. Two permanent errors remain, listed only as objset IDs (<0x667f>:<0x161a>), which no longer resolve because that dataset has since been destroyed. No I/O errors in dmesg on that date. My reading: these are historical, written while the faulty onboard controller was still in play. No new CKSUM has accrued since.

29 August, 02:00:42: Sustained write load (scheduled replication from another machine). Burst of failures on sdb only, all within a single second:

sd 0:0:1:0: [sdb] Sense Key : Hardware Error [current]
             Add. Sense: Internal target failure
critical target error, dev sdb, sector 3000 op 0x1:(WRITE)
zio pool=tank vdev=...WX32D121YDNL-part1 error=121

Failing offsets covered all four vdev labels (0, 225280, 270336, 487424 and the end-of-disk pair) plus one data offset, read and write simultaneously. error=121 is EREMOTEIO. No link reset, no COMRESET — completely different signature from July.

Since then: smartctl -x on sdb fails with INQUIRY failed while the device node is still enumerated with stale by-id symlinks. Hot rescan across every scsi_host brings nothing back. Device is now entirely gone from lsblk. Drive is offlined, pool is DEGRADED on sda alone.

Already ruled out

PSU (swapped for full ATX under the same reproducing load, no change), cables, ports, onboard controller, and the drive itself (tested clean on another machine after being condemned once already).

What I'm stuck on

The invariant across two months is that failures only ever appear under sustained write load — never at idle. What changed with the HBA swap wasn't the trigger, only how it gets reported: AHCI reported link resets, mpt3sas reports internal target failure and drops the device. That makes me doubt the HBA is the root cause rather than the latest messenger.

I've been advised to memtest, but the corruption is clustered in a specific time window (late July / early August snapshots), stopped when the controller changed, and never produced a single segfault or any corruption on the NVMe root — with Docker, Immich and Postgres running 24/7 on the same RAM. Memtest is still on the list, but the history doesn't fit RAM well.

CPU and motherboard are the only components never swapped or tested, and 30 minutes of fio plus stress-ng on the NVMe (bypassing the HBA path entirely) produced nothing.

Questions

1. Is IR mode on a SAS3008 a plausible explanation for a drive being quarantined by firmware — device node retained, INQUIRY refused, no re-enumeration on rescan? Worth flashing to IT before anything else?
2. Does Internal target failure / EREMOTEIO with all four labels failing simultaneously read as drive death, or as the HBA giving up on the target?
3. How would you build a write test on the HBA path that fails on demand? Everything so far has been validated by absence of symptoms, which has already fooled me once.
4. Anyone seen 10Gtek SAS3008 cards misbehave thermally under sustained load without active cooling?

so im basicly getting a nas to start my decloud project. bitrot protection is why im choosing zfs, and using truenas instead og ugreen/terramaster. i dont see my usage is going to increase dramaticly, i have around 2-3tb of cloud data today. Mainly will be images and videos for immrich, and some backups from proxmoxhost

Initially i was dead set on a 2 bay NAS, coupled with 2x 8TB drives in mirror after reading lots of guides recommending this. With todays HDD prices though "wasting 50% vs 25-33%" is hurting more than id like to admit.

Would i be better off 3x 4TB drives here in raidz1 with a 4bay nas? any advice appreciated greatly

OpenZFS for Windows 2.4.1 rc15    

https://github.com/openzfsonwindows/openzfs/releases/tag/zfswin-2.4.1rc15
https://github.com/openzfsonwindows/openzfs/issues

** rc15

• Fixed a BSOD (MEMORY_MANAGEMENT) that could occur running .exe files directly from a ZFS volume, caused by a Valid Data Length race during cached writes.
• Fixed data corruption during large downloads to a ZFS volume (e.g. LM Studio model downloads) — FSCTL_QUERY_FILE_REGIONS was reporting invalid data regions incorrectly, and several related races/deadlocks in the cached-write fast path (fastio_write) under concurrent writers have been fixed.
• Fixed a second, related MEMORY_MANAGEMENT BSOD that could occur when a file was reopened while another handle already had it memory-mapped.
• Fixed filenames being mangled to the wrong case when a pool is mounted to a folder (reparse point) instead of a drive letter.
• Fixed pool mount failures when Driver Verifier is enabled.
• Fixed partition number (dki_partition) always reporting 0 on Windows.
• Healed pre-existing non-power-of-2 block sizes that could otherwise cause issues when a file's block size grows.
• Fixed several bugs in zfsinstaller found during code review.
• Fix BSOD in shani benchmark at boot

I want to switch to ZFS [mainly for my incus containers] and also want the root on ZFS and ZFS bootmenu.

Is CachyOS or ubuntu better for a work PC? I don't want to spend extra time configuring or wasting time.

I think CachyOS as slightly more options for ZFS boot menu but not sure.

What do you suggest?

I read lots of stuff saying that zfs (if given a special vdev for small blocks) parks the start of the file on the special vdev and the main data on the main data vdev (hopefully the former on sdd or nvme, the latter prolly on spinning rust). BUT when a just do a file copy from my camera's SD card, the program is just going to read all equal sized chunks until it gets EOF, so zfs will never see that the front of the file is special and handle it differently? Or is there something else going on I don't know about?

Setup: 2x4TB WD Red Plus mirror on a ZimaBlade, TrueNAS SCALE 25.10.5, ZFS 2.3.4. Pool went offline after a brief version upgrade that I reverted shortly after. zpool import has failed for about two weeks now, in every mode and flag combination.

**Where things stand (with a lot of help from someone on the OpenZFS forum working from exact source references ; full thread here: https://github.com/openzfs/zfs/discussions/18954

• SMART, disk labels, uberblocks, partition tables ; all clean on both disks. No hardware issue.
• zdb -e fully loads the pool's trusted config ; the pool structure and data are demonstrably intact and readable.
• Root cause of the import failure: spa_load_verify (spa.c:2831) unconditionally walks every dataset via dmu_objset_find_dp before allowing import, regardless of flags. One object ; a DSL props ZAP (object 293) ; fails checksum (ECKSUM) during that walk, which aborts the entire import. No combination of -f, -F, -X, -o readonly=on, -N, or spa_load_verify_metadata=0 bypasses this; -F/-X actually make it worse since they re-enable full data verification.
• A full zdb -bcsvL traversal found only 8 checksum errors out of 29.2 million block pointers total, and every one is confined to pool-internal bookkeeping (SPA space map, DSL props) or disposable .system/samba4 and .system/netdata-* service datasets. None of my actual data (Photos, Documents, Archived, etc.) shows any corruption.

Current blocker: Since import can't succeed, the suggested path is zdb -B (backup stream extraction ; reads a dataset directly, bypassing the import verification entirely). But every dataset I try ; encrypted or not, parent or child dataset, various sizes ; fails identically:

dump_backup: dmu_send_obj: Permission denied

Running as root via sudo, confirmed with id. Output file is always 0 bytes, exit code 0. The zfs_dbgmsg buffer (checked via zdb -G) shows nothing related to this error ; the pool-load trace completes normally and logs LOADED, but the permission denial itself isn't logged anywhere I can find.

zdb -O/-r (path-based lookups, a different code path) also fail, but with a different error ; Input/output error ; so this may be a separate issue from the -B failure.

What I'm hoping someone here can help with:

• Has anyone seen dmu_send_obj: Permission denied from zdb -B before? Trying to figure out if this is a known/expected behavior, a TrueNAS SCALE-specific build difference, or something environment-specific I'm missing.
• Any idea what permission check dmu_send_obj performs that would fail even under root, and whether there's a way to satisfy it or get more diagnostic detail out of it?

Not time-pressured; data is important but not critical/urgent, just want to get it off before doing anything destructive to the pool. Happy to provide any further diagnostics.

Also posted in r/truenas; will update both if I get an answer anywhere.

Hi guys,

In the last few weeks I developed a tool in Python that provides a web interface to help browse your snapshots more easily. For each directory and file, it shows when they changed from one snapshot to another, which makes tracking history very straightforward.

It auto-discovers datasets and their snapshots, and even works without zfs-utils available (e.g., if the filesystem is mounted over NFS), as long as snapshot names can be parsed into timestamps for chronological sorting.

You can run it directly or as a Docker container with read-only mounts (:ro). It requires no persistent storage and no database.

Initially, I worked on the backend logic myself until everything ran properly. Since frontend isn't my main area of expertise, I used Gemini Flash in Antigravity IDE to give the UI a clean finish.

GitHub: https://github.com/NicolasGoeddel/zfs-snapshot-explorer

Let me know what you think or if you run into any edge cases!

Edit: New version available: https://github.com/NicolasGoeddel/universal-snapshot-explorer/releases

I almost ran out of space because I didn’t realize how many duplicate model files and cache folders were sitting on my drives. After a lot of trial and error, I put together a cleanup guide focused on LLMs, but the same method works for diffusion models too. It covers the biggest space hogs and the commands to remove them safely.

https://interconnectd.com/forum/thread/233/fix-disk-space-full-from-llms-ultimate-cleanup-guide/

I am in dire need and shall try to be succinct to respect the time of anyone who might be willing to help.

Currently running TrueNas Scale (Goldeneye). TLDR; Post upgrade pool is reverted to a many years old state.

TrueNas Core U13, 6 drive, RaidZ2. Had a drive fail, replaced under warranty. Resilvered -> Migrated to TrueNas Scale via fresh install -> Restored Config from File -> Upgraded ZFS Pool. Seemed fine, everything completed without errors, zpool status was healthy. Having recently moved workstation(s) to Linux I began transitioning from SMB to NFS and that's when I noticed the pool was in a severely regressed state. Old datasets present, newer datasets missing etc. As an estimate I would say it looks like at least 2-3 years of reversion. I have automated snapshots but only of certain datasets and with different frequencies and retentions, so the state is pool-wide, not representative of any combination of snapshots.

Began diagnostics with some cautious AI assistance, but nothing useful. `list`, `history`, 'status' et al just showed a record of the automated snapshots I run, 31 uberblocks all created in the past 20 minutes. No sign of any missing datasets. Looking at txg's each drive (part2) has the same number, but that just maps to one made recently, post the problem. Eventually around 2AM I exported the pool and that's the state the system is currently in.

I know "raid is not a back-up" but having a full replication of a pool this size is just entirely out of budget and climbing hardware prices has pushed the possibility even further away. I have 4TB offsite, a few 100Gb in cloud, and another few TB in active storage on the workstation but if the pool is permanently "time-travelled" as I fear it might be, a great amount of data is lost.

Is there anything I can do, what should my next steps be, and how on earth did this happen? I'm not an admin by trade, but I've tried to learn enough to support my business and I don't understand what went wrong.

Thank you in advance for anyone who is willing to provide advice.

(Cross-posted in the TrueNas forum)

napp-it cs has now all major features of napp-it se (Solaris edition since 2009)
plus many new unique features, update see menu About > Update

napp-it cs 26.08.23.15
- Localisation en,cn,de,es,fr,it,ru
starting with menus (About > Settings
with fast en switch in topmenu)

26.08.21.23
-security audit and hardening
-iSCSI targets (Linux,Illumos,Solaris)
-iSCSI initiators (Linux,Illumos,Solaris,Windows)
-Target zvol -> replicate -> import target=preserve target LU/NA

About 6 weeks ago I posted here about a site I built comparing hard drives: CMR vs SMR, and real Backblaze failure rates per model.

It's a free tool to help you select drives, and it got a lot of interest, and the one question that kept coming up was "what about SSDs?" So I added them.

Since this is a ZFS subreddit, SSDs here aren't just boot drives. They end up as a SLOG, an L2ARC, a special vdev, or a full all-flash pool - the exact spots where a drive's behavior under load actually matters.

And that's the catch: the specs that decide how an SSD behaves in a pool aren't printed on the box. Two drives that look identical on the product page can be completely different underneath.

The three that bite people most:

• QLC is the SMR of SSDs. Fast in reviews thanks to the SLC cache, then it falls off a cliff on sustained writes. Fine for light use, painful mid-resilver.
• No PLP means unsafe sync writes. The whole point of a SLOG is safe sync writes, and without power-loss protection a bad power cut can drop writes the drive already acknowledged. Most consumer NVMe has none.
• DRAM-less, low-endurance drives quietly struggle under the constant small writes a pool throws at them.

So I built the SSD side to filter on exactly those things: interface, TLC/QLC, DRAM, PLP, and endurance (TBW/DWPD), with price per TB where I have it. You can screen out the drives that only look good on paper before you buy.

One honest note: there's very little public per-model failure data for SSDs, so I left the failure-rate column off rather than guess. For flash, those specs are the better tell anyway.

SSDs: https://www.nasdisks.com/ssd/
CMR vs SMR (the original): https://www.nasdisks.com/
Disclosure: the site has Amazon affiliate links, that's how I cover hosting.

If there's a drive you think belongs in there, tell me and I'll add it.

I've been doing homelab/selfhosting since more than 15 years. Like many, I started with commercial solutions like Synology and QNAP, trying not to "fall" into their proprietary apps designed to keep you locked in. I never liked their software and they ended up being just data "storage", with the actual apps running on another CPU.

The evolution was the logical one: go from the commercial NAS to a lightweight TrueNAS microserver, and some time later, merge server + storage back into a single machine (it's a homelab). I tried TrueNAS under Proxmox but it was a pain: slow load times, a VM wasting resources, containers waiting on reboots, and the host has no visibility or control over the pools.

So I moved everything to Proxmox and honestly it was the best decision, although I found it hard to remember all the commands, so I started using scripts. Later I missed a visual panel to see everything "at a glance" with notifications, and I couldn't find anything simple, lightweight, non-intrusive and very visual, so I started developing my own panel. I've been using it for a while and went through everything (failed disks, bad cables, RAID rebuilds...). I kept improving and polishing the app, passing it to other colleagues in similar situations, and with everything I learned, I got motivated to do a final pass on bugs and small improvements (thanks AI), and release it under AGPL in case it's useful for other people.

It's free and open source, I don't ask anything in return. If you like it I'd love to know, and if you have some constructive criticism, same.

You can see it at https://easyzfs.cloudless.club/

I've been looking into power saving for ZFS servers and found that:

- there is no cross-OS approach
- there is no approach that is ZFS pool-based instead of disk-based and, for example, does not sleep the SLOG
- there is no approach that supports active pools with VMs (Proxmox hibernate)
- there is no approach that treats backup pools differently (sleep + export/import)
- there is no approach that allows time windows for both forced sleep and optional sleep

__cs-sleeper — Highlights__

cs-sleeper is a standalone Go daemon that sends idle ZFS disks to standby to save power — the key difference from classic OS/disk sleep: it thinks in pools, not in individual disks. The disks of a pool are managed together, differentiated by two pool types: active pools with VMs (disks go to standby, the pool stays imported, optionally coupled with VM suspend/shutdown) and backup pools (fully exported/taken offline when idle and re-imported on demand). It is replication-aware (no sleep/export during a running zfs send/receive) and never touches boot disks or SLOG/L2ARC/special/dedup flash devices. Waking up is fast: a pool's disks are woken in parallel (not sequentially), and automatic on-access wake additionally wakes a disk transparently on access. The time control is flexible: activity windows (e.g. 12–14, 18–6) determine when sleep is allowed at all, and one-shot scheduled actions (--at HH:MM) trigger sleep/wake for a specific pool at a fixed time.

[](https://github.com/guenther-alka/cs-sleeper/)<https://github.com/guenther-alka/cs-sleeper/> (part of the napp-it cs ZFS management suite, but also works standalone)

Further napp-it cs tools (also work standalone)

- cs-sync triple endpoint uni/bi sync, e.g. S3/SMB/backup sync

- cs-send alert/status mails/chat messages

- cs-stream encrypted zfs send/filesync with transfer buffer and limits

- cs-freeze4snap ZFS pool-based method for safe and restorable states of Proxmox VMs

Every clever thing ZFS does comes from one decision taken before a line of it was written, and the decision was to stop pretending that a volume manager and a filesystem are two different problems. …

My first time building a homelab, using:

- Machine: HP Proliant ML310e G8 (2013)

- OS: Gentoo (OpenRC)

- Initramfs generator: Ugrd (Loading the ZFS module)

- Bootloader: Limine

- Partition layout: GPT + BIOS (Machine doesnt have UEFI)

- Boot partition: In a SD card

- zpool config: 4 3TB SAS disks in RAIDZ1 using device names (/dev/sdx)

- Disks: disks uses 512byte sectors, used ashift=9 (following the Handbook)

So, i installed the system following the Gentoo Handbook, they have a specific entry related to ZFS. Only thing done different is by being a RAIDZ1 pool

And when booting, the system isnt importing the pools (zfs import and mount services are enable and in the sysinit level)

At the moment from what i searched and tracked from the files

Some possibilities to fix (gonna try tomorrow)

1. Generate zfs.cache in /etc/zfs. It doesnt exist, and from what i understood from the zfs-import service in the init.d. Its needed?

2. Delay the service startup by 30 seconds using sleep. Saw some people doing that for HDDs that take some time to spin

3. Change the zpool disks, from device names (/dev/sdx) to device IDs (ata-xxxxxxxxx...) (Arch Wiki has a warning)

I dont know if they will work, since im gonna try tomorrow after i arrive from work

Forgot to say that to mount the disks in the live CD, i can only import the pools by generating a hostid with the same one from when it was created for the first time.

so, first zgenhostid -f xxxxxxxx then zpool import -N -R mount pool. then mount with zfs mount

If someone have more ideas, i appreciate that and leave in the comments

I just upgraded a couple of my fileservers as a test, and `sg3-utils-udev` stopped working! Suddenly all my physical devices were wwn-* or scsi-35* instead of their proper names.

It appears to be a regression. I spotted this Ubuntu bug which seems to be the closest to the cause, and posted a simple fix on it.

Put this in /etc/udev/rules.d/00-scsi-sg3_config.rules

ACTION!="add|change", GOTO="scsi_identify_end"
SUBSYSTEMS=="scsi", GOTO="scsi_identify"
GOTO="scsi_identify_end"
LABEL="scsi_identify"
ENV{ID_SCSI_INQUIRY}="0"
ENV{.SCSI_SYMLINK_SRC}="TLVS"
LABEL="scsi_identify_end"

Then udevadm control -R and udevadm trigger and you'll have all your device links back.

You then have to do the mildly annoying fix of

cd /dev/disk/by-id
rm -f wwn* scsi-35* nvme-eui*
zpool export -a
zpool import -d /dev/disk/by-id -a
zpool status

That should re-import all your zpools with the correctly named volumes.

Background

After setting up a syncoid backup on my main zpool, I realized rsyncing my desktop to a zpool is janky and horrible. Instead of reinstalling Void with zfsbootmenu, I decided to do it in-place; /home already has backups, so worst consequence is I reinstall anyway.

File-level backups give me the heebie-jeebies. Several years ago I trusted Deja-Dup/Duplicity, and it couldn't handle the sheer size of my backup when I needed a full restore. rsync is leagues better but still an extremely slow, potentially imperfect copy that can fail.

Here's what I did:

• Original state: 80 GiB / and 300 GiB /home on a single 1 TB ext4 partition

• Boot into hrmpf and install ZFS to the livecd environment

• Shrink the ext4; e2fsck; create a full-size partition for a single-vdev pool with -o mountpoint=none and a /home dataset with -o mountpoint=/home

• Mount the / partition to e.g. /mnt/root

• rsync -aHAXP --numeric-ids /mnt/root/home/ /home/

• (ideally verify transfer); rm -rf /mnt/root/home

• umount /mnt/root

• Shrink the ext4 to file size + buffer; 150 GiB in my case; e2fsck

• Make a third partition; zpool add

• Create child datasets with syncoid:no-sync=true for cache/junk files

• Chroot into /mnt/root to ensure ZFS is installed (don't have 30 old kernels like me or it'll take hours!), and to add zpool import -a and zfs mount -a to rc.local or whatever systemd uses.

• Set and verify UID/perms on all the mountpoints

Results

It was quite a slow and dangerous operation, and there was one instance of metadata corruption after the second shrink, which e2fsck fixed. Also nearly fat-fingered the new partition overtop the others. However, it worked out in the end. Striping across the two vdevs should be safe because they're partitions on the same NVMe anyways. If I did reinstall, the transfer still would've taken at least 12 hours because AT&T Fiber LAN is limited to 10/100.

Killer App: syncoid

If you haven't tried syncoid (which comes with sanoid), you should. It's a fancy script wrapper around zfs send | zfs receive. By default, it replicates a dataset (+ child datasets with -r) to a dataset on another pool along with snapshots; by default it creates a temporary snapshot to synchronize and resume interrupted transfers.

The big advantage is that I can create exact copies of my pools without any configuration, and use the same command in reverse to restore the pool if necessary. It does NOT copy pool attributes because that would interfere with the target dataset's needed attributes.

In my case, my workstation can now run

syncoid pool/home glados:slab/home-backup

In turn, GLaDOS has an @hourly crontab to run

syncoid -r slab backup/slab

When you need to recover a pool/dataset, simply zfs destroy pool/home if necessary and run syncoid in reverse

syncoid glados:slab/home-backup pool/home

Im trying to get it working

Im at this point in the guide

By default ZFS kernel modules are loaded upon detecting a pool. To always load the modules at boot:

echo zfs > /etc/modules-load.d/zfs.conf
By default ZFS kernel modules are loaded upon detecting a pool.
To always load the modules at boot:
echo zfs > /etc/modules-load.d/zfs.conf

I dissabled secure boot to get the previous step to work, this is my input:

~$ sudo echo zfs > /etc/modules-load.d/zfs.conf
-bash: /etc/modules-load.d/zfs.conf: Permission denied