Setup

Debian 13, kernel 6.12.101, OpenZFS. Single pool tank, one mirror vdev, 2× WD Red 4TB. Non-ECC RAM. HBA is a 10Gtek LSI SAS3008, firmware 16.00.10.00, currently in IR mode (not IT).

• sda — WD40EFRX-68N32N0, 57,191 hours, SMART completely clean (0 reallocated, 0 pending, 0 UDMA CRC, 0 ICRC, 0 R_ERR)
• sdb — WD40EFZX-68AWUN0

Timeline

July 2026: ZFS starts throwing errors, one drive drops out of the mirror. Errors reproduce reliably under sustained large-file writes, with frequent SATA link resets and COMRESET. Over several days I swapped, one variable at a time: PicoPSU replaced with a full ATX supply (errors identical), all cables, both ports, and finally the onboard AHCI controller replaced with the SAS3008 HBA. Symptoms stopped after the HBA swap.

The suspect drive was pulled and tested on a separate machine — 40 GB sustained dd write at 208–215 MB/s, zero errors. It went back into the mirror.

After that scrub completes. scrub repaired 0B in 06:57:52 with 148 errors. 296 CKSUM on each drive — identical counts on both independent members, and repaired 0B. Two permanent errors remain, listed only as objset IDs (<0x667f>:<0x161a>), which no longer resolve because that dataset has since been destroyed. No I/O errors in dmesg on that date. My reading: these are historical, written while the faulty onboard controller was still in play. No new CKSUM has accrued since.

29 August, 02:00:42: Sustained write load (scheduled replication from another machine). Burst of failures on sdb only, all within a single second:

sd 0:0:1:0: [sdb] Sense Key : Hardware Error [current]
             Add. Sense: Internal target failure
critical target error, dev sdb, sector 3000 op 0x1:(WRITE)
zio pool=tank vdev=...WX32D121YDNL-part1 error=121

Failing offsets covered all four vdev labels (0, 225280, 270336, 487424 and the end-of-disk pair) plus one data offset, read and write simultaneously. error=121 is EREMOTEIO. No link reset, no COMRESET — completely different signature from July.

Since then: smartctl -x on sdb fails with INQUIRY failed while the device node is still enumerated with stale by-id symlinks. Hot rescan across every scsi_host brings nothing back. Device is now entirely gone from lsblk. Drive is offlined, pool is DEGRADED on sda alone.

Already ruled out

PSU (swapped for full ATX under the same reproducing load, no change), cables, ports, onboard controller, and the drive itself (tested clean on another machine after being condemned once already).

What I'm stuck on

The invariant across two months is that failures only ever appear under sustained write load — never at idle. What changed with the HBA swap wasn't the trigger, only how it gets reported: AHCI reported link resets, mpt3sas reports internal target failure and drops the device. That makes me doubt the HBA is the root cause rather than the latest messenger.

I've been advised to memtest, but the corruption is clustered in a specific time window (late July / early August snapshots), stopped when the controller changed, and never produced a single segfault or any corruption on the NVMe root — with Docker, Immich and Postgres running 24/7 on the same RAM. Memtest is still on the list, but the history doesn't fit RAM well.

CPU and motherboard are the only components never swapped or tested, and 30 minutes of fio plus stress-ng on the NVMe (bypassing the HBA path entirely) produced nothing.

Questions

1. Is IR mode on a SAS3008 a plausible explanation for a drive being quarantined by firmware — device node retained, INQUIRY refused, no re-enumeration on rescan? Worth flashing to IT before anything else?
2. Does Internal target failure / EREMOTEIO with all four labels failing simultaneously read as drive death, or as the HBA giving up on the target?
3. How would you build a write test on the HBA path that fails on demand? Everything so far has been validated by absence of symptoms, which has already fooled me once.
4. Anyone seen 10Gtek SAS3008 cards misbehave thermally under sustained load without active cooling?