My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

That's a firmware bug, not wear.

I ordered my NAS drives on Amazon, to avoid getting the same batch (all consecutive serial numbers) I used amazon.co.uk for one half and amazon.de for the other half of them. One could also stage the orders in time.

That depends entirely on how good their Q&A and manufacturing quality is - the better it is, the more likely eh?

Especially in an array where it’s possible every drive operation will be identical between 2 or 3 different drives.

This is the reason why I would always use RAID 6. A second drive failing during rebuild is significantly likely.

Drive like a maniac to the datacenter to shake 'em up a bit

Yeah. In a home environment you should absolutely use desktop gear. I have 5 80mm and one 120mm PWM fans in my NAS and they are essentially silent as they can't be heard over the sound of the drives (which is essentially the noise floor for a NAS).

It is necessary to use good PWM fans though if concerned about noise as cheaper ones can "tick" annoyingly. Two brands I know to be good in this respect are Be Quiet! and Noctua. DC would in theory be better but most motherboards don't support it (would require an external controller and thermal sensors I think).

> 80mm is the smallest standard fan (aside from 40mm's in the near-extinct Flex ATX PSU)

Those 40mm PSU fans, and the PSU, are what they are replacing with a DC bus bar.

It’s moderately smart - there’s a PID loop with per-component target temperatures, so it’s trying not to do more work than necessary.

(source: I wrote it, and it’s all published at https://github.com/oxidecomputer/hubris/tree/master/task/the... )

We also worked with the fan vendor to get parts with a lower minimum RPM. The stock fans idle at about 5K RPM, and ours idle at 2K, which is already enough to keep the system cool under light loads.

> just curious, are you associated with them, as these are very obscure youtube videos :D

Unassociated, but tech-y videos are often recommended to me, and these videos got pushed to me. (I have viewed other, unrelated Tech Day videos, so probably why I got that short. Also an old Solaris admin, so aware of Cantril, especially his rants.)

> Love it though, even the reduction in fan noise is amazing. I wonder why nobody had thought of it before, it seems so simple.

Depends on the size of the server: can't really expand fans with 1U or even 2U pizza boxes. And for general purpose servers, I'm not sure how many 4U+ systems are purchased—perhaps some more now that perhaps GPUs cards may be a popular add-on.

For a while chassis systems (e.g., HP c7000) were popular, but I'm not sure how they are nowadays.

Agree. I do weekly backups, the backup NAS is only switched on and off 52 times a year. After 5-6 years the disks are probably close to new in term of usage vs disks that have been running continuously over that same period.

Which leads to another strategy which is to swap the primary and the backup after 5 years to get a good 10y out of the two NAS.

There used to be some tutorials going around to flash the firmware to turn greens into reds I believe. Which simply disables the head parking.

Citations needed.

Counterpoints for each: Heads don't suffer wear when parking. The armature does.

If the platters are not spinning fast enough, or the air density is too low: the heads will crash into the sides of the platters.

The main wear on platter bearings is vibration, it takes an extremely long time for the lube to "gum up." If its still a thing at all. I suspect it used to happen because they were petroleum distilate lubes. So, shorter chains would evaporate/sublimate leaving longer more viscous chains. Or straight polymerize.

With fully synthetic PAO oils, and other options they won't do that anymore.

What inrush? They're polyphase steppers. The only reason for inrush is that the engineers didn't think it'd affect lifetime.

Counter: turn your drives off, thebsaved power of 8 drives being off half the day easily totals $80 a year- enough to replace all but the highest capacities.

> Keep them running [...] Bearings will last longer; they might also seize up if left stationary for too long. Likewise the drive motor.

All HDD failures I've ever seen in person (5 across 3 decades), were bearing failures, in machine that were almost always on with drives spun up. It's difficult to know for sure without proper A-B comparisons, but I've never seen a bearing failure in a machine where drives were spun down automatically.

It also seems intuitive that for mechanical bearings the longer they are spun up the greater the wear and the greater the chance of failure.

> The only reasons to turn your hard drives off are to save power, reduce noise, or transport them.

One reason some of my drives get powered down 99+% of the time is that its a way to guard against the whole network getting cryptolockered. i have a weekly backup run by a script that powers up a pair of raid1 usb drives, does and incremental no-delete backup, then unmounts and powers them back down again. Even in a busy week theyre rarely running for more than an hour or two. I'd have to get unlucky enough to not have the powerup script detect being cryptolockered (it checks md5 hashes of a few "canary files") and powering up the bav=ckup drives anyway. I figure that's a worthwhile reason to spin them down weekly...

Yes - although it's worth bearing in mind the number of load/unload cycles a drive is rated for over its lifetime.

In the case of the IronWolf NAS drives in my home server, that's 600,000.

I spin the drives down after 20 minutes of no activity, which I feel is a good balance between having them be too thrashy and saving energy. After 3 years I'm at about 60,000 load unload cycles.

Using power creates heat. Thermal cycles are never good. Heating parts up and cooling them down often reduces their life.

I think NAS systems in particular are often configured to prevent drives from spinning down

There seems to be a huge difference between spin-down while NAS is up vs shutting the whole NAS down and restart. When I start my NAS, it takes a bunch of time to be back up: it seems to do a lot of checking on/ syncing off the drives and puts a fair amount of load on them (same is true for CPU as well, just look at your CPU load right after startup).

OTOH, when the NAS spins up a single disk again, I haven't notice any additional load. Presumably, the read operation just waits until the disk is ready.

> Hard drives are often configured to spin down when idle for a certain time. This can cause many spinups and spindowns per day.

I was under the impression that this was, in fact, known to reduce drive longevity! It is done anyway in order to save power, but the reliability tradeoff is known.

No idea where I read that though, I thought it was "common knowledge" so maybe I'm wrong.

Yep, ZFS reads everything in the array and validates checksums. ZFS (at least on Linux) ships with scrub systemd timers: https://openzfs.github.io/openzfs-docs/man/master/8/zpool-sc...

https://en.wikipedia.org/wiki/ZFS?wprov=sfti1#Resilvering_an...

It's ZFS, so that's built-in. A scrub does precisely that.

Yes, zfs includes file-level checksums.

New filesystems seems to have a chicken and egg problem really. It's not like switching from Nvidia's proprietary drivers to nouveau and then back if it turns out they don't work that well. Switching filesystems, especially in larger raid setups where you desperately need more testing and real world usage feedback, is pretty involved, and even if you have everything backed up it's pretty time consuming restoring everything should things go haywire.

And even if you have the time and patience to be one of these early adopters, debugging any issues encountered might also be difficult, as ideally you want to give the devs full access to your filesystem for debugging and attempted fixes, which is obviously not always feasible.

So anything beyond the most trivial setups and usage patterns gets a miniscule amount of testing.

In an ideal world, you'd nail your FS design first try, make no mistakes during implementation and call it a day. I'd like to live in an ideal world.

Context. Linux regrets it because bcachefs doesn't have same commitment to stability as Linux.

Kent wants to fix a bug with large PR

Linux doesn't want to merge and review PR that touches so many non-bcachefs things.

They're both right in a way. Kent wants bcachefs to be stable/work good, Linus wants Linux to be stable.

After reading the email chain I have to say my enthusiasm for bcachefs has diminished significantly. I had no idea Kent was that stubborn and seems to have little respect for Linus or his rules.

As usual, the top comments in that submission are very biased. I think HN should sort comments in a random order in every polarizing discussion. Anyone reading this, do yourself a favor and dig through both links, or ignore the parent's comment altogether.

Linus "regretted" it in the sense "it was a bit too early because bcachefs is moving at such a fast speed", and not in the sense "we got a second btrfs that eats your data for lunch".

Please provide context and/or short human-friendly explanation, because I'm pretty sure most readers won't go further than your comment and will remember it as "Linus regrets merging bcachefs", helping spread FUD for years down the line.

i do this on my synoligy using btrfs. i’m still not convinced SSD caching gives any benefit for a home user. 5 spindle drives can already read and write faster than line rate on the NIC (1gbe) so what is the point of adding another failure point?

Drives are sealed anyway. Humidity maybe. Dust can’t really get in. Power or bad batch of drives.

They're airtight now (at the high end or enterprise level). They weren't airtight not very long ago and had filters to regulate the air exchange.

everything else isn't. the dust can get into power supplies and cause irregularities.

Only when checking for finger prints :-)

Well this is an other order of magnitude than 'spikes on the net'. The electrical field is so intense that current will easily cross large air gaps.

Nothing is really going to protect you from a direct lightning strike. Lightning strikes are on the order of millions of volts and thousands of amps. It will arc between circuits that are close enough and it will raise the ground voltage by thousands of volts too. You basically need a lighting rod buried deep into the earth to prevent it hitting your house directly and then you’re still probably going to deal with fried electronics (but your house will survive). Surge protectors are for faulty power supplies and much milder transient events on the grid and maybe a lightning strike a mile or so away.

Yes. In most cases, assuming you live in a 220V country, a surge protector will absorb the upwards spike, and the voltage range (a universal PSU can go as low as 107V) will handle the brownout voltage dip.

Pretty sure surge protectors are less effective against dips than they are spikes

Assuming you live in the Netherlands judging just by name: our power grid is pretty damn reliable with little shenanigans. I'd take that risk indeed.

Filters won't help against prolonged periods of higher/lower voltages though.

We don't really have to guess. Backblaze posted their stats for 4 TB HGST drives for 2024, and of their 10,000 drives, 5 failed. If OP's 2014 4 TB HGST drives are anything like this, then this is just snake oil and magic rituals and it doesn't really matter what you do.

it's (1-p)^24^10, where p is drive failure rate per year (assuming it doesn't go up over time). so at 1% that's about 9% or a 1/10 chance of this result. Not exactly great, but not impossible.

the backblaze rates are all over the place, but it does appear they have drives that this rate or lower: https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-...

The failure rate is not truly random with a nice normal distribution of failures over time. There are sometimes higher rates in specific batches or they can start failing altogether etc.

Backblaze reports always are interesting insights into how consumers drives behave under constant load.

I'm curious what the "Stop/Stop cycle count" is on these drives and roughly how many times per week/day you are accessing the server.

I have a 22TB RAID10 system out in my detached garage that works as an "off-site" backup server for all my other systems. It stays off most of the time. It's on when I'm backing up data to it, or if it's running backups to LTO tape. Or it's on when I'm out in the garage doing whatever project, I use it to play music and look up stuff on the web. Otherwise it's off, most of the time.

In the early '90s, some Quantum 105S hard drives had a "stiction" problem, and were shipped with Sun SPARCstations.

IME at the time, power off a bunch of workstations, such as for building electrical work, and probably at least one of them wouldn't spin back up the next business day.

Pulling the drive sled, and administering percussive maintenance against the desktop, could work.

https://sunmanagers.org/1992/0383.html

I debated posting because it felt like shitstirring. I think overwhelmingly what you're doing is right. And if a remote power on eg WOL works on the device, so much the better. If I could wish for one thing, it's mods to code or documentation of how to handle drive power down on zfs. The rumour mill is zfs doesn't like spindown.

What limitation is it downplaying? If there are hidden downsides to the proposed solution, I would certainly like to know.

Compared to a RAID setup, this requires a lot less money. It's really good for residential use.

I don’t know about other people but I run Plex because it lets me run my home movie collection through multiple clients (Apple TV, browser, phones, etc). iTunes works great with content bought from Apple, but is useless when playing your own media files from other sources.

I just want every Disney movie available so my kids can watch without bothering me.

Exactly that. I'm not even shooting in ProRes or similar "raw" video. But one video project easily takes 3TB. And I'm not even a professional.

Thanks, and all good - it was my fault for not following the link in this story to your post about the actual build, before starting on my mini-rant.

I'd heard the original ZFS memory estimations were somewhat exuberant, and recommendations had come down a lot since the early days, but I'd imagine given your usage pattern - powered on periodically - a performance hit for whatever operations you're doing during that time wouldn't be problematic.

I used to use mdadm for software RAID, but for several years now my home boxes are all hardware RAID. LVM2 provides the other features I need, so I haven't really ever explored zfs as a replacement for both - though everyone I know that uses it, loves it.

It was a handwavey rule of estimation for dedup, handwavey because dedup scales on number of records, which is going to vary wildly by recordsize.

Indeed. I'd started to add an aside to the effect of 'ten years ago it was probably easier to go ECC'. I'll add it here instead.

A decade ago if you wanted ECC your choice was basically Xeon, and all() Xeon motherboards would accept ECC.

I agree that these days it's much more complex, since you are ineluctably going get sucked into the despair-spiral of trying to work out what combination of Ryzen + motherboard + ECC RAM will give you actual, demonstrable* ECC (with correction, not just detection).

I have about about 80TB (my wife's data and mine) backed up to LTO5 tape. It's pretty cheap to get refurb tape drives on ebay. I pay about $5.00/TB for tape storage, not including the ~$200 for the LTO drive and an HBA card, so it was pretty economical.

As part of resilience testing I've been turning off our 24 drive backup drive array daily for two years, by flicking the wall switch. So far nothing happened.

I loved the idea around the 10Gb NICs but wondered what model switch are you connecting to? And what throughput is this topology delivering?

I use a wifi controlled powerpoint to power up and down a pair of raid1 backup drives.

A weekly cronjob on another (always on) machine does some simple tests (md5 checksums of "canary files" on a few machines on the network) then powers up and mounts the drives, runs an incremental backup, waits for it to finish, then unmounts and powers them back down. (There's also a double-check cronjob that runs 3 hours later that confirms they are powered down, and alerts me if they aren't. My incrementals rarely take more than an hour.)

Some bios and firmware support turning on at a certain time. Maybe cron was a way to simplify.

Just power, not unplugged. It's simply

0 2 * * * /usr/sbin/rtcwake -m off -s 28800 # off from 2am to 10am

"and unplugged" was referring to OP's setup, not mine

Off the shelf commodity home NAS systems with ZFS onboard?

eg: https://www.qnap.com/en-au/operating-system/quts-hero

IIRC (been a while since I messed with QNAP) QuTS hero would be a modded Debian install with ZFS baked in and a web based admin dasboard.

https://old.reddit.com/r/qnap/comments/15b9a0u/qts_or_quts_h...

As a rule of thumb (IMHO) steer clear of commodity NAS Cloud add ons, such things attract ransomware hackers like flies to a tip whether it's QNAP, Synology, or InsertVendorHere.

I feel that on HN people tend to be a bit pedantic about topics like data integrity, and in business settings I actually agree with them.

But for residential use, risks are just different and as you point out, you have no options except to only use a desktop workstation with ECC. People like/need laptops so that’s not realistic for most people. Just run Linux/freebsd with ZFS isn’t reasonable advice to me.

What I feel most strongly about is that it’s all about circumstances, context and risk evaluation. And I see so much blanket absolutist statements that don’t think about the reality of life and people’s circumstances.

“Tainting” the kernel doesn’t affect operations, though. You’re not allowed to redistribute it with changes — but you, as an entity, can freely use ZFS and the kernel together without restriction. Linux plus zfs works fine.

> - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL)

> - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel.

So modify the ZFS source so it appears as a external GPL module... just don't tell anyone or distribute it...

I can't say much about dracut or having to build the module from source... as a Gentoo user, I do it about once a month without any issues...

> I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+.

I've been running btrfs as the primary filesystem for all of my desktop machines since shortly after the on-disk format stabilized and the extX->btrfs in-place converter appeared [0], and for my home servers for the past ~five years. In the first few years after I started using it on my desktop machines, I had four or five "btrfs shit the bed and trashed some of my data" incidents. I've had zero issues in the past ~ten years.

At $DAYJOB we use btrfs as the filesystem for our CI workers and have been doing so for years. Its snapshot functionality makes creating the containers for CI jobs instantaneous, and we've had zero problems with it.

I can think of a few things that might separate me from the folks who report issues that they've had within the past five-or-ten years:

* I don't use ANY of the built-in btrfs RAID stuff.

* I deploy btrfs ON TOP of LVM2 LVs, rather than using its built-in volume management stuff. [1]

* I WAS going to say "I use ECC RAM", but one of my desktop machines does not and can never have ECC RAM, so this isn't likely a factor.

The BTRFS features I use at home are snapshotting (for coherent point-in-time backups), transparent compression, the built-in CoW features, and the built-in checksumming features.

At work, we use all of those except for compression, and don't use snapshots for backup but for container volume cloning.

[0] If memory serves, this was around the time when the OCZ Vertex LE was hot, hot shit.

[1] This has actually turned out to be a really cool decision, as it has permitted me to do low- or no- downtime disk replacement or repartitioning by moving live data off of local PVs and on to PVs attached via USB or via NBD.

I had a significant data loss years ago.

I was young and only had a deskop, so all my data was there.

So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good.

Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young with much less disposable income).

Eventually I purchased a 500GB drive.

But what I didn't know is my desktop drive was dying. Bits were flipping, a lot of them.

So when I did my first backup with the new drive I copied all my data off my desktop along with the corruption.

It was months before I realised a huge amount of my files were corrupted. By that point I'd wiped the old backup drive to give to my Mum to do her own backups. My data was long gone.

Once I discovered ZFS I jumped on it. It was the exact thing that would have prevented this because I could have detected the corruption when I purchased the new backup drive and did the initial backup to it.

(I made up the drive sizes because I can't remember, but the ratios will be about right).

To paraphrase, "ZFS is the worst filesystem, except for all those other filesystems that have been tried from time to time."

It's far from perfect, but it has no peers.

I spent many years stubbornly using btrfs and lost data multiple times. Never once did the redundancy I had supposedly configured actually do anything to help me. ZFS has identified corruption caused by bad memory and a bad CPU and let me know immediately which files were damaged.

> No serious person designing a filesystem today would say it's okay to misplace your data.

Former LimeWire developer here... the LimeWire splash screen at startup was due to experiences with silent data corruption. We got some impossible bug reports, so we created a stub executable that would show a splash screen while computing the SHA-1 checksums of the actual application DLLs and JARs. Once everything checked out, that stub would use Java reflection to start the actual application. After moving to that, those impossible bug reports stopped happening. With 60 million simultaneous users, there were always some of them with silent disk corruption that they would blame on LimeWire.

When Microsoft was offering free Win7 pre-release install ISOs for download, I was having install issues. I didn't want to get my ISO illegally, so I found a torrent of the ISO, and wrote a Python script to download the ISO from Microsoft, but use the torrent file to verify chunks and re-download any corrupted chunks. Something was very wrong on some device between my desktop and Microsoft's servers, but it eventually got a non-corrupted ISO.

It annoys me to no end that ECC isn't the norm for all devices with more than 1 GB of RAM. Silent bit flips are just not okay.

Edit: side note: it's interesting to see the number of complaints I still see from people who blame hard drive failures on LimeWire stressing their drives. From very early on, LimeWire allowed bandwidth limiting, which I used to keep heat down on machines that didn't cool their drives properly. Beyond heat issues that I would blame on machine vendors, failures from write volume I would lay at the feet of drive manufacturers.

Though, I'm biased. Any blame for drive wear that didn't fall on either the drive manufacturers or the filesystem implementers not dealing well with random writes would probably fall at my feet. I'm the one who implemented randomized chunk order downloading in order to rapidly increase availability of rare content, which would increase the number of hard drive head seeks on non-log-based filesystems. I always intended to go back and (1) use sequential downloads if tens of copies of the file were in the swarm, to reduce hard drive seeks and (2) implement randomized downloading of rarest chunks first, rather than the naive randomization in the initial implementation. I say naive, but the initial implementation did have some logic to randomize chunk download order in a way to reduce the size of the messages that swarms used to advertise which peers had which chunks. As it turns out, there were always more pressing things to implement and the initial implementation was good enough.

(Though, really, all read-write filesystems should be copy-on-write log-based, at least for recent writes, maybe having some background process using a count-min-sketch to estimate locality for frequently read data and optimize read locality for rarely changing data that's also frequently read.)

Edit: Also, it's really a shame that TCP over IPv6 doesn't use CRC-32C (to intentionally use a different CRC polynomial than Ethernet, to catch more error patterns) to end-to-end checksum data in each packet. Yes, it's a layering abstraction violation, but IPv6 was a convenient point to introduce a needed change. On the gripping hand, it's probably best in the big picture to raise flow control, corruption/loss detection, retransmission (and add forward error correction) in libraries at the application layer (a la QUIC, etc.) and move everything to UDP. I was working on Google's indexing system infra when they switched transatlantic search index distribution from multiple parallel transatlantic TCP streams to reserving dedicated bandwidth from the routers and blasting UDP using rateless forward error codes. Provided that everyone is implementing responsible (read TCP-compatible) flow control, it's really good to have the rapid evolution possible by just using UDP and raising other concerns to libraries at the application layer. (N parallel TCP streams are useful because they typically don't simultaneously hit exponential backoff, so for long-fat networks, you get both higher utilization and lower variance than a single TCP stream at N times the bandwidth.)

The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues, never so with other file systems. The data always comes back, it's just that it needs petting.

Earlier, there used to be lot of fears around the license, with Torvalds advising against its use, both for that reason and for lack of maintainers. Now i believe that has been mostly ironed out and should be less of an issue.

> the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist.

There were good filesystems before ZFS. I would love to have a versioning filesystem like Apollo had.

How are the memory overheads of ZFS these days? In the old days, I remember balking at the extra memory required to run ZFS on the little ARM board I was using for a NAS.

That's all fine and good until that one random lone broken block stops you from opening that file you really need.

Most people run Windows on their Laptop (without ReFS), and many people use paid data restore services if something "important" gets missing/corrupt.

>let’s not pretend it’s such a huge deal

Depends on the importance of you data right?