datahoarder
Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
view the rest of the comments
Use ZFS instead of ext4... it has a lot built into it which protects data integrity, and it's always been suitably fast for me. My current large arrays use raid-z2 (which is equivalent to RAID6, having two drives of redundancy), however I started out with the insane configuration of five drives in a raid0, lost power and dropped one of more drives several times, and was able to bring it back up without any data loss. I would NOT recommend this to anyone, but it took me time to learn that my drives were dropping due to a poor quality power supply, and the recovery impressed me so much that I have always used ZFS since then.
There are a few optimizations you can do when you set up your array such as configuring the stripe width and sector sizes, although I believe the default settings on these are pretty ideal now and may no longer require tweaking.
As for overall speed... My most recent array is built from eight 18TB drives, formatted out to around 90TB of usable space. While designing my external rack I knew that HDDs can't really reach SATA3 speeds, but an array can out-perform that by accessing multiple drives at once. I built my assembly with cheap SATA2 backplanes and LSI SAS cards. Even right now with the array in use by multiple servers I am still getting speeds between 483-597MB/s while copying 10GB of random data and nearly 900MB/s copying from /dev/zero. You're obviously not going to see that kind of speed from USB-connected drives, but the point is that ZFS itself will not slow you down in any way.
Yeah, seems like ZFS is an option in my setup.