MUFF WIGGLER Forum Index
 FAQ & Terms of UseFAQ & Terms Of Use   Wiggler RadioMW Radio   Muff Wiggler TwitterTwitter   Support the site @ PatreonPatreon 
 SearchSearch   RegisterSign up   Log inLog in 
WIGGLING 'LITE' IN GUEST MODE

Let's talk RAID storage setups
MUFF WIGGLER Forum Index -> General Gear Goto page 1, 2  Next [all]
Author Let's talk RAID storage setups
Rex Coil 7
Me thinks I understand the essential concept of what a RAID system does, and (more or less) why they are useful. More or less. Or something. Kinda.

If I get it, several drives are used as an array. Data is moved around to the most healthy sectors, and there are redundant copies of data stored across several physical drives. When one drive dies, pull it out of the array, replace it with a new one, go about your merry way. The data is safe since it was stored as redundant copies that were spread out across the other drives in the array. Is that correct?


** So is there some type of "frame" or "case" that drives are mounted in?

** Does a RAID array require some sort of software?

** If so, which ones (brands/models) are best suited for music studio use?

** How are they connected to the motherboard? (SATA3? USB3? Other?).

** Is there a "best" strategy to use (smaller HDDs, larger HDDs, other??).

** My guess is that if there are four 1TB drives in the array, then the max capacity would not be 4TB, it would still be 1TB since the data needs room to be moved so it may be copied. Yea/no?

** How many physical HDDs are recommended in an average music studio applied RAID system?

I'm interested in this method of storage. It seems these days CDs are considered "legacy" methods of storage and archiving. SSDs have their limits as well (lifespan issues and not much notice prior to failure). All of that said, plus the fact that HDDs are so cost effective these days (and I presume more reliable than they were years ago), the idea of a R.A.I.D. system is appealing.

My PC/Win10 system is using motherboard-mounted "M.2 NVME SSD" storage for the OS and the DAW software; a 250GB Samsung 970 EVO which is allegedly one of the top drawer M.2 SSD units. For mass storage I'm using the rather mundane Toshiba 1TB HDD (7200 rpm SATA3). 16GB (2x8GB) DDR4 3000mhz if that matters. And one of the most current "darlings" motherboard (ASUS Prime Z370-A) and an Intel i7-8700 6 core/12 thread CPU with aftermarket "heat pipes with fan" type cooler.

OH RIGHT! .. and if it matters my computer has a 500 watt PSU.

I'm just now considering breaking into the "RAID club" so I'd appreciate any useful information or hand-holding. Meanwhile I'm going to go do some more research, then catch a couple hours of shut eye (it's 3:24am).

I'll be back!!

Thankies, mangs!


cool

EDIT: Ooo! .. ooo! .. ooo! .. Mr. Kotter! .. I forgot, the motherboard also has a "Type C" port.

Ok, that's all.

Mr. Green
euxine
I am not uber-IT expert extraordinaire, but I used to build my own machines and I am also using a RAID array for my studio *qualification/apology section over*

RAID arrays are a way to get improved throughput AND/OR redundancy in a data system, yes.

Do they require a box? No. Software? Yes - some type of 'RAID controller' thing. The most convenient way to do this today is to get something called a NAS enclosure (Network Attached Storage). This has multiple bays, expects you to build a RAID array, and will configure and maintain it with minimal input. In addition the main players have extra software that is probably useful for your needs (i.e. backup, network music player, etc).

The drives being made today seem to be using SATA connections - seagate Iron Wolf, WD Red, etc etc.

Unless you have some massive data needs, or a specific application, I'd say only sweat two types of RAID array for now: RAID 0 and RAID 1.

RAID 0 (known as 'striping') gives you a performance benefit by well.. striping the data across two drives, giving you better I/O performance on read and write. There is no redundancy at all. In short, getting two HDDs to act like one. If one fails, you're dead (but that's the same as a normal HDD).

RAID 1 - fully redundant. Basically, get two HDDs of the same make/model/size, and RAID 1 will maintain two independent, complete copies of the data. No performance benefit (or hit), full redundancy. If one drive fails, just plop another one in, and you can recreate your RAID 1 again.
This is the one I use, because the drives are fast enough for storage already, and I do not write to my NAS drive when recording/producing.

All the other flavours of RAID are somewhere in between.


Main NAS enclosure brands are Synology and QNAP, and main NAS HDD manufacturers are Seagate and WD. I made a 4TB RAID 1 in a very nice enclosure for £550. but you could probably do it for less.

Hope that helps! w00t
Rex Coil 7
HELLS yea! Very much lots big helpola major size universal large galactic expand super big big big help!!!

Lots to chew on, and even some nibbles in there as well. I've not really looked over costs yet, so I may be thinking ambitiously here, but at present I am considering a four drive (four 1TB) setup. But again, I say that without checking into them and consulting with the accounting department (my wife) yet.

I'm not allowed to have money, I get a $5.00 per week allowance. lol

Wow, super helpful, thanks again!

Brian.
Rex Coil 7
One thing ... if everything is SATA3 how is the case (or drives) connected to the computer? I'm envisioning an external case/box/whatever with two or more HDDs and perhaps a power supply of sorts to power the mess (maybe not, depending on how it's connected to the computer).

I mean, if it's all connected to the computer via SATA3, and it's an external setup, how do I get SATA3 cables outside of the computer case? PCIe card or something?

~duh~ .... Drunken Homer Simpson

There are great piles of this RAID stuff on eBay.

Thanks.
tehyar
Personally I use a fast internal ssd for recording and such, and I have a NAS box for longterm mega storage. NAS = network attached storage. I dropped a few drives in it and put it in the closet. I back my whole internal drive up to it for safekeeping, and also "archive" projects when I get full (move them to the NAS).

Bonus: you get all the other benefits of a NAS, like whole house media storage. If that sounds appealing at all.

So in your case, me having your hardware, I would drop that 1tb internal drive for maybe a 500 -1000gb ssd, money permitting, and get a NAS with 3 drives in it to start. (for raid 5) But, I do get a little more than $5 a week. hihi

I've been building many various raid pcs for about 20 years now, ever since it became reasonably priced for home. Desktop, server, even laptops, using stuff from mobo chip raid up to dedicated multichannel cards. This is my first NAS, and I'm way happier not doing all the maintenance myself. Too many issues trying to make things faster with raid 0 that are not needed now with nvme drives being ridiculously fast.


edit: Helpful: http://www.raid-calculator.com/
Rex Coil 7
One other option:

What about putting the RAID drives (the whole setup, actually) inside of the computer case?

As a reminder, I am building my own computer case (out of wood) with a well thought out fan driven, filtered air flow/cooling system. The case is horizontal (lays flat) and since I am designing/building the case myself I can create any space required and any HDD drive mounting frames (or whatever) to get it all done.

So with those details aside ....

What of an internal RAID 1 setup? Perhaps a pair of 1TB HDDs, SATA3? If this is possible, what I need is the proper language to begin doing research on various components. Y'know, names of whatever it takes to install an internal RAID 1 setup using two (or more) 1TB 3.5" HDDs.

Thanks bunches!

pbear :( cookie?!?
Rex Coil 7
tehyar wrote:
Personally I use a fast internal ssd for recording and such, and I have a NAS box for longterm mega storage. NAS = network attached storage. I dropped a few drives in it and put it in the closet. I back my whole internal drive up to it for safekeeping, and also "archive" projects when I get full (move them to the NAS).

Bonus: you get all the other benefits of a NAS, like whole house media storage. If that sounds appealing at all.

So in your case, me having your hardware, I would drop that 1tb internal drive for maybe a 500 -1000gb ssd, money permitting, and get a NAS with 3 drives in it to start. (for raid 5) But, I do get a little more than $5 a week. hihi

I've been building many various raid pcs for about 20 years now, ever since it became reasonably priced for home. Desktop, server, even laptops, using stuff from mobo chip raid up to dedicated multichannel cards. This is my first NAS, and I'm way happier not doing all the maintenance myself. Too many issues trying to make things faster with raid 0 that are not needed now with nvme drives being ridiculously fast.


edit: Helpful: http://www.raid-calculator.com/
Ok, so this is great stuffs. But one thing ....

tehyar wrote:
...I would drop that 1tb internal drive for maybe a 500 -1000gb ssd, money permitting...
I already have a 250GB M.2 NVME SSD (it mounts directly on the motherboard using an M.2 socket) ... it's a Samsung 970 EVO SSD with absolutely SICK ASS speeds (like 20x that of a SATA3 SSD).

I also already have a 1TB Toshiba SATA3 HDD (7200 rpm).

So why couldn't I just add a second one or two of the Toshiba 1TB HDDs and use them for the RAID rig ... and use the M.2 250GB NVME SSD as the OS/DAW/Boot drive?

That way I'm not selling stuff and I'd also be nearly there, all I'd need would be two more Toshiba 1TB HDDs and whatever it takes to complete the NAS/RAID rig?

Plus also what I said in the post I was drafting when you posted your comment ... howzabowt mounting the entire RAID rig inside of the computer case (read my post that is just after yours about the case).

hmmm.....

EDIT: Ok, I just re-read your post .... so lots to consider, I like your plan as much as my own.

....... hmmm..... hmmm..... hmmm.....
rowsbywoof
I run an old ReadyNAS and I'm just about to redo my RAID setup. I was using a four drive RAID 5 setup, but will be replacing the drives as they're now five years old and they're coming toward the end of life for drives that are on 24/7 and expected to be reliable. Right now I have four 2TB enterprise level drives for around 6TB of space.

I've since grown weary of RAID 5, and it's a little less safe if something happens than going RAID 1 and having true one-to-one mirroring of data. So, the rebuild is going to be four 4TB or 6TB enterprise drives which will be cheaper than the original 2TB I bought five years ago by a LOT, and that will give me 8TB to 12TB of solid RAID space with true one to one mirroring.

A ReadyNAS or similar is really worth the expense to me. It's a single small box that has been reliable and bulletproof for five years and I've thought about upgrading that as well, but I'm probably going to open this little guy up and check the PSU to see if it needs recapping and basically leave it alone since... Well... It's working great and I have no complaints.

My advice. If you're going to do this, do it right. Buy server class enterprise drives that are meant for constant 24/7 use. It's worth the expense. The whole point of this is to have a reliable space to back things up. Don't try to shave a few bucks off here or there and then regret it when you have a data failure. Sure, off the shelf consumer drives might be fine, but from someone who's worked in data management for nearly two decades the amount of enterprise class drives I've seen go down I can count on one hand.... Consumer class drives, I've long since lost count.

If you're going to do it, do it right.
euxine
Rex Coil 7 wrote:
One other option:

What about putting the RAID drives (the whole setup, actually) inside of the computer case?


To try to cover a lot of your questions at once - being that one of the main tenets of data storage integrity is to have the data in different physical locations (and by inference, on different devices) I think the NAS approach is the best.

Even without being hollywood studio person, you can easily get data in three locations:

- Cloud Storage
- Local Machine
- NAS

If you learn about RAID types 2 -> N-whatever, please tell me about it too.

NAS devices have slots that the SATA drive just drop into. It's slick.
rean1mator
What type of data are you planning to store on your raid setup?
roger
euxine wrote:
RAID 1 - fully redundant. Basically, get two HDDs of the same make/model/size, and RAID 1 will maintain two independent, complete copies of the data.


Actually, don't. If you get two drives of the same make/model/size at the same time, chances are they are produced around the same time. If one has a fault, it is likely the second one has that fault aswell and is just and accident waiting to happen. Or might aswell happen when you use if further(i.e., copying your backup).
RickKleffel
...Or, buy 2 5 TB drives at Costco, and a drive duplication program (in the Mac world, Super Duper), save everything to the first, dupe 1->2 as often as you can, every night can be easily scheduled and spend less time worrying and more time creating. In the Mac world, set up Time Machine to catch in-the -day fluffs. Be thankful if all this time and money is “wasted.”
dj2sday
if you go internal buy a raid controller. I don't recommend internal raids otherwise as it requires the MB to work harder which shortens its lifespan. If you MB craps out you are left with multiple drives that need costly help to reconfigure.

raid 5 has both the speed and redundancy raid is known for. a nice 4 disk raid 5 thunderbolt 3 pegasus3 r4drive array is a great option for media professionals.

https://www.bhphotovideo.com/c/product/1293606-REG/promise_technology_ p3r4hd12us_12tb_pegasus3_r4_thunderbolt.html/?ap=y&gclid=Cj0KCQiAhKviB RCNARIsAAGZ7Cd_4NtYuKvk5gvDwP9lq7em2HoaZloxV7b-l4Y8lU8F3ANTvbrZZI8aAgm qEALw_wcB&lsft=BI%3A514&smp=Y
ersatzplanet
Make your life easier and get a Drobo Drive array. They have lots of different options, 5 and 8-drive options, 2.5" and 3.5" drive types. Network or direct attached. Lots of connection types. They don't need the same size drives in them. They don't need all the bays filled. One drive dies, plop in another and the system rebuilds the drive.

Drobo uses their own system they call "Beyond Raid" (I know, marketing speak) and it does have some advantages over regular raid. Here is their page describing the differences - http://www.droboworks.com/Drobo-5C.asp

The main plus is that the drive don't have to be the same size. You can buy an empty box from them and put in all the dives you have laying around. It will work. Once you can afford a larger new drive, pull the smallest out of the box and put in the new one. With 1TB drives easily under $100 now you can have a decent array pretty quickly.
corpusjonsey
I've been running a 2012 MacBook pro with the disc drive removed and replaced with another SSD bay. I put 2 identical 256gb Samsung 840 pro drives in and set them up to run RAID 0. My machine boots in less than 9 sec. I have an external drive that I use for backup. No problems so far (~4yrs).
sparood
I agree with euxine, you should have your storage stored at multiple locations. Yes, RAID is a nice option, but when your house burns down you have nothing, zilch.

Actually, in IT there is the 3-2-1 backup rule. Have at least 3 copies of your data (local + 2 backup), on at least 2 different backup media, and have 1 offsite backup. It isn't a golden rule, but it's usually a good consideration between safety and costs.

Also, RAID is no backup! Please don't use RAID on your drives and think you are done with it.
strangegravity
Rex Coil 7 why do you want RAID. Exceeding the throughput on SATA3, recording audio, would be difficult.

I have a removable HDD bay on my system. I pop in a drive, back up and pop it back out.

I used to keep an offsite copy as well.
rowsbywoof
sparood wrote:
I agree with euxine, you should have your storage stored at multiple locations. Yes, RAID is a nice option, but when your house burns down you have nothing, zilch.

Actually, in IT there is the 3-2-1 backup rule. Have at least 3 copies of your data (local + 2 backup), on at least 2 different backup media, and have 1 offsite backup. It isn't a golden rule, but it's usually a good consideration between safety and costs.

Also, RAID is no backup! Please don't use RAID on your drives and think you are done with it.


Good advice. I have my NAS as the main central data server for the house. Even with the RAID setup, I’m always considering a failure, so I back up my NAS to removable drives monthly, and one is always in my always with me backpack, or my oh-shit-grab-the-dog-and-backpack backpack, if the place is on fire.

Haha. My first job I was given the sage advice, “There are only two types of hard drives: ones that have failed, and ones that will.” That’s stuck with me ever since. Last thing I need is a damaged RAID controller or dead hard drive interface rendering my perfectly good drives unreadable. cry

Best to back up your backup.
sduck
I'll parrot ersatzplanet and suggest a drobo of some sort. Except for the no moneys part - they cost quite a bit. They're extremely easy to use, and extremely reliable, but you pay for that. I have a drobo mini, a discontinued product, but it's soldiering on reliably.
powertran
Raid arrays provide redundancy for the disks, not the data. Use robocopy on Windows or Rsync on Mac/Linux to dump your wav files to a standard USB disk. Or say goodbye to your wav files while you spend months trying to source the failed raid array disk-controller card/un-proprietary system motherboard.
JohnLRice
Rex Coil 7 wrote:
What about putting the RAID drives (the whole setup, actually) inside of the computer case?
Sure, you can just 'permanently mount the drives inside but I'd recommend using a removable drive bay instead. You can get just a 2 or 3 drive bay for a small RAID array, or get something like a 4 or 5 drive bay so you can make a bigger RAID and or do a small RAID plus your recording disk and maybe system drive. Like this one or similar . . .shop around for best price and what ever brand and model is currently the 'best':
https://www.amazon.com/Supermicro-CSE-M35TQB-Swap-Mobile-Black/dp/B000 HWJGEW



I haven't used any RAID setups for years at home and go with JBOD (Just a Bunch Of Disks).
DSC
John's on the right track with the JBOD setup, but I run those in RAID setups too! Dealing with fiber cables can be fun and setting up more complex 'fabrics' can be challenging but having every workstation's data connected and backed up is priceless! I can lose four hard drives in each of those racks and still be redundant.



Rex Coil 7
JohnLRice wrote:
Rex Coil 7 wrote:
What about putting the RAID drives (the whole setup, actually) inside of the computer case?
Sure, you can just 'permanently mount the drives inside but I'd recommend using a removable drive bay instead. You can get just a 2 or 3 drive bay for a small RAID array, or get something like a 4 or 5 drive bay so you can make a bigger RAID and or do a small RAID plus your recording disk and maybe system drive. Like this one or similar . . .shop around for best price and what ever brand and model is currently the 'best':
https://www.amazon.com/Supermicro-CSE-M35TQB-Swap-Mobile-Black/dp/B000 HWJGEW



I haven't used any RAID setups for years at home and go with JBOD (Just a Bunch Of Disks).
I had bombed this thread with so many questions I elected to (in a very rare instance) embrace brevity when I asked about "going internal". What I meant was pretty much precisely what you've depicted and explained here. A "chassis" mounted in the main computer case however still providing full external access to the drives themselves. I was thinking of cooling issues when that internal thing came up. Since the computer case will be outfitted a somewhat pressurized cooling system using filtered intake air, and the case will require exhaust ports, I thought I could put the RAID array inside of the case, with the drives accessible from outside. The pressurized/filtered cooling air would then exhaust past/through the the RAID drive array. And since the air would be blowing OUT of the case through the RAID drive, the outward force of air would prevent any lint/dust/cat hair from getting in/around to the RAID drives as well as the computer case.

After all, I live here .....



I was looking at portable/external RAID arrays and saw that some had fan cooling systems. All I could think of was how much lint/cat hair would accumulate between the drives and in the fan Since my computer case will be using filtered cooling air, and the case will require exhaust ports anyway, the thought was to use some of that exhaust air to cool the RAID array as it exits the case.

There will be other exhaust ports, and the main power supply also has it's own cooling fan that inhales cool air and the heated air is forced out of the back vents. So between the power supply's intake/exhaust system and air exhausting through the RAID array, it just seemed like an efficient way to cool the RAID array with filtered air. It would also serve to provide one of the exhaust vents for the computer case. Beyond that, there may be one or two round 2" exhaust ports covered with green 3M pads in the inside (to prevent any lint/cat hair from finding it's way into the case when the computer is not powered up).

The intake filter is fourteen inches in diameter and four inches tall, and the intake fans are a pair of 120mm chassis fans mounted directly beneath the intake filter. This method effectively "pressurizes" the computer case, which would force any air out of small slits, joints, holes, ports, etc. A pressurized vessel keeps lint/dust/cat hair from entering the vessel (aka "computer case"). All that is required beyond that is some exhaust ports. The power supply's exhaust vent, one or two (maybe three) 2 inch round exhaust holes, and theoretically the spaces between the RAID array drives would provide that needed exhaust venting.

So no dust/lint gets in, everything is cooled by directed air flow, happy happy joy joy.

So that is what pretty much motivated the notion of placing the RAID array "inside" of the computer case. Again, the image in my head depicts full external access the all of the RAID drives ... exhaust air just sortof leaks out around the gaps between and around the hard drives (really no different than an "internally mounted" CD drive ... the front of the drive is accessible to the end/user, but the drive is mechanically mounted inside of the case.

Air pressure from the case is vented, the drives get cooled, crap has no access to the inside of the case or anything mounted within it.

Simple dimple. Well, at least to me.

I'm not defending the idea in this post, I'm just providing a more detailed and longer winded description. Brevity and I don't mix well, so it seems.





Now then .... I'll need to tackle the other posts/suggestions in another reply. What strikes me most about all of the replies is how different everyone sees this idea of RAID systems. What that tells me is that there are fifty ways to do it right. There's very little consensus between all of the replies, that implies there are many ways to get it done.

I am probably more security minded than most people due to my background, and I probably go through more "crisis rehearsal" ideas than most people, also due to my background, training, and experience. I'm ALWAYS thinking about how to outsmart the next disaster. In 2001 the 7,000 sq/ft shop my business was located within was nearly destroyed by the largest fire in my city's entire history. A quarter million wooden pallets were set on fire, which were stacked outside directly against the back wall of my shop (a pallet shop was located in the same industrial park that my business was in, imagine two football fields of wooden pallets stacked three stories high). You couple that nightmare with the decades of security training and crisis response training and experience behind me as a six year veteran as a security specialist in the US military as well as a seeing the "hells" during my time as a first responder after the military and it creates an "always ON attitude" of always (ALWAYS) thinking of and preparing for the next tragedy.

That said, I'm always thinking about methods to save my data in the event of a house fire or having our roof blown off during one the raging furious Monsoon storms this area has a well earned reputation for Crisis response and loss prevention is something that is CEASELESSLY on my waking mind.

If I am awake, I am thinking about how to prevent, defeat, respond to, and deal with the next big bad thing. BOOM!

It's part of the basket of gifts Complex PTSD gives you each and every day. I try like hell to take advantage of it as best I can, and use the anxiety as a tool.

So yea, scrambling to get my data out of a burning house is one of the "somethings" that motivate my design ideas.

Woa ... this shit got dark.

Movin on .....

I'll work on replies to all of the other suggestions. THANK YOU ALL SO VERY MUCH!! I know your ideas come from only the best places in your hearts.

Mo' later ...

Dancing Star
JohnLRice
DSC wrote:

we're not worthy thumbs up
Rex Coil 7
JohnLRice wrote:
I haven't used any RAID setups for years at home and go with JBOD (Just a Bunch Of Disks).


DSC wrote:
John's on the right track with the JBOD setup...
So I'm right back to using a CD/DVD drive then?

Well, ok. It would certainly save money, the CD/DVD drive I was eyeballing is less than $30.00 bucks, and I already probably have 200 blank CDs on hand.

I guess simpler is most always best.

Alright then, I trust pretty much everyone I engage with regularly here .... and if there is anything that has been the most common reply it's been "RAID = No".

Thanks for the help. Everyone has done precisely as I'd hoped which was to assist me in making a choice regarding something I know very little about.

Checkered Flag Checkered Flag Checkered Flag Checkered Flag Checkered Flag
MUFF WIGGLER Forum Index -> General Gear Goto page 1, 2  Next [all]
Page 1 of 2
Powered by phpBB © phpBB Group