Restore RAID 5 array

If your topic has nothing to do with Spyderco, you can post it here.
User avatar
jabba359
Member
Posts: 4958
Joined: Fri Feb 10, 2006 11:07 pm
Location: North Hollywood, CA U.S.A. Earth
Contact:

Restore RAID 5 array

#1

Post by jabba359 »

And here I was, just thinking about picking up an external drive to backup all my data to last week...

I've never had a problem with my RAID setup before, but yesterday I was using my computer, took a break, and when I came back, couldn't access my D:/ drive (my RAID drive). Looking at my BIOS, it shows that two of my four drives had dropped out of the array and were showing up as non-RAID disks. I can't re-add them my RAID through the BIOS, as it says it will erase all existing data. While I haven't installed anything the last few days, my computer did tell me to restart to finish Windows updates. Not sure if that would be the culprit.

Setup:
4x 1TB Samsung F3 hard drives in RAID 5
1x 80GB Intel Solid State Boot/Application Drive
Running on Asus P6x58D-E motherboard from the integrated Intel ICH10R controller

Anybody here have any ideas on how to resolve this issue?
-Kyle

:bug-red
Latest arrivals: Lava Flow CF DLC Para2, Magnacut Mule, GITD Jester

http://www.spydiewiki.com
User avatar
c.joe
Member
Posts: 698
Joined: Tue Mar 23, 2010 10:32 am

#2

Post by c.joe »

Have you tried a simple system restore? Sometimes that can do wonders.
User avatar
jabba359
Member
Posts: 4958
Joined: Fri Feb 10, 2006 11:07 pm
Location: North Hollywood, CA U.S.A. Earth
Contact:

#3

Post by jabba359 »

c.joe wrote:Have you tried a simple system restore? Sometimes that can do wonders.
I tried it already and it didn't work. Thanks though.
-Kyle

:bug-red
Latest arrivals: Lava Flow CF DLC Para2, Magnacut Mule, GITD Jester

http://www.spydiewiki.com
User avatar
cgjones
Member
Posts: 476
Joined: Fri Sep 24, 2004 10:33 am
Location: Upstate New York

#4

Post by cgjones »

I've never worked with a RAID 5 array, but I did a fair amount of research for a mail server I setup. If I remember correctly, RAID 5 can only tolerate losing 1 drive in the array.
User avatar
Donut
Member
Posts: 9568
Joined: Sat Sep 05, 2009 5:47 pm
Location: Virginia Beach, VA, USA

#5

Post by Donut »

cgjones wrote:I've never worked with a RAID 5 array, but I did a fair amount of research for a mail server I setup. If I remember correctly, RAID 5 can only tolerate losing 1 drive in the array.
This is the bad news right here. You say the raid controller is still seeing the drives, so maybe two of the drives aren't bad.

You can go into the add/remove programs and see the updates that were installed. The versions of windows are each a little different, but usually you have to click a checkbox to include windows updates, then you have to pick from a list that you want it to show you when it was installed, then you can choose to uninstall the last few updates.

If that doesn't work, and I am guessing it probably won't work, then I don't think you have much hope. :(
-Brian
A distinguished lurker.
Waiting on a Squeak and Pingo with a Split Spring!
User avatar
jackknifeh
Member
Posts: 8412
Joined: Fri Jul 09, 2010 6:01 am
Location: Florida panhandle

#6

Post by jackknifeh »

cgjones wrote:I've never worked with a RAID 5 array, but I did a fair amount of research for a mail server I setup. If I remember correctly, RAID 5 can only tolerate losing 1 drive in the array.
One drive fails, no problem. Replace it and it will rebuild.

More than one drive fails, problem. Everything is gone. Replace bad drives, rebuild drive array, reinstall windows (or other OS), restore all backed up files.

I'm pretty sure that's correct. There may be other options since technology changes.

Jack

PS
I just read the original post again. Is the OS on a non-raid drive? If so that's good. I know you have all your data files backed up. ???
User avatar
jabba359
Member
Posts: 4958
Joined: Fri Feb 10, 2006 11:07 pm
Location: North Hollywood, CA U.S.A. Earth
Contact:

#7

Post by jabba359 »

I make a weekly backup of my boot/OS/application drive, so when the array failed I reverted back to an image of the previous week's drive. That didn't fix anything. If I understand how a RAID array functions, however, when drives are dropped out of the array, the computer marks them so that they don't automatically join back in the array. The only way to get them join in is to reformat the array and add them in (losing all data).

I've gotten some demo software that creates a virtual software RAID, and it looks like most of my information is intact. They want $180 for it, so I was looking for other possible solutions.

I had thought a RAID 5 would keep things like this from happening, as I've never had a hard drive fail and the chances of two drives going bad at the exact same time are astronomically slim. Guess I didn't account for software errors taking the drives offline. Lesson learned. Even if I have a redundant system, back it up anyway!
-Kyle

:bug-red
Latest arrivals: Lava Flow CF DLC Para2, Magnacut Mule, GITD Jester

http://www.spydiewiki.com
User avatar
jackknifeh
Member
Posts: 8412
Joined: Fri Jul 09, 2010 6:01 am
Location: Florida panhandle

#8

Post by jackknifeh »

jabba359 wrote:I make a weekly backup of my boot/OS/application drive, so when the array failed I reverted back to an image of the previous week's drive. That didn't fix anything. If I understand how a RAID array functions, however, when drives are dropped out of the array, the computer marks them so that they don't automatically join back in the array. The only way to get them join in is to reformat the array and add them in (losing all data).

FROM JACK
NOT TRUE!!! Do not re-insert a bad drive back into the array. That will destroy data. A bad drive needs to be replaced. If only one drive fails the drive array will still work. On your server, are the drives hot swap (can you install one with the PC still running)? If they are there may be LEDs on the face of the drive indicating it's condition with different colors. Replace the bad drive with a new on and it will rebuild back into the array.


I've gotten some demo software that creates a virtual software RAID, and it looks like most of my information is intact. They want $180 for it, so I was looking for other possible solutions.

FROM JACK Never heard of this but that doesn't mean it won't work. I've just never heard of it.


I had thought a RAID 5 would keep things like this from happening, as I've never had a hard drive fail and the chances of two drives going bad at the exact same time are astronomically slim. Guess I didn't account for software errors taking the drives offline. Lesson learned. Even if I have a redundant system, back it up anyway!
From Jack, Is you array set up in the BIOS or the OS? I used the BIOS level since it is created prior to the OS.

RAID 5
You need at least 3 hard drives the same size. RAID 2 is used with 2 drives and it's called a mirror as the second drive is exactly like the first.

You have 4. That means 1 of the 4 drives is not seen as usable drive space. If each drive equals 100 gb that equals 400 gb of space. However, you only have 300 gb of usable space. The other 100 gb is used as an online backup for lack of a better phrase. The data is scattered throughout the drives. That way if one drive fails you can replace it with a new one and it will be built back into the array with no data loss. As you said though two drives don't fail at the same time in a perfect world. Your data is gone gone gone. That's why it's still important to backup whatever is on the array drives to another place before this happens. Re-writable DVDs are an easy option. They are cheap and you can use them over and over. If your OS was on the other drive you should still be able to boot up, just not see whatever you had on the drive array (I think you gave it the drive letter D :) .

According to me all data on the drive array is gone. There may be options I'm not aware of. It's been over 5 years since I've worked on this kind of stuff.

Hope this is informative even if you can't get your data back. It may help next time. Always have backups of anything you can't afford to loose and always test restoring files that have been backed up. I've seen companies loose thousands worth of data and customer info. Think of a bank that lost your mortgage with no backup. No more house payments for you. Yeah!

I had a customer once that had a project on a drive valued at a whole lot. The controller went bad. That's the circuit board mounted on the drive. They sent the drive to a company that takes the drive apart and tries to get the data off the disks themselves. I heard the bill for that was over $4000.00.

Good luck,
Jack
User avatar
TooSharp
Member
Posts: 140
Joined: Tue May 26, 2009 3:04 pm
Location: Fayetteville, AR

#9

Post by TooSharp »

Same as what everyone else says here. If two go down, your hosed, unless you had another drive acting as a hot swap, which you probably don't or your RAID would still be working.

Data loss sucks bad, sorry to hear your having problems.
User avatar
jackknifeh
Member
Posts: 8412
Joined: Fri Jul 09, 2010 6:01 am
Location: Florida panhandle

#10

Post by jackknifeh »

TooSharp wrote:Same as what everyone else says here. If two go down, your hosed, unless you had another drive acting as a hot swap, which you probably don't or your RAID would still be working.

Data loss sucks bad, sorry to hear your having problems.
Hot spare would have saved you but you would have had to sell a lot of knives or never buy another one. :eek:

Just curious. What do you use that PC for? Is it a server? Sounds like a lot of storage and money for a workstation (depending on drive size I guess).

Jack
User avatar
jabba359
Member
Posts: 4958
Joined: Fri Feb 10, 2006 11:07 pm
Location: North Hollywood, CA U.S.A. Earth
Contact:

#11

Post by jabba359 »

The drives didn't fail as far as I can tell, as I can set up a software RAID (using the free demos of RAID recovery software I mentioned) and my files are still intact. I can preview the files, but can't extract them unless I pony up the $180-700 for the full version of the software, which I may end up having to do if there aren't any other solutions. Just thought I'd see if anyone smarter than me in this area had any suggestions.

I have a spare on hand in case one of the drives did go down, but that didn't help any since two went off line at the same time. Since the two drives went offline within two hours of each other (they were working when I walked away, but weren't when I came back 2 hours later), I'm guessing it was a software glitch. The likelihood that two failed mechanically within that short of a time is small. Most of my really important stuff is backed up on various other drives, it will just be very time consuming to put it all back together.

My PC is used for video editing, mainly. That's why I have the 3TB of hard drive space. Fortunately, I had delivered all the jobs I was working on already, so it's just some personal projects that are currently in RAID limbo. Looking at craigslist, I may enlist the help of a pro, as there are quite a few out there that will recover my data for less than the mid-high priced software.
-Kyle

:bug-red
Latest arrivals: Lava Flow CF DLC Para2, Magnacut Mule, GITD Jester

http://www.spydiewiki.com
User avatar
cgjones
Member
Posts: 476
Joined: Fri Sep 24, 2004 10:33 am
Location: Upstate New York

#12

Post by cgjones »

jabba359 wrote:The drives didn't fail as far as I can tell, as I can set up a software RAID (using the free demos of RAID recovery software I mentioned) and my files are still intact. I can preview the files, but can't extract them unless I pony up the $180-700 for the full version of the software, which I may end up having to do if there aren't any other solutions. Just thought I'd see if anyone smarter than me in this area had any suggestions.
You might want to try running mdadm from a Linux live CD (Knoppix, Ubuntu, etc). This might accomplish the same thing as the trial software, without the cost.

On a side note, while doing some research, I came across this article. It sounds much like the issue you are having.
User avatar
jabba359
Member
Posts: 4958
Joined: Fri Feb 10, 2006 11:07 pm
Location: North Hollywood, CA U.S.A. Earth
Contact:

#13

Post by jabba359 »

cgjones wrote:You might want to try running mdadm from a Linux live CD (Knoppix, Ubuntu, etc). This might accomplish the same thing as the trial software, without the cost.

On a side note, while doing some research, I came across this article. It sounds much like the issue you are having.
Ah, thanks for the link! My BIOS got reset the day before, giving me the same error as them. Strange that everything ran fine for another day before I got the error. R-Studio is one of the programs I'm trying out.
-Kyle

:bug-red
Latest arrivals: Lava Flow CF DLC Para2, Magnacut Mule, GITD Jester

http://www.spydiewiki.com
User avatar
monsterdog
Member
Posts: 327
Joined: Sun Apr 19, 2009 11:42 am

#14

Post by monsterdog »

Another vote to see if a Linux live cd can save you. Stitching the disks together can be tricky, but might work.

Have you considered using a mirror or a separate NAS RAID server running something like ZFS? RAID-5 is notoriously crappy and definitely not a substitute for backups.

This sounds close to your problem, and was recovered using a Linux live cd (from 2005, there might be nice tools available for this now):

http://www.freesoftwaremagazine.com/art ... overy_raid
User avatar
jackknifeh
Member
Posts: 8412
Joined: Fri Jul 09, 2010 6:01 am
Location: Florida panhandle

#15

Post by jackknifeh »

Here is something you might want to try if you haven't already.

If all drives are still connected to the computer and you haven't removed the bad ones (do you know which ones are not working?) you can shut down the computer and unplug the power cable. Wait about 5 minutes to make sure any components are completely shut down. Some NICs stay powered up for remote power up, things like that. Then unplug the data cables and power cables from all 4 hard drives and reseat them. I would unplug one and plug it back up before unplugging anything else to prevent connecting a drive back in the wrong place in the cable. What I'm suggesting is just a cable reseat. That has fixed many computers I've worked on when something all of a sudden wasn't seen anymore by the system. Be sure to reseat the connector on your array controller also. Since two drives are problematic it may be something in common with all drives which would be the controller. Try reseating the controller in the system board too while you are at it I guess. Non of this "should" cause any additional problems. Be sure to discharge any static in your hands before touching anything. Do this by touching the frame of the PC before unplugging it from the power cable. Any static in your body will be discharged via the ground wire in the AC wiring in your home.

IF HOWEVER, you have already disconnected a problem drive from your computer and powered it back on DO NOT do any of what I've suggested here. Plugging a bad drive back into an array aftere it has been removed and the PC powered back on will blow the array away.

You mentioned getting a pro to help you. Keep in mind computer technicians and engineers (which is really what you want) come in all different skill levels just like auto mechanics, doctors, etc. I worked on computers for about 10 years and I was pretty good but there were people who could make me look incompetent in some situations like when there is a symptom that isn't the normal problem. I don't think any company will guarantee getting your data back even if it is possible right now and doing some things will destroy what you still may be able to salvage. Getting the array working again isn't a problem even if you have to buy new drives. The data is the tricky part.

It sounds like you are getting quite a bit of help from several people on this forum. That is really a great thing. If you do get a "pro" the company I worked for would probably charge around $125.00 per hour for what you need so any help you can get to do it yourself is good, plus any additional knowledge or skill to be gained.

Again, good luck,
Jack
User avatar
jabba359
Member
Posts: 4958
Joined: Fri Feb 10, 2006 11:07 pm
Location: North Hollywood, CA U.S.A. Earth
Contact:

#16

Post by jabba359 »

Thanks for all the suggestions guys! I ended up paying the $80 for a program called ReclaiMe that recovered all of my files (only took a few hours) and dumped to an external hard drive (only another 14 hours). Currently reinitializing my RAID array (it takes quite a while to initialize a 3TB array) and then will dump everything from the external hard drive back to the array (probably another 14 hour process...). Then off to purchase a 2 or 3TB USB 3 external drive for making consistent backups. USB 2 is just way too slow for all these huge video files.

Backup! Backup! Backup! Lesson learned. That's $80 that could have gone toward another :spyder: .
-Kyle

:bug-red
Latest arrivals: Lava Flow CF DLC Para2, Magnacut Mule, GITD Jester

http://www.spydiewiki.com
User avatar
jackknifeh
Member
Posts: 8412
Joined: Fri Jul 09, 2010 6:01 am
Location: Florida panhandle

#17

Post by jackknifeh »

jabba359 wrote:Thanks for all the suggestions guys! I ended up paying the $80 for a program called ReclaiMe that recovered all of my files (only took a few hours) and dumped to an external hard drive (only another 14 hours). Currently reinitializing my RAID array (it takes quite a while to initialize a 3TB array) and then will dump everything from the external hard drive back to the array (probably another 14 hour process...). Then off to purchase a 2 or 3TB USB 3 external drive for making consistent backups. USB 2 is just way too slow for all these huge video files.

Backup! Backup! Backup! Lesson learned. That's $80 that could have gone toward another :spyder: .
Glad you got your data back because that's the main thing. Spending money on redundancy (your RAID setup and backups) is like spending money on auto insurance. You hope you never need it but are glad when it's there if you do. If you mess with computers a lot it's not a question of if you will need your backups but when will you need them.

Jack
User avatar
On Edge
Member
Posts: 436
Joined: Fri Aug 13, 2010 8:01 am
Location: Virginia, USA, Earth

#18

Post by On Edge »

jabba359 wrote:" …
Backup! Backup! Backup! Lesson learned. That's $80 that could have gone toward another :spyder: ."
Sorry about your woes, but the price of that lesson could've been a lot steeper.

Glad it worked out.

Hey, didn't you save like at least $100 over that other solution … ? :rolleyes: :spyder: :rolleyes:
"In theory, there is no difference between theory and practice ... In practice, there is."
Post Reply