The outage today was caused by a drive failure which managed to completely pollute my main SAN array which is where the data for the HPC:Factor server lives.
There was more than enough RAID parity for everything to be OK, but whaterver went wrong with the drive decimated the I/O on the SAN channel group for *mysterious* reasons. The fault seems to have occurred ~6:22am and I was aware of it by 7:05am and started scrambling to rescue data. The RAID controller firmware only noticed at 10:45am, setting off an alarm which made me jump out of my skin; finally identifying the drive through its forced eviction from the group. Once evicted the drive the array correctly went to over to parity mode and everything settled down.
The backups were all in place and are OK from last night, but through some luck, nothing has needed to be restored.
The HBA is damaged and I'm definately lost 1 drive. The drive I have RMA'd, the controller is out of warranty. Ouch.
It is 5pm now... that was a waste of a day.
Fngers crossed I can keep everything up until parts arrive.
And how many others reading this post are clueless to what Chris just entered except that to know something broke and he fixed it...if you are...then join the club with me and there are now two of us...
Yes, I noticed the outage. I am glad it has been rescued. Thank you for the hard work. I hate when server related issue happens. I once had to re-flash BIOS for some reason some time ago.
Thank you for the hard work. I hate when server related issue happens. I once had to re-flash BIOS for some reason some time ago.
