On a scale only
(just
) surpassed by the great Wall St. crash of 1929. You will likely be aware by now that HPC:Factor has been off-line since the 10th February.
So I'm sure you are all wondering exactly what happened
As you will mostly be aware, the site is run on a trans-Atlantic, with our main operations being in Florida. The servers you are talking to right now are all based in Florida, however a lot of information and all of the development is located in the UK.
On Friday 9th my data storage server failed
(no data loss
)
This was followed the next day by the failure which you are all aware of. In this case the drives on HPC:Factors main HTTPd server failed
(catastrophic data loss
).
Finally the drive array on my test server got dropped in software at over the weekend
(eventual data loss
).
Factoring in the hardware losses, and the fact that the local and off-site database backup stores were lost in the space of a couple of days. The site has been off-line for the period. My data server was required to rebuild the other servers, my test server had the main repository of off-site backups on it and the HTTPd server in reality hasn't been fixed.
There was some relief in that all the content backups were safe, however it has been the database backups that have been the main victims in this affair. The last salvageable Forums backup being from Christmas Eve. Luckily the Central database, because of its fairly recent implementation was fairly up to date on my development system, so data loss there has been limited to a couple of days. The Forum and News/RSS databases are the worst hit, with no recoverable & recent backups being available/safe.
What isn't working?
This is a known list of everything that is known to have been lost from the public site at the time of writing.
- Home Page Poll >= 14th January 2007
- Central (HCL/Community/QLink etc) => a.m. 10th February 2007
- Main News/RSS = 25th December (all else recovered)
- Forum Posts/Threads >= p.m. 24th December 2006
- Forums Private Messages (PM) >= p.m. 24th December 2006
- Forum User Registrations >= p.m. 24th December 2006
- Forum Attachments = ALL
- Forum Avatars (stored locally) = ALL
- Forum Personal Photo's (stored locally) = ALL
If you have cached information covering the missing 25th December posts, I would love to hear from you so we can get them back in. These could be in your browser cache, or in your RSS Aggregator if you have been subscribed to our feeds for more than 2 months. If you do find out that you have something and would like to send it along. I will keep the emergency contact mail address open for a little while longer: fema@hpcfactor.com
Yes; a little humour for our American Friends.
I and the other owners would like to extend specific personal thanks to
Gary Emmert and
Silvio Vernillo, without whom the reconstruction of the Main News and RSS feed databases would not have been anywhere near as successful as it has been with their gracious assistance.
I would also like to extend my thanks and surprise towards the warm hearted and passionate emails I have received from community members offering support, concern, motivation and
kind words.
There are clearly some lessons to be learnt from this ordeal - perhaps a cynic would say I should never go away for a couple of days; and my bank manager might tell me I can't afford to take days off of work to fix servers. The sad truth is that what has happened is primarily the result of a series of unrelated, unfortunate errors which have nothing in common other than bad luck.
The surprising truth is what has turned out to be the size and scale of this website. Perhaps to my eternal detriment, I originally quoted an estimate of 6 hours of upload time to have enough information available to start the site up again. It actually took 45 hours to get there. I'm editing this post right now and the clock behind me is chiming the bells for 1am. A staggering 53 hours later, files are still being restored from my servers to the Florida ones - and I suspect there are a couple more hours yet in line!
I'm not entirely sure in my own mind of everything that has been lost, and also how that is going to impact you guys or the site as a whole. It has been a painful experience and the decisions to keep the site down were not easy; though necessary. Unfortunately the impact of this will be rippling out in gentle waves for some time yet.
Search engines have already started de-listing the site, various people have got themselves into various 'states', the data we have lost contains the thoughts and feeling of you, the H/PC Community and they can never be recovered; at least not in the same spirit. Though I'm confident we'll all be having lengthy and stimulating conversations for a very long time to come.
Technically speaking, the site will have to be placed into a state of flux as its emergency configurations at the moment have to be spanned out into stable production systems. We need new hardware in Florida, it takes time and money and the ability of the current system to cope with site load is a matter of some serious concern.
I would like to end on a personal note, a gripe if you will permit me.
I've served this community for some 7 years, Clinton and John around the same if not a little longer, and here I address a very small minority of those who so
contemptibly vocalised their opinions to me. I have been appalled by the reactions of these certain insignificant
minority of people throughout this last week.
That we would just flick a switch, and, without so much as a word walk away from all that has been achieved.
To those who sent those venomous emails and thought that after spending so much time and effort in keeping this site alive for so many years, that we would treat the members of this community so disdainfully. I suggest to them that they don't know us at all. I suggest to them that it is they who must learn some tact and decency. And I suggest to them; that they stay out of my way.
I am pleased to say, however, that we are indebted to all of those people - some I know well and others new faces - who did write to Clinton, to John and myself with words of support. They certainly made this headache a lot easier to deal with.
Now, on with the show! Lets get that post count back to 72,000 posts; post haste!
Chris
I may even blame it on HPC:Factor…
