The Great Server Crash of Summer '09

From RationalWikiWiki

The Great Server Crash of Summer '09 was the the worst server crash in RationalWiki 2.0's history. It combined the "perfect storm" of what should have been a minor hardware or software glitch with Trent's absence from the dedicated server for a visit with family and friends.

Contents 1 Signs and portents 2 The Crash 3 Where they went 4 Contact points which could have been used but weren't really 5 Why Teflpedia? 6 Speculation 6.1 Official cause 7 The return 8 References

[edit] Signs and portents

On Sunday, August 16, a few days before Trent was to leave on his biennial trip home, the server crashed and was inaccessible for a short time. He reported difficulties with MySql connections ^[1] π was allegedly seen tampering with the server. This came as a great surprise to Trent, who couldn't work out how Pi had got into his apartment, and to the Canadian immigration authorities, who couldn't work out how he got into the country without a passport. Some have accused him of being the cause of the crash (see also The Great Tor Debate);^[2] whilst the police say he slipped across the border from the US where, even after 9/11, they still have not fixed security holes that allow undocumented Antipodeans entry to the country.

[edit] The Crash

On Wednesday, August 19, the server crashed again. Having already left, Trent did not have physical access to the server and was unable to reach the server through the internet.^[3]

There was nothing he could do short of:

• hopping on a plane and getting back home ASAP, but he didn't do that, the lazy bastard.
• asking someone with access to his house/lodgings/flat/mansion/castle to reboot the server. But presumably nobody had a key.
• asking someone to break down the door with a sledgehammer. Come on, it's only a door. Priorities man!

[edit] Where they went

Several users invited the mob to their own websites, including Bob M, who turned over his wiki, teflpedia, to the mob. Somewhat weirdly, this was the most popular destination for refugees,^[4] and versions of some of the WIGOs were recreated, along with a replacement for the Saloon bar, the Student Bar. This had the effect of sending Teflpedia's editing rate through the roof; driving any regular editors into horrified silence; and eventually burning through the last of its bandwidth on the final day of August.

In addition, several articles were created there, to be imported once RationalWiki came back up. Slightly surreal hilarity ensued. A RW template was also issued to try and keep things separate.

Besides Teflpedia, there was extra activity at a number of places including.

• Rationalwiki blog Status of problem and updates the most logical place to look.
• A Storehouse of Codswallop Ask certainly seemed to have become more active during the period, but whether it actually benefited from the increase is a moot point. The fun included blocks, unblocks, insults and counter insults.
• Wikisynergy piqued the interest of Human, Gooniepunk2005, Bob M and a few others. In fact the site became more densely populated with skeptics than wooers. But, as with Ask, it's not clear if this was a benefit to the site. Furthermore The Battle of WikiSynergy led to a security incident (in which a direct threat was made to the site owner) took the site off-line in early September (while RW was down) but it later came back up with significantly enhanced editing restrictions.
• Wikiindex where a number of debates about something began. Initially the debate was about Proxima Centauri's editing and locking the RW article; but later the somewhat weird re-editing beliefs of User:Lumenos left everybody confused about the issue being debated. But, whatever it was, it was very important. The end result was The WikiIndex Incident in which any wiki even tangentially involved in the debate (whatever it was) has its entry blanked.

Within a few days of the return of RW the wikis returned to their normal statuses. Teflpedia whitewashed the Students' bar and returned to its plodding academic ways; Lumenos tried to keep a debate going at WikiSynergy but shortly thereafter people started to ignore him and he ended up talking to himself (although he seems to be quite happy doing this); Ask lost some of the more interesting RW members, although RW activity continued; Wikisynergy suffered most of all - the combined effects of its temporary down-time, the departure of the troll Goose Green, its changed editing rules and the reopening of RW all conspired to reduce its daily edit rate to single figures.

[edit] Contact points which could have been used but weren't really

A few people may have made a couple of comments on these sites, but there wasn't much activity.

• Liberapedia (where Proxima Centauri & SuperJosh rolled out the red carpet for exiled RationalWikians). Unfortunately, despite Proxima's slightly desperate pimping of it, it was not a popular destination. This was probably a lost opportunity for LP as it could be argued that it would have been the most obvious place to congregate.
• Psygremlin's CP blog Which claimed to be "continuing the good work of pointing out the insanity at CP." Moderation was removed during the outage in order to give editors from RW a freer hand.
• Rationalwiki IRC IRC channel - Had some minor chat, but not as much as might have been expected.
• RationalWikiWikiWikisWiki, a joke wiki, which somebody wanted use to be made of but nobody really did. We at RWW think that is shame because we would like somebody to report on us.

Rumors that some RationalWikians and fans of the site turned off their computers and got lives cannot be confirmed.

[edit] Why Teflpedia?

Nevertheless it was Teflpedia which got the lion's share of the action. One might speculate on why it was that Teflpedia - a wiki for English teachers - became the preferred summer holiday location instead of other locations which might, at first glance be more appropriate. Reasons may include:

• It was fully owned by "one of the gang" who would not object.
• It had no pre-conceived stance in respect of the issues in which RW is interested.
• It was simply the first place people went when they had an offer.
• The wiki is not highly active anyway, and in the summer it was even more quiet.
• The website is wiki-based, and attracted the wiki-comfy users.
• Everyone loves Bob.

[edit] Speculation

As nobody believes "official explanations" many conspiracy theories have been advanced about the "real" cause of the problem. Some more serious than others.

• Pi's robot (what?) was suffering from Alzheimer's and had been misplacing its keys and forgetting his daughter's name.
• The powerful Astrologers Union used their Mafia contacts to intimidate Trent, leading to suggestions that his "holiday" is actually a cover for him entering the "rationalist protection program".^[5]
• "Moslim" terrorists had taken control of the server, in an attempt to obtain the names of all the atheists.^[6]
• ListenerX's contacts from the coven had cursed the wiki in response to members of Rationalwiki challenging the witches to curse them.
• It was part of a social experiment by Trent to study how users respond when they find that their addictions cannot be fulfilled.
• It was the beginning of the final phase of Ken's Operation Flying Fortress And Ladies Undergarments. In order to defeat evolution on the Internet in a convincing way, it was claimed he had started a massive denial of service campaign against pro-evolution websites, beginning with RationalWiki.
• It was an act of God. ^[7]

[edit] Official cause

The official cause for the crash has been listed as a failing power supply.^[8]

[edit] The return

As promised, Trent turned the server back on on the 6th and, after a bit of magical data work by the data miners, everything came back on line. As a result of the crash he improved RationalWiki's fault recovery and remote administration capabilities to avoid any recurrence.^[9]

[edit] References

1. ↑ http://rationalwiki.blogspot.com/2009/08/server-down.html
2. ↑ http://www.astorehouseofknowledge.info/index.php?title=User_talk:WebOfLies&diff=prev&oldid=22893
3. ↑ http://rationalwiki.blogspot.com/2009/08/pretty-much-worse-case-scenario.html
4. ↑ For example, this article was initially created there.
5. ↑ Anyone can see these RationalWikians are being deliberately unpleasant. Only close insiders know what the real problem is though if the problem is what it seems RationalWiki will probably be back on or soon after the 6th of September. I don’t rule out that there may be a worse problem, I don’t rule out that there are legal problems involving those connected with Conservapedia or with any of the many who RationalWiki has branded practitioners of Pseudoscience. Many astrologers and other similar people quite likely have their livelihoods affected by what RationalWiki says and they will pay large sums of money to lawyers to stop what RationalWiki says about them. Yes, she really wrote this. [1]
6. ↑ In some Moslim if it comes out that a person isan atheist that can mean a death sentence, therfore users need a warning that their privacy isn't safe at RationalWiki. Yes, she said this too [2]. Despite accusing RationalWiki of grassing to Al-Qaeda, Proxima Centauri has taken no action to hide the same information on her Wikipedia user's page.
7. ↑ "This news made my day"
8. ↑ Server crash post-mortem, RationalWiki Technical blog
9. ↑ Fault protection take 2, RationalWiki Technical blog