Host: Major System-Wide Outage

Important news and updates from the Fembot Central staff about the site.
(Only staff can start topics but members can post replies.)
Posts: 519
Joined: Fri May 24, 2002 6:26 pm

Host: Major System-Wide Outage

Postby KOS-MOS » Sat Apr 30, 2011 3:57 pm

You may have noticed Fembot Central and Fembot Wiki are sluggish today. This is due to a "Major System-Wide Outage" our host, DreamHost, is experiencing. At any time you can check their list of emergency announcements at their DreamHost Status Blog or follow @dhstatus on twitter.

At the time of this posting this is what they have to say about the situation:
We are investigating the cause and will update this post as soon as we know more.

Update 2:54pm PST: One of our network providers is having issues right now. Our network admins are working on moving all traffic to our two other service providers. We will keep updating this post as soon as we have more information.

Update 3:10pm PST: We are having issues with one of our transit links to our LAX datacenter, our network administrators are investigating the issue and we hope to have it resolved as soon as humanly possible. This is causing problems with both external and internal networking – we’ll update this post as soon as we have more information or the issue is resolved.

Update 3:30pm PST: Our network engineers are on site at the source of the issue and working on resolving it. More information to follow as it becomes available.

User avatarRobotman
Fembot Central Staff
Posts: 7667
Joined: Tue May 13, 2003 7:01 pm
Location: Calgary

Re: Host: Major System-Wide Outage

Postby Robotman » Sat Apr 30, 2011 6:13 pm

The saga contirnues:
Update 3:50pm PST: We are still working feverishly to resolve this issue – some portions of our network are responding better now but there’s more work to be done. -John

Update 4:10pm PST: We are seeing improvement but are still working on this issue. -John

Update 4:40pm PST: Our network engineers rebooted the core router at our LAX facilities but our providers are still flapping. It appears that the LAX facility may be stabilized but we’re experiencing high load on our core routers as a result of all the BGP flapping. We are still investigating the issue but current traffic patterns possibly indicate an attack on our network. -John

Update 5:00pm PST: We are still working to defeat this attack on our network. Nullifying certain source IPs has seemed to help matters, but that attack vectors and profiles are still being introduced and identified. -John

Update 5:25pm PST: Some attacks have been blocked that were inbound and we have identified some that were outbound and blocked those as well. We’re still working on the matter and the network responsiveness is improving but the attacks are still causing BGP flapping. -John

Update 5:50pm PST: We’ve pinned down the sources and nature of the attacks and are in the process of scanning for more attack software as well as filtering bad traffic at the core routers. The issue is not completely resolved yet but we’re making good progress now. More updates to follow. -John


Return to Announcements



Who is online

Users browsing this forum: No registered users and 1 guest