Results 1 to 5 of 5

Thread: More info on outage?

  1. #1
    Junior Member Newbie
    Join Date
    Feb 2009
    Posts
    3

    Default More info on outage?

    Luckily I didn't have all my sites on one account, because my VPS came up some time last night, while my shared plan still hasn't come back up 18 hours later. Since this handles my business email, the $13/mo doesn't matter at all. All that matters is that my customers can't see my website and worse: I can't receive any emails.

    It'd sure be nice to get more information than some vague "we're working on it" with such a major problem. I'd also like to know what you guys are going to do to avoid this problem in the future. Electrical failure at the datacenter? Um, when I worked for a hosting company, we had huge generators on the roof of the data center, massive fire suppression inside, and we did regular off-site backups so that we could get SOMEthing up quickly if needed.

  2. #2
    Junior Member Newbie
    Join Date
    Feb 2009
    Posts
    3

    Default

    So, 21 hours. Nothing?

  3. #3
    Junior Member Newbie
    Join Date
    May 2009
    Posts
    1

    Default

    I am in the same situation, and agree.

    This low level of communication is unbelievable and unacceptable.

    -k

  4. #4
    Junior Member Newbie
    Join Date
    Feb 2009
    Posts
    3

    Default

    I'm thinking of starting a will-hear-from-spry clock, like the U.S. debt clock in Times Square, on my website.....oh wait....no website.....

  5. #5
    Senior Member Power Poster
    Join Date
    Dec 2007
    Posts
    359

    Default

    All shared hosting services should be back online at this time - please accept our apologies for the delay in response on this issue, it appears as though e-mail notifications from the Spry billing system (as well as forum post notifications to our support department) have been stalled.

    An outgoing notification to all shared hosting account holders has been sent, however, pending its arrival, here is a copy for immediate review:


    Firstly, please accept our sincere apologies for the events of the past twenty four hours - we have identified many issues with redundant systems which deserve an explanation and immediate action on our part. This incident represents a "perfect storm" of failures in our electrical failover systems and, as with any incident of this scope, we're taking every opportunity to improve our processes to prevent its recurrence.

    To summarize the incidents of the past twenty four hours:

    • 5/18/2009 15:10 PST - A measurable loss in voltage from our datacenter's main feeds at Seattle City Light was recorded. This loss in voltage triggered our UPS failovers and all shared hosting servers and routers began running on UPS failover power.
    • 5/18/2009 16:05 PST - UPS systems' battery reserves began to deplete and shared hosting servers began to shut down. We have confirmed that the UPS systems were working properly - the loss in voltage tripped their circuits and forced all connections with our datacenter's feeds down. This action, unfortunately, prevented the UPS systems from recharging their batteries from available power following the brownout.
    • 5/18/2009 19:00 PST - We did not have sufficient evidence to believe that the issue at Seattle City Light had been resolved. Our UPS service company and a technician arrived, an electrician from Titan Electric was dispatched, and our datacenter technicians switched from the Seattle City Light feed to diesel backup generator backups. Upon confirming that all shared hosting servers had sufficient power, our system administration team began bringing servers and key routing hardware back up (total potential outage time for all services at this point: three hours). Our internal services (including our ticketing system, our billing system, the My.Spry.com control center, and the forums) were unavailable for the duration of the outage. We were not able to send broadcasts or post to the forums regarding the cause of and action being taken on this issue.
    • 5/18/2009 20:00 PST - We began restoring shared hosting servers, starting with filesystem integrity checks. Key routing hardware was restored to service. Internal systems remained offline while our system administrators and developers worked to restore service to nodes hosting internal databases.
    • 5/18/2009 21:30 PST - Our ticketing system was restored to service and broadcast replies were sent to all tickets received.
    • 5/19/2009 01:30 PST - All internal services were restored to service and a notification was posted to the Spry Forums approximately eight hours after the incident began. We recognize that hours of downtime without explanation or a way to establish contact with support and find out what has happened is unacceptable, and we want to ensure that everyone affected by this outage has the opportunity to claim service credits in accordance with our Service Level Agreement.
    • 5/19/2009 23:40 PST - All shared hosting servers have been restored to service. If you encounter any difficulty accessing your service, please create a support ticket for immediate assistance.


    We are presently investigating appropriate courses of action to prevent anything like this from occurring again at our datacenter and we will post updates on progress at the Spry Forums:

    Spry Datacenter Incident 5/18/2009 approx 18:30 PST - ongoing
    URL: http://forums.spry.com/showthread.php?t=3881

    Finally, we have had the opportunity to review the very reasonable requests submitted by our subscribers in response to this incident and the communication blackout which occurred will not occur again. By June 1st, 2009 we will have a redundant page in place to address any issues which affect the Spry.com site and ticketing system by providing a direct e-mail address to contact our staff.

    Additionally, we have created a twitter.com account for all significant events on our network and any news items which may affect your service (particularly for instances when our notification tools are unavailable):

    Twitter.com: spry
    https://twitter.com/spryhosting

    Twitter.com Spry RSS Feed:
    http://twitter.com/statuses/user_timeline/41295499.rss

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •