SiteMaker Server Status - Service status

COMPLETED: Scheduled Downtime 08:30-09:00 GMT 09 Mar 2009

Eirik Pettersen — Tue, 02 Mar 2010 12:35:00 +0000

REASON: To commission new database replication technology

PLANNED DURATION: 30 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE 08:48: Maintenance completed successfully

RESOLVED: Service Outage 10:39 GMT 26 Feb 2010

Eirik Pettersen — Fri, 26 Feb 2010 10:38:00 +0000

DETAIL: A routine disk resize has hung and we need to restart the filers

RESPONSE: We are currently rebooting the necessary servers. Service should be restored shortly

UPDATE 10:48: Service has been restored. Sorry for any inconvenience caused.

UPDATE 10:52: Another reboot has been required.

UPDATE 11:00: Service has been restored.

RESOLVED: Service Outage 20:57 GMT 05 Jan 2010

Eirik Pettersen — Tue, 05 Jan 2010 21:42:00 +0000

DETAIL: Our database server has required an emergency restart

RESPONSE: We are currently restarting all dependent services

UPDATED 21:47: Services have been resolved

UPDATED 10:17 GMT 06 Jan 2010: Our apologies for this very unexpected outage. All website data is safe and all services were returned to normal in 50 minutes.

We were due to upgrade the software licences on our database layer which should have happened automatically. Unfortunately, there was a miscommunication with our database vendor that caused our database layer to shut down. After emergency discussion with our vendor the issue was resolved.

This is a unique event and will not happen again. We can only apologise for the inconvenience caused by this downtime.

COMPLETED: Scheduled Downtime 08:30-09:00 GMT 8 Dec 2009

Eirik Pettersen — Thu, 03 Dec 2009 15:42:00 +0000

REASON: Upgrade database software

PLANNED DURATION: 20 minutes

NOTES: We are upgrading our database software to the latest version. During the downtime, sites will be unavailable. Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATED 08:46: Upgrade completed without issue, Actual downtime 10 minutes. We are monitoring closely.

UPDATED 10:03: We required to perform a configuration change which required a database restart, resulting in a brief one minute downtime.

UPDATED 10:55: We required to perform another configuration change which required a database restart, resulting in a brief one minute downtime.

RESOLVED: Service Outage 14:31 GMT 27 Nov 2009

Walt — Fri, 27 Nov 2009 14:40:00 +0000

DETAIL: We are currently experiencing a problem accessing our filers

RESPONSE: We will need to reset the service, this should not take much time

UPDATED 14:47: The restart was successful.

RESOLVED: Service Outage 22:09 GMT 24 Nov 2009

Eirik Pettersen — Tue, 24 Nov 2009 22:55:00 +0000

DETAIL: We are currently experiencing another a problem with our internet connectivity.

RESPONSE: We are working with our provider to restore services as soon as possible.

UPDATED 22:24: Our network provider has confirmed the problem and are working to restore service.

UPDATED 22:52: Services appear to have been restored.

UPDATED 10:03 30 Nov: Incident report from Telstra Added

FOLLOW UP: We have received the following incident report from our upstream provider:-

UPDATE TO TELSTRA INCIDENT REPORT FOR OUTAGES ON 23 AND 24 NOV 2009

Having identified the root cause of the previous day’s incident engineers planned an activity to configure the x.x.0.0/16 summary route on to the appropriate Juniper core routers to allow a future decommissioning of the legacy routers. The configuration of this summary route on the Juniper core routers resulted in Internet services for certain customers being affected again.

The engineers were unable to quickly isolate the cause of this issue and so reversed the change in order to restore service. However once the change had been reversed service was not restored for all customers as it should have been. The engineers identified a spurious route being received from the legacy routers which appeared to be causing the problem. The engineers reset the BGP sessions to the legacy routes which removed the spurious route and restored service to affected customers.

The engineers later identified a Juniper OS bug that had caused the reversal to be unsuccessful. Telstra has already been testing a later version of the Juniper OS in their labs which is intended for network wide deployment. Juniper has confirmed that this specific bug is resolved in the release in test but Telstra will also include this bug in their test planning prior to deployment in production networks.

REMEDIAL ACTION

An urgent cross-functional review of the current MPLAN process has been scheduled (including a detailed analysis of our planning and handling of this incident).
All MPLAN’s now have an extended Director level approval policy whilst we review the current planned works process.
All MPlans will be checked after completion to ensure that the works have been carried out in accordance with the plan.

Telstra would like to take this opportunity to sincerely apologise for the disruption and inconvenience that these incidents have caused. Please be assured that the immediate actions stated above have been given the highest priorities within Telstra to be implemented as quickly as possible. This is in order to avoid further incidents and to provide the highest levels of service to our customers.

RESOLVED: Service Outage 13:02 GMT 23 Nov 2009

Walt — Mon, 23 Nov 2009 13:11:00 +0000

DETAIL: We are currently experiencing a problem with our internet connectivity.

RESPONSE: We are working with our provider to restore services as soon as possible.

UPDATED 13:25: Full service has been restored. We are now in contact with our provider to find out the cause of the outage.

FOLLOW UP: We have received the following incident report from Telstra

Telstra Customer Incident Report

1. Affected Services

Multiple service impacts for London Hosting Centre customers.

2. Start Date & Time

Start: 23/11/09 12:50

End: 23/11/09 13:22

Telstra Support Ticket: Master ticket 535029

3. Outage Description

Partial loss of connectivity at London Hosting Centre

4. Impact

Services hard down during incident timeline.

Telstra MSO TT535029 20091123

5. Summary of Events

23/11/2009 12:50 Service alert details correlated and incident designated as MSO (Major Service Outage).

- Investigations found that the outages were caused by a missing line of configuration from the core network.

- Route announcement was reapplied and route tables correctly converged.

26/08/09 13:22 All affected services showing as restored.

6. Remedial Action

During routine planned works to decommission old equipment, a route announcement was briefly removed. This was discovered and replaced and the route tables were correctly re-converged.

COMPLETED: Scheduled Mail Service Maintenance 00:00-06:00 GMT 28, 29, 30 Oct 2009

Walt — Wed, 28 Oct 2009 17:42:00 +0000

REASON: Network distribution maintenance

PLANNED DURATION: 3 days (limited to a few hours each morning)

NOTES: Our domain partner Gandi will be performing major upgrades to their network distribution infrastructure during the nights of 28 to 30 October. During this time email services will experience a number of intermittent disruptions in connectivity beginning at 12:00am (GMT) with maintenance activities finishing by 6:00am (GMT) each day. We will endeavour to minimise impact to services where possible, but given the scale of the upgrades being performed, most services will encounter at least some disruption. No mail will be lost but access may be restricted and delivery of mail delayed.

We apologise in advance for any inconvenience this work may cause.

RESOLVED: Service Disruption 15:45 BST (GMT+1) 17 Oct 2009

Eirik Pettersen — Sat, 17 Oct 2009 15:41:00 +0100

DETAIL: We are currently subject to a distributed denial of service attack, which is causing response times to be slow.

RESPONSE: We are investigating the source of the attack and expect to have this resolved promptly.

UPDATED 16:12: We have identified the signature of the attack and have put a block in place. Service should be back to normal.

RESOLVED: Service Disruption 15:30 BST (GMT+1) 15 Oct 2009

Walt — Thu, 15 Oct 2009 15:37:00 +0100

DETAIL: We are currently experiencing some unexpected problems with our service delivery.

RESPONSE: We are investigating this and expect to have this resolved promptly.

UPDATED 16:20: We were performing a routine maintenance operation to expand disk space. This normally requires no downtime, however the operation hung causing a cascading problem which required us to restart the entire service. Because the restart was unscheduled we took great care to minimise any associated risks. We apologise for any inconvenience caused.

COMPLETED: Essential Maintenance 08:00 BST (GMT+1) 8 Oct 2009

Walt — Wed, 07 Oct 2009 18:35:00 +0100

REASON: Essential Database Maintenance

PLANNED DURATION: We estimate no more than 1 hour

UPDATED 09:12: We restarted the database this morning as scheduled at 08:00. This was on the recommendation of our database vendor's technical support team to clear an urgent but non-critical issue. Although we expected this to operation to take no more than a few minutes of downtime, it has taken us a little longer. We apologise for the late notification and any inconvenience caused.

POSTPONED: Scheduled Downtime 06:00-08:00 BST (GMT+1) 25 Sep 2009

Josh — Tue, 22 Sep 2009 13:29:00 +0100

REASON: Data cleansing and archiving

PLANNED DURATION: 2 hours

NOTES: To improve the efficiency of our database, we plan to archive legacy data. This operation needs to be done off-line, that is all websites must be taken off-line for this period. Customers will be shown a page advising them that their site is unavailable and will be back up shortly.

We apologise for any inconvenience.

COMPLETED: Scheduled Downtime 07:30-08:30 BST (GMT+1) 21, 24, 26 Aug 2009

Josh — Fri, 14 Aug 2009 16:33:00 +0100

REASON: Database maintenance

PLANNED DURATION: 1 hour each (total of 3 hours)

NOTES: We will be performing some routine maintenance on the database which should result in improved overall performance of sites. The changes have been split into three different time slots during our periods of low activity in order to reduce the impact on customers.

UPDATED 08:39 BST 21 Aug: Maintenance completed successfully and service fully restored.

UPDATED 08:32 BST 24 Aug: Maintenance completed successfully and service fully restored.

UPDATED 08:15 BST 26 Aug: Maintenance completed successfully and service fully restored.

RESOLVED: Service Disruption 09:11 BST (GMT+1) 2 Jul 2009

Walt — Thu, 02 Jul 2009 09:46:00 +0100

DETAIL: We are experiencing a distributed denial of service (DDoS) attack on the services, which is currently overloading our firewalls.

RESPONSE: Our technical team are working hard to get this attack under control and return full service to all customers. We hope to have this completely resolved shortly.

UPDATED 12:32: We've managed to block what appears to have been three different types of DDoS attack. Combating the three attacks has taken us longer than expected but we appear to have got them under control as of 11:35 but continued to experience minor issues until 12:32.

RESOLVED: Service Disruption 06:10 BST (GMT+1) 25 Jun 2009

Eirik Pettersen — Thu, 25 Jun 2009 07:49:00 +0100

DETAIL: We are experiencing a large DDoS on one of our hosted sites, which is currently overloading our firewalls.

RESPONSE: Our technical team are working to resolve this issue as quickly as possible and should have it resolved shortly.

UPDATED 11:42: We are being flooded by an attack on one of our sites with 75% of all incoming traffic from over 2,300 IPs, which is causing all traffic to be congested and page load times to be affected. We are still currently investigating how to dynamically block this content.

COMPLETED: Scheduled Downtime 09:30-10:30 BST (GMT+1) 18 May 2009

Eirik Pettersen — Mon, 11 May 2009 11:04:00 +0100

REASON: Database upgrade and updates

PLANNED DURATION: Up to 60 minutes

NOTES: We will be upgrading the database software to the latest version, and performing a number of data updates in preparation for the release of the next version of SiteMaker.

Thank you for your patience and apologies for any inconvenience caused.

COMPLETED: Essential Maintenance 11:20 BST (GMT+1) 9 Apr 2009

Walt — Thu, 09 Apr 2009 11:30:00 +0100

REASON: Our investigations into the recent service disruptions indicated that by running consistency checks on the affected data, the file system was stabilised. To be certain that the data is secure we will be running consistency checks on the remaining data in a controlled manner.

PLANNED DURATION: 40 minutes per site

NOTES: Those communities not already affected by the recent disruptions will go Read-Only for about 40 minutes. This will only affect file uploads, new site building, forum posts and saving of data. All sites will remain on view to visitors and owners. There will be six periods of 40 minutes during which 10% of sites will be affected.

UPDATED: This has now been completed. The consistency checks have been successful.

RESOLVED: Service Disruption 16:40 BST (GMT+1) 7 Apr 2009

Walt — Tue, 07 Apr 2009 23:44:00 +0100

DETAIL: On April 7 a section of our system went read-only. This currently affects only 10% of our customers and has an impact on file uploads, new site building, form submissions and forum posts. The other 90% of our customers are completely unaffected by this event.

RESPONSE: Our technical team are working to resolve this issue as quickly as possible and should have it resolved shortly.

UPDATED 01:45: All affected systems have now been fully restored. We will continue to monitor the situation and post more details later in the morning.

RESOLVED: Service Disruption 14:34 BST (GMT+1) 4 Apr 2009

Walt — Mon, 06 Apr 2009 09:56:00 +0100

DETAIL: A section of our system went read-only over the weekend. This has affected 20% of our customers and has an impact on file uploads, new site building, form submissions and forum posts. The other 80% of our customers are completely unaffected.

RESPONSE: Our technical team are working to resolve this issue as quickly as possible.

UPDATED 09:56 4 Apr: We are working on repairing the file system for the affected customers. Current estimate for completion is 2 - 4 hours.

UPDATED 13:40: We will be taking the service off-line for 10 minutes at approximately 13:55 to reconfigure the file system as part of our recovery actions.

UPDATED 14:04: The file system reconfiguration was successful and enabled us to restore full service to half of the affected customers. We expect to restore the service fully to the remaining 10 % of customers (the remaining half affected by this issue) within the next couple of hours.

UPDATED 15:02: Full service has now been restored to all customers.

RECOVERY: We failed over to our backup file server and performed consistency checks to ensure that the data was not corrupted.

FOLLOW UP: We will run background consistency checks on all our data over the next few days to ensure that no further problems occur. Visitors may possibly experience slower page loads while these checks take place. We will also conduct an investigation into the cause of this event and take steps to mitigate a repeat of this incident.

RESOLVED: Service Disruption 09:12 GMT 10 Mar 2009

Walt — Tue, 10 Mar 2009 13:02:00 +0000

DETAIL: This morning we suffered a distributed denial of service attack on one of our sites. This affected the speed at which others sites could be accessed which in some cases resulted in time-outs for some users.

RESPONSE: Engineers identified the problem early, However time was needed to establish how the attack was being conducted in order to block it.

UPDATED 10:01: The attack has been blocked and services should have returned to normal.

RECOVERY: Some users may have experienced login issues following the resolution, although clearing the cache would have easily resolved this problem. Other than this no other sites have been impacted by this event.

FOLLOW UP: We are now reviewing what steps we can take to minimise the risks of this occurring again.