SiteMaker Server Status

RESOLVED: Service Outage 20:16 BST (GMT+1) 07 May 2012

Eirik Pettersen — Mon, 07 May 2012 20:34:00 +0100

We are currently experiencing issues and investigating the cause. We'll update this post as we get more information.

UPDATE 20:46: Services have returned to normal. We have identified and blocked incoming traffic which was causing our load balancers to fail. We are currently investigating the structure of the requests to determine why they caused a denial of service and once we have determined this we can put measures in place to prevent this from reoccurring. Apologies for any inconvenience.

UPDATE 21:49: The malicious traffic has returned from another address - we are diagnosing and hope to have it blocked as soon as we can.

UPDATE 22:41: We have identified the vulnerability that has been exploited that caused the denial-of-service situation earlier tonight. We are upgrading the relevant software which should protect against further attacks.

UPDATE 00:32: We have now successfully updated and reconfigured the relevant software packages and configuration files and have tested successfully that we are no longer vulnerable to this susceptibility.

COMPLETED: Scheduled Downtime 08:00-08:10 BST (GMT +1) 02 May 2012

Josh — Mon, 30 Apr 2012 19:56:00 +0100

REASON: Database Updates

PLANNED DURATION: 10 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE 08:07: Maintenance completed successfully.

CANCELLED: Scheduled Downtime 08:00-08:10 BST (GMT +1) 26 Apr 2012

Josh — Wed, 25 Apr 2012 13:41:00 +0100

REASON: Database Updates

PLANNED DURATION: 10 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE 2018: This has now been cancelled.

COMPLETED: Scheduled Downtime 08:00-08:10 BST (GMT +1) 24 Apr 2012

Josh — Fri, 20 Apr 2012 19:25:00 +0100

REASON: Database Updates

PLANNED DURATION: 10 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE 19:45: This has now been postponed to 08:00 BST (GMT +1) 25 Apr 2012.

UPDATE 09:05: We are experiencing some performance issues following the maintenance. Engineers are working to restore normal service.

RESOLVED: Service Outage 23:20 BST (GMT+1) 5 Apr 2012

Josh — Thu, 05 Apr 2012 23:20:00 +0100

We have currently carrying out emergency maintenance on a critical database server. Engineers are working to resume full service as soon as possible.

UPDATE 00:01: Service has now been restored.

RESOLVED: Service Outage 16:40 BST (GMT+1) 5 Apr 2012

Russell — Thu, 05 Apr 2012 16:50:00 +0100

We have experienced a failure of a critical database server. Engineers are working to resume full service as soon as possible.

UPDATE 17:18: Service has now been restored.

RESOLVED: Service Outage 13:25 GMT 26 Mar 2012

Russell — Mon, 26 Mar 2012 14:48:00 +0100

We experienced a brief outage at 13:25 GMT today (14:25 BST) during a routine server update. The cause was identified and service restored by 13:37 GMT.

RESOLVED: Service Degradation 09:00 GMT 23 Mar 2012

Russell — Fri, 23 Mar 2012 12:09:00 +0000

We have been experiencing a database performance issue this morning, resulting in customer sites loading more slowly than normal. Engineers are investigating.

UPDATED 17:42 GMT: The cause has been identified as a hardware failure. Replacement hardware has been ordered and will be installed this evening.

UPDATED 07:35 GMT 24 Mar: The hardware was replaced and the rebuld and repair completed at 03:06 GMT. We expect the issue has been resolved but we will keep a close eye on it today.

RESOLVED: Email delays 16:15 GMT 10 Mar 2012

Russell — Sat, 10 Mar 2012 22:25:00 +0000

22:06 GMT 10 Mar 2012 We have been experiencing an outage on the Sitemaker email platform since earlier this evening causing delays to email transmission.

Emails will not be lost. They are sitting in queues waiting to go out.

Unfortunately attempts to resolve the problem remotely have failed and an engineer will be attending the datacentre to bring services back online.

COMPLETED: Planned Engineering Works 22:00-02:00 GMT 8 Mar 2012

Josh — Wed, 07 Mar 2012 12:59:00 +0000

REASON: Replacement of structured cabling by our infrastructure provider

PLANNED DURATION: Up to 5-10 minutes loss of service

Siteleaders and visitors will not be able to access their sites for 5-10 minutes during the maintenance window.

COMPLETED: Scheduled Downtime 08:00-08:10 BST (GMT) 27 Feb 2012

Josh — Mon, 27 Feb 2012 17:22:00 +0000

REASON: Database Updates

PLANNED DURATION: 10 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE: Completed 08:10 GMT 27 Feb 2012

RESOLVED: Service Degradation 15:26 GMT 14 Feb 2012

Russell — Tue, 14 Feb 2012 18:30:00 +0000

16:30 14 Feb 2012 Some customers will be unable to upload files to their sites. This affects approximately 10% of all sites. This was caused by a failure on one of our storage servers. Affected sites can still be edited and viewed without issue. This may take time to repair as our main concern is to keep all sites - including the 10% affected by this problem - online, editable and available.

We will provide an update as soon as we have more information.

UPDATE 07:01 15 Feb: We have come across some issues in performing the recovery overnight. We are continuing attempts today using an alternative recovery method.

UPDATE 17:00 15 Feb: We are still working on recovery and have been making progress throughout the day.

UPDATE: 10:50 16 Feb: This issue should now be resolved, we'll be running a number of tasks in the background but this should not impact any sites.

COMPLETED: SiteMaker Update and Scheduled Downtime 08:30-09:00 GMT 15 Feb 2012

Eirik Pettersen — Mon, 13 Feb 2012 17:01:00 +0000

REASON: Updating SiteMaker software to version R5.7. The release incorporates database updates that require us to be offline to apply.

PLANNED DURATION: 10 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE: The release will now go ahead at 08:30 GMT 16 Feb 2012

UPDATE: Completed 09:00 GMT 16 Feb 2012

RESOLVED: Service Outage 09:56 GMT 13 Feb 2012

Russell — Mon, 13 Feb 2012 10:15:00 +0000

We are experiencing an outage following a routine storage expansion. Engineers are working to resolve the issue and we expect to restore service soon. Sorry for the inconvenience.

UPDATE 10:35: Service was fully restored at 10:25 GMT.

COMPLETED: SiteMaker Update 10:30 GMT 26 Jan 2012

Josh — Wed, 25 Jan 2012 13:54:00 +0000

We will be updating the core software to version R5.6 at 10:30 GMT, 26 January 2012. There may be a service disruption lasting up to 30 minutes. We recommend users save any changes before this time. After this time, we advise customers to hit reload on their browsers to ensure they have the latest version.

RESOLVED: Service Outage 00:00 GMT 9 Jan 2012

Josh — Mon, 09 Jan 2012 00:17:00 +0000

We are experiencing problems with our firewall routers. Someone is on site at the data center working on this issue. Sorry for the inconvenience.

UPDATE 01:00 GMT: the firewalls are now up and seem to be failing over correctly.

RESOLVED: Service Outage 17:20 GMT 6 Jan 2012

Eirik Pettersen — Fri, 06 Jan 2012 17:32:00 +0000

DETAIL: We are experiencing more problems with our firewall routers. The cause is being investigated. Sorry for the inconvenience

UPDATE 19:20 GMT: Service was restored at 18:07 GMT and the firewall hardware has since been replaced. We expect this to resolve the issue once and for all.

RESOLVED: Service Outage 20:19 GMT 4 Jan 2012

Eirik Pettersen — Wed, 04 Jan 2012 20:25:00 +0000

DETAIL: We are experiencing a problem with our firewall routers. The cause is being investigated.

UPDATE 20:38: It appears our primary firewall layer froze and the secondary firewall failed to pick up the problem, therefore knocking out access to the platform. A forced reboot of the primary caused the secondary to correctly take control and restore access.

RESOLVED: Service Outage 18:30 GMT 30 Dec 2011

Russell — Fri, 30 Dec 2011 19:05:00 +0000

DETAIL: We are experiencing problems with our firewall routers which are causing intermittent brief outages. The cause is being investigated.

UPDATE 20:50: Services have been more stable over the past hour as failover systems have taken control, although brief outages may still be occurring. An engineer is travelling to the data centre to stabilise services further.

UPDATE 23:40: Services have been fully stable for 1 hour now. Enhanced diagnostics have been put into place and monitoring will continue.

COMPLETED: SiteMaker Update 10:30 GMT 15 Dec 2011

Walt — Wed, 14 Dec 2011 18:14:00 +0000

We will be updating the core software to version R5.5.2 at 10:30 GMT, 15 December 2011. There may be a service disruption lasting up to 30 minutes. We recommend users save any changes before this time. After this time, we advise customers to hit reload on their browsers to ensure they have the latest version.