SiteMaker Server Status

DETAIL: We are experiencing an unscheduled outage caused by a number of rogue processes taking up resources on the web-servers. This is resulting in slow access times and dropouts for all websites.

RESPONSE: The rogue processes are being identified and eliminated

UPDATED 12:48: All services have resumed We apologize for any inconvenience caused.

DETAIL: We have suffered two incidents today, one starting at 12:52 lasting for 13 minutes resulting in very slow response times, and another one starting at 15:32 lasting for 1hr 40 minutes, which included some periods of full loss of service.

RESPONSE: Engineers are working to identify the causes

UPDATED 17:12: Service has been restored to normal and all sites appear to be up and running with normal response times.

DETAIL: We appear to have suffered a power cut at our data center.

RESPONSE: We are currently investigating and working to restore service.

UPDATED 21:00: All services are now up and running, customers may experience a little bit of latency in accessing their sites. We apologise for any inconvenience caused and we continue to investigate the cause of this outage.

UPDATED 10:12 24 Oct: Because the power failure caused our machines to shut down unexpectedly, our disk array which contains uploaded files restarted in 'read only' mode as a precaution. This means we have to run full disk integrity checks before we can go back to 'write' mode. Only 50% of sites have been affected by this, and we will repair them 10% at a time.

In any event, all content on these disks is backed up both on and off site, so in event of any errors in the disk, we can fully restore the files.

We will let you know when this is complete.

UPDATED 14:00: All services have been restored to normal operation. We continue to investigate the causes of this outage and will provide a full report as soon as it is available.

DETAIL: We are experiencing an unscheduled outage.

RESPONSE: This appears to be caused by a number of rogue processes taking up resources on the web-servers. This resulted in slow access times and dropouts for all websites. The process was killed and steps have been taken to prevent this from happening again.

UPDATED 20:20: Service has returned to normal. We apologize for any inconvenience caused.

REASON: Upgrade of our gateway firewall

PLANNED DURATION: 1 hour

NOTES: Once the upgrade is complete, normal service will resume. We thank you in advance for your co-operation.

DETAIL: We experienced a service outage between 14:24 and 14:55.

RESPONSE: A database deadlock situation caused a backlog for the requests coming in. We were forced to take our webservers off-line to ease the load on the database long enough to resolve the deadlock.

DETAIL: We experienced a service outage between 10:45 and 11:10.

RESPONSE: This was caused by our database refusing to accept new connections thus requiring us to restart the service. The reason for this is still unclear but we will continue to monitor the situation closely.

DETAIL: SiteMaker services are currently due to a network outage at our data centre affecting not just SiteMaker.

RESPONSE: Our network providers engineers are working to resolve the issue as soon as possible. We will provide updates as soon as we have them.

UPDATED 10:15: All services should be back to normal. We will provide further information on the cause of the outage once or network provider makes it available to us. Apologies for this service disruption.

UPDATED: We have now received an update from our connectivity provider regarding the outage last Friday. A severe software bug was encountered in one of their core routers Our provider sought emergency support from the hardware vendor after being unable to remedy the situation themselves. The hardware vendor identified the issue as a 'major severity problem' and instructed them through the necessary remedial steps. We have been assured that the workaround configuration provided by the hardware vendor should pose no further threat of similar issues. The software bug is due to be fixed in an imminent new release and once available our provider will take steps to upgrade.

REASON: Apply latest database patches

PLANNED DURATION: 15 minutes

NOTES: Rescheduled from 9 Nov 2007

UPDATED 10:13: Started at 09:50. Actual duration 23 minutes. Completed without issue.

DETAIL: We have lost internet connectivity

RESPONSE: We are working with our connectivity provider to understand the cause of the fault.

UPDATED: Normal service has been restored. Our network provider has suffered an outage in core equipment which has taken down all their London sites. We are told this is being upgraded to prevent the problem reoccuring. We will follow up with our service provider and post a summary of what happened as we know more. Apologies for the loss in service during this time.