SiteMaker Server Status

REASON: To commission new database replication technology

PLANNED DURATION: 30 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE 08:48: Maintenance completed successfully

DETAIL: Our database server has required an emergency restart

RESPONSE: We are currently restarting all dependent services

UPDATED 21:47: Services have been resolved

UPDATED 10:17 GMT 06 Jan 2010: Our apologies for this very unexpected outage. All website data is safe and all services were returned to normal in 50 minutes.

We were due to upgrade the software licences on our database layer which should have happened automatically. Unfortunately, there was a miscommunication with our database vendor that caused our database layer to shut down. After emergency discussion with our vendor the issue was resolved.

This is a unique event and will not happen again. We can only apologise for the inconvenience caused by this downtime.

DETAIL: We are currently experiencing a problem accessing our filers

RESPONSE: We will need to reset the service, this should not take much time

UPDATED 14:47: The restart was successful.

DETAIL: We are currently experiencing a problem with our internet connectivity.

RESPONSE: We are working with our provider to restore services as soon as possible.

UPDATED 13:25: Full service has been restored. We are now in contact with our provider to find out the cause of the outage.

DETAIL: We are currently subject to a distributed denial of service attack, which is causing response times to be slow.

RESPONSE: We are investigating the source of the attack and expect to have this resolved promptly.

UPDATED 16:12: We have identified the signature of the attack and have put a block in place. Service should be back to normal.

REASON: Essential Database Maintenance

PLANNED DURATION: We estimate no more than 1 hour

UPDATED 09:12: We restarted the database this morning as scheduled at 08:00. This was on the recommendation of our database vendor's technical support team to clear an urgent but non-critical issue. Although we expected this to operation to take no more than a few minutes of downtime, it has taken us a little longer. We apologise for the late notification and any inconvenience caused.

REASON: Database maintenance

PLANNED DURATION: 1 hour each (total of 3 hours)

NOTES: We will be performing some routine maintenance on the database which should result in improved overall performance of sites. The changes have been split into three different time slots during our periods of low activity in order to reduce the impact on customers.

UPDATED 08:39 BST 21 Aug: Maintenance completed successfully and service fully restored.

UPDATED 08:32 BST 24 Aug: Maintenance completed successfully and service fully restored.

UPDATED 08:15 BST 26 Aug: Maintenance completed successfully and service fully restored.

DETAIL: We are experiencing a large DDoS on one of our hosted sites, which is currently overloading our firewalls.

RESPONSE: Our technical team are working to resolve this issue as quickly as possible and should have it resolved shortly.

UPDATED 11:42: We are being flooded by an attack on one of our sites with 75% of all incoming traffic from over 2,300 IPs, which is causing all traffic to be congested and page load times to be affected. We are still currently investigating how to dynamically block this content.

REASON: Our investigations into the recent service disruptions indicated that by running consistency checks on the affected data, the file system was stabilised. To be certain that the data is secure we will be running consistency checks on the remaining data in a controlled manner.

PLANNED DURATION: 40 minutes per site

NOTES: Those communities not already affected by the recent disruptions will go Read-Only for about 40 minutes. This will only affect file uploads, new site building, forum posts and saving of data. All sites will remain on view to visitors and owners. There will be six periods of 40 minutes during which 10% of sites will be affected.

UPDATED: This has now been completed. The consistency checks have been successful.

DETAIL: A section of our system went read-only over the weekend. This has affected 20% of our customers and has an impact on file uploads, new site building, form submissions and forum posts. The other 80% of our customers are completely unaffected.

RESPONSE: Our technical team are working to resolve this issue as quickly as possible.

UPDATED 09:56 4 Apr: We are working on repairing the file system for the affected customers. Current estimate for completion is 2 - 4 hours.

UPDATED 13:40: We will be taking the service off-line for 10 minutes at approximately 13:55 to reconfigure the file system as part of our recovery actions.

UPDATED 14:04: The file system reconfiguration was successful and enabled us to restore full service to half of the affected customers. We expect to restore the service fully to the remaining 10 % of customers (the remaining half affected by this issue) within the next couple of hours.

UPDATED 15:02: Full service has now been restored to all customers.