SiteMaker Server Status

We will be updating the core software to version R5.3.5 at 10:00 BST (GMT+1), 23 June 2011. There may be a momentary service disruption, but this should be unnoticeable. We recommend users save any changes before this time. After this time, we advise customers to hit reload on their browsers to ensure they have the latest version.

DETAIL: At 08:03 BST (GMT+1), we saw a spike in traffic resulting in a high load on our dynamic serving layer.

RESPONSE 08:20: Service has returned to normal.

UPDATE 15:13 BST 16 May: On further investigation, the platform outrage seems to be related to a known existing bug that can cause a cascade of blocking operations on the backend layer. The bug has been reprioritised and should be resolved shortly, preventing such incidents from happening again in the future.

We will be updating the core software to version R5.3.0 at 10:30 BST (GMT+1), 19th April 2011. There may be a momentary service disruption, but this should be unnoticeable. We recommend users save any changes before this time. After this time, we advise customers to hit reload on their browsers to ensure they have the latest version.

UPDATE 10:40: There was a slight problem with the release, which is currently being addressed. Once we get a patch, we plan to continue with the release once we've verified the fix. We hope to rollout this patch within the next hour.

UPDATE 12:10: The patch was successful. The update has been rolled out. Thanks for your patience and we apologies for any inconvenience.

DETAIL: Last night one of our senior technical staff were doing a routine operation to improve capacity to one of our file servers. This can be done without downtime which benefits customers, but during this operation there was a lockup which left the resize process "waiting". This lead to some web requests (website access by customers or visitors) to wait for disk access along with the already waiting resize process. Once the queue of waiting web requests got to capacity the platform went down. Rebooting the file server and letting the fail over mechanism kick in solved the issue. The period of outage lasted from 19:10 until 19:46.

RESPONSE: This issue was due to human an error by SiteMaker. While we do sincerely apologise for this incident we will also be updating our processes to make sure this does not happen again.

We will be updating the core software for SiteMaker to version R5.2.0 at 12:30 GMT, 3 Mar 2011. There may be a momentary service disruption, but this should be unnoticeable. We recommend users save any changes before this time. After this time, customers should reload their sites to ensure they have the latest version.

DETAIL: We have lost outbound connectivity from our datacentre, This is unrelated to the software release this morning.

RESPONSE: We are currently working with our network providers to find out the cause.

UPDATE 17:18: Connectivity seems to have been restored. We are still trying to determine the cause.

UPDATE 18:34: We are still chasing our network provider for an explanation.

UPDATE 11:56 05 Feb: Our network provider came back with the following response:

Telstra can advise that the outage of your service may have been
affected indirectly by an emergency network change. Our 3rd line network
support team were required to make a small amendment to a network IP
address in one of our core devices and your service may have been
affected during a short period whilst the change converged through our
network. The change should have been non service affecting. However it
appears that the times you reported, and the fact that your cct is build
across the same core device we believe that the cct may have taken
itself down whilst the routing re-established itself.

We will be updating the core software for SiteMaker to version R5.1.0 at 09:30 GMT, 27 Jan 2011. There may be a momentary service disruption, but this should be unnoticeable. We recommend users save any changes before this time. After this time, customers should reload their sites to ensure they have the latest version.

UPDATE 09:06: We have postponed the update for 30 minutes. It will now go out at 10:00 GMT.

UPDATE 10:00: We are still having some configuration issues. It will now go out at 10:30 GMT

UPDATE 10:20: We believe we've identified the cause of our last minute issue. We have to write, test and apply a patch. We estimate now a go live time of 11:30 GMT. Apologies for the inconvenience.

UPDATE 11:06: The patch is taking longer than expected to complete. We've pushed the go live time back to 12:30 GMT.

UPDATE 12:30: We have gone live with the update.

UPDATE 13:10: The new code is still causing load issues and we have been forced to roll back. We will advise you of the new release date once the issues have been diagnosed and fixed. Again, sorry for the inconvenience and thank you for your patience.

DETAIL: There has been a severe degradation in service in the new caching layer that has locked up the entire platform down

RESPONSE: We are reverting to the old caching layer to restore service

UPDATE 18:28: Service has been restored

UPDATE 19:30: The indices on the caching layer had been incorrectly configured and the caching layer had slowed down gradually until finally locking up at 18:28 today. The index configuration has been corrected and the issue should not happen again.

REASON: To commission new database replication technology

PLANNED DURATION: 30 minutes

Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.

UPDATE 08:48: Maintenance completed successfully

DETAIL: Our database server has required an emergency restart

RESPONSE: We are currently restarting all dependent services

UPDATED 21:47: Services have been resolved

UPDATED 10:17 GMT 06 Jan 2010: Our apologies for this very unexpected outage. All website data is safe and all services were returned to normal in 50 minutes.

We were due to upgrade the software licences on our database layer which should have happened automatically. Unfortunately, there was a miscommunication with our database vendor that caused our database layer to shut down. After emergency discussion with our vendor the issue was resolved.

This is a unique event and will not happen again. We can only apologise for the inconvenience caused by this downtime.