Tuesday, March 2 2010
By Eirik Pettersen on Tuesday, March 2 2010, 12:35
REASON: To commission new database replication technology
PLANNED DURATION: 30 minutes
Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.
UPDATE 08:48: Maintenance completed successfully
Friday, February 26 2010
By Eirik Pettersen on Friday, February 26 2010, 10:38
DETAIL: A routine disk resize has hung and we need to restart the filers
RESPONSE: We are currently rebooting the necessary servers. Service should be restored shortly
UPDATE 10:48: Service has been restored. Sorry for any inconvenience caused.
UPDATE 10:52: Another reboot has been required.
UPDATE 11:00: Service has been restored.
Tuesday, January 5 2010
By Eirik Pettersen on Tuesday, January 5 2010, 21:42
DETAIL: Our database server has required an emergency restart
RESPONSE: We are currently restarting all dependent services
UPDATED 21:47: Services have been resolved
UPDATED 10:17 GMT 06 Jan 2010: Our apologies for this very unexpected outage. All website data is safe and all services were returned to normal in 50 minutes.
We were due to upgrade the software licences on our database layer which should have happened automatically. Unfortunately, there was a miscommunication with our database vendor that caused our database layer to shut down. After emergency discussion with our vendor the issue was resolved.
This is a unique event and will not happen again. We can only apologise for the inconvenience caused by this downtime.
Thursday, December 3 2009
By Eirik Pettersen on Thursday, December 3 2009, 15:42
REASON: Upgrade database software
PLANNED DURATION: 20 minutes
NOTES: We are upgrading our database software to the latest version. During the downtime, sites will be unavailable. Siteleaders will not be able to login to their sites and visitors to sites will see a unbranded page letting them know that essential maintenance is underway and that they should return later.
UPDATED 08:46: Upgrade completed without issue, Actual downtime 10 minutes. We are monitoring closely.
UPDATED 10:03: We required to perform a configuration change which required a database restart, resulting in a brief one minute downtime.
UPDATED 10:55: We required to perform another configuration change which required a database restart, resulting in a brief one minute downtime.
Friday, November 27 2009
By Walt on Friday, November 27 2009, 14:40
DETAIL: We are currently experiencing a problem accessing our filers
RESPONSE: We will need to reset the service, this should not take much time
UPDATED 14:47: The restart was successful.
Tuesday, November 24 2009
By Eirik Pettersen on Tuesday, November 24 2009, 22:55
DETAIL: We are currently experiencing another a problem with our internet connectivity.
RESPONSE: We are working with our provider to restore services as soon as possible.
UPDATED 22:24: Our network provider has confirmed the problem and are working to restore service.
UPDATED 22:52: Services appear to have been restored.
UPDATED 10:03 30 Nov: Incident report from Telstra Added
Monday, November 23 2009
By Walt on Monday, November 23 2009, 13:11
DETAIL: We are currently experiencing a problem with our internet connectivity.
RESPONSE: We are working with our provider to restore services as soon as possible.
UPDATED 13:25: Full service has been restored. We are now in contact with our provider to find out the cause of the outage.
Wednesday, October 28 2009
By Walt on Wednesday, October 28 2009, 17:42
REASON: Network distribution maintenance
PLANNED DURATION: 3 days (limited to a few hours each morning)
NOTES: Our domain partner Gandi will be performing major upgrades to their network distribution infrastructure during the nights of 28 to 30 October. During this time email services will experience a number of intermittent disruptions in connectivity beginning at 12:00am (GMT) with maintenance activities finishing by 6:00am (GMT) each day. We will endeavour to minimise impact to services where possible, but given the scale of the upgrades being performed, most services will encounter at least some disruption. No mail will be lost but access may be restricted and delivery of mail delayed.
We apologise in advance for any inconvenience this work may cause.
Saturday, October 17 2009
By Eirik Pettersen on Saturday, October 17 2009, 15:41
DETAIL: We are currently subject to a distributed denial of service attack, which is causing response times to be slow.
RESPONSE: We are investigating the source of the attack and expect to have this resolved promptly.
UPDATED 16:12: We have identified the signature of the attack and have put a block in place. Service should be back to normal.
Thursday, October 15 2009
By Walt on Thursday, October 15 2009, 15:37
DETAIL: We are currently experiencing some unexpected problems with our service delivery.
RESPONSE: We are investigating this and expect to have this resolved promptly.
UPDATED 16:20: We were performing a routine maintenance operation to expand disk space. This normally requires no downtime, however the operation hung causing a cascading problem which required us to restart the entire service. Because the restart was unscheduled we took great care to minimise any associated risks. We apologise for any inconvenience caused.
Wednesday, October 7 2009
By Walt on Wednesday, October 7 2009, 18:35
REASON: Essential Database Maintenance
PLANNED DURATION: We estimate no more than 1 hour
UPDATED 09:12: We restarted the database this morning as scheduled at 08:00. This was on the recommendation of our database vendor's technical support team to clear an urgent but non-critical issue. Although we expected this to operation to take no more than a few minutes of downtime, it has taken us a little longer. We apologise for the late notification and any inconvenience caused.
Tuesday, September 22 2009
By Josh on Tuesday, September 22 2009, 13:29
REASON: Data cleansing and archiving
PLANNED DURATION: 2 hours
NOTES: To improve the efficiency of our database, we plan to archive legacy data. This operation needs to be done off-line, that is all websites must be taken off-line for this period. Customers will be shown a page advising them that their site is unavailable and will be back up shortly.
We apologise for any inconvenience.
Friday, August 14 2009
By Josh on Friday, August 14 2009, 16:33
REASON: Database maintenance
PLANNED DURATION: 1 hour each (total of 3 hours)
NOTES: We will be performing some routine maintenance on the database which should result in improved overall performance of sites. The changes have been split into three different time slots during our periods of low activity in order to reduce the impact on customers.
UPDATED 08:39 BST 21 Aug: Maintenance completed successfully and service fully restored.
UPDATED 08:32 BST 24 Aug: Maintenance completed successfully and service fully restored.
UPDATED 08:15 BST 26 Aug: Maintenance completed successfully and service fully restored.
Thursday, July 2 2009
By Walt on Thursday, July 2 2009, 09:46
DETAIL: We are experiencing a distributed denial of service (DDoS) attack on the services, which is currently overloading our firewalls.
RESPONSE: Our technical team are working hard to get this attack under control and return full service to all customers. We hope to have this completely resolved shortly.
UPDATED 12:32: We've managed to block what appears to have been three different types of DDoS attack. Combating the three attacks has taken us longer than expected but we appear to have got them under control as of 11:35 but continued to experience minor issues until 12:32.
Thursday, June 25 2009
By Eirik Pettersen on Thursday, June 25 2009, 07:49
DETAIL: We are experiencing a large DDoS on one of our hosted sites, which is currently overloading our firewalls.
RESPONSE: Our technical team are working to resolve this issue as quickly as possible and should have it resolved shortly.
UPDATED 11:42: We are being flooded by an attack on one of our sites with 75% of all incoming traffic from over 2,300 IPs, which is causing all traffic to be congested and page load times to be affected. We are still currently investigating how to dynamically block this content.
Monday, May 11 2009
By Eirik Pettersen on Monday, May 11 2009, 11:04
REASON: Database upgrade and updates
PLANNED DURATION: Up to 60 minutes
NOTES: We will be upgrading the database software to the latest version, and performing a number of data updates in preparation for the release of the next version of SiteMaker.
Thank you for your patience and apologies for any inconvenience caused.
Thursday, April 9 2009
By Walt on Thursday, April 9 2009, 11:30
REASON: Our investigations into the recent service disruptions indicated that by running consistency checks on the affected data, the file system was stabilised. To be certain that the data is secure we will be running consistency checks on the remaining data in a controlled manner.
PLANNED DURATION: 40 minutes per site
NOTES: Those communities not already affected by the recent disruptions will go Read-Only for about 40 minutes. This will only affect file uploads, new site building, forum posts and saving of data. All sites will remain on view to visitors and owners. There will be six periods of 40 minutes during which 10% of sites will be affected.
UPDATED: This has now been completed. The consistency checks have been successful.
Tuesday, April 7 2009
By Walt on Tuesday, April 7 2009, 23:44
DETAIL: On April 7 a section of our system went read-only. This currently affects only 10% of our customers and has an impact on file uploads, new site building, form submissions and forum posts. The other 90% of our customers are completely unaffected by this event.
RESPONSE: Our technical team are working to resolve this issue as quickly as possible and should have it resolved shortly.
UPDATED 01:45: All affected systems have now been fully restored. We will continue to monitor the situation and post more details later in the morning.
Monday, April 6 2009
By Walt on Monday, April 6 2009, 09:56
DETAIL: A section of our system went read-only over the weekend. This has affected 20% of our customers and has an impact on file uploads, new site building, form submissions and forum posts. The other 80% of our customers are completely unaffected.
RESPONSE: Our technical team are working to resolve this issue as quickly as possible.
UPDATED 09:56 4 Apr: We are working on repairing the file system for the affected customers. Current estimate for completion is 2 - 4 hours.
UPDATED 13:40: We will be taking the service off-line for 10 minutes at approximately 13:55 to reconfigure the file system as part of our recovery actions.
UPDATED 14:04: The file system reconfiguration was successful and enabled us to restore full service to half of the affected customers. We expect to restore the service fully to the remaining 10 % of customers (the remaining half affected by this issue) within the next couple of hours.
UPDATED 15:02: Full service has now been restored to all customers.
Tuesday, March 10 2009
By Walt on Tuesday, March 10 2009, 13:02
DETAIL: This morning we suffered a distributed denial of service attack on one of our sites. This affected the speed at which others sites could be accessed which in some cases resulted in time-outs for some users.
RESPONSE: Engineers identified the problem early, However time was needed to establish how the attack was being conducted in order to block it.
UPDATED 10:01: The attack has been blocked and services should have returned to normal.
« previous entries - page 1 of 3