All systems operational
We will try to refresh every 5 minutes Operational Operational Operational Operational Operational Operational Operational

Incident history

✓ Resolved after 47h 20m of downtime disrupted for planned maintenance

June 12, 2019 at 3:00 PM

Maintenance complete: The planned maintenance is complete. Write access should become available once your DNS cache is flushed. The TTL is 1 hour. (Jun 12, 2019 - 15:34)

Planned maintenance: is currently read-only as we complete planned maintenance on the service. (Jun 12, 2019 - 15:00)

✓ Resolved after 216h 0m of downtime

Planned maintenance on

May 01, 2019 at 12:00 PM

Upcoming planned maintenance: We have an upcoming planned database outage on May 10th. This will cause a temporary service disruption for all services. The outage is expected to last less than 2 hours. (May 10, 2019 - 12:00)

✓ Resolved

Planned maintenance on

February 18, 2019 at 12:00 PM

Maintenance complete: Everything appears to be in good working order. (Feb 18, 2019 - 20:00)

Upcoming planned maintenance: We’re doing planned maintenance on Monday that will cause sporadic outages throughout the day as our hosts are moved into a larger rack. In exchange for your patience, we’ll be installing a much more powerful host for builds at the same time. (Feb 18, 2019 - 12:00)

✓ Resolved after 1h 31m of downtime outage

February 13, 2019 at 3:11 PM

Full service restored DNS udpates should be well and propegated by now, and the host is confirmed to be in working order. Apologies for the disruption. (Feb 13, 2019 - 16:42)

Rolling back migration There are other problems with the new host - DNS is being rolled back to the old host and another attempt to migrate will be taken at a later date. If you updated your /etc/hosts entry, please reset it to the old host:

DNS error: The DNS records were incorrectly updated to the wrong IP address, resulting in an outage which will last for the duration of the TTL (approximately 1 hour). A workaround is to add to your /etc/hosts file: the correct IP address is Please set a reminder to remove this entry, as future DNS updates may cause local outages for you if you do not. (Feb 13, 2019 - 15:12)

✓ Resolved after 2h 0m of downtime

Planned login outage

December 29, 2018 at 1:00 AM

Maintenance complete: We have completed this period of maintenance, has been restored to full service. (Dec 29, 2018 - 03:00)

Partial outage during planned maintenance: A planned service upgrade requires logins to be intermittedly for some services. Users who are already logged in should not experience a disruption, though minor issues here and there may occur. (Dec 29, 2018 - 01:00)

✓ Resolved after 1h 3m of downtime

Service disruption

November 24, 2018 at 7:39 PM

Debian & FreeBSD builds now available: Full service has been restored. (Nov 27, 2018 - 03:54)

Arch Linux builds now available: Arch is available and Debian is coming. (Nov 25, 2018 - 18:01)

Alpine builds now available: The next step is to start pulling build images down to the new build runner. Currently Alpine is available and Arch is being worked on next. (Nov 25, 2018 - 17:01)

Restoring builds service: Work is ongoing to provision a new build host. Service is expected to be restored within a few hours. (Nov 25, 2018 - 15:23)

Temporary database server provisioned: Full service has been restored to all services except for, which requires the provisioning of a new build slave. Until then no builds will be run. (Nov 24, 2018 - 22:34)

Replacing the database server: The database server has been taken out of service for maintenance, and in the meantime I’m provisioning a replacement. I believe that there has been no data loss. (Nov 24, 2018 - 22:06)

Reopened: Failures are back up. Investigating. (Nov 24, 2018 - 21:03)

Resolved: The issue has been resolved, and we’re monitoring the system for more information. (Nov 24, 2018 - 20:42)

Investigating: The same database server causing issues yesterday is causing more problems today. Investigating. (Nov 24, 2018 - 19:40)

✓ Resolved after 5h 27m of downtime

Database RAID failure

November 23, 2018 at 6:00 PM

Service restored: The database server has been returned to service. (Nov 23, 2018 - 23:27)

Restoring service: The RAID rebuild completed successfully and the server is on its way back to the datacenter to be entered back into service. (Nov 23, 2018 - 22:25)

Resolving: An issue with one of the hard drives on one of our database servers was discovered, resulting in a total outage for this server. The RAID array is being rebuilt and service is expected to be restored within a few hours. (Nov 23, 2018 - 20:00)

Investigating: One of our database servers has gone offline and is unresponsive. The issue will require manual intervention at the datacenter, but the NOC is slow to respond during the holiday weekend. Once we gain access to the server, an update will be posted. (Nov 23, 2018 - 18:00)