Builds.sr.ht on sr.ht status

SourceHut disrupted due to DDoS attack

Sun, 05 Apr 2026 10:00:00 +0000

SourceHut was disrupted by a DDoS attack.

Over the holidays SourceHut experienced a DDoS attack primarily targetting git.sr.ht. For a period of several hours we experienced severe performance issues and intermittent outages as a result, primarily affecting git.sr.ht.

Our upstream network provider identified the malicious traffic and blackholed the appropriate routes, but due to a configuration error SourceHut’s AMS subnet was entirely null routed for a period of about one hour yesterday, causing a complete outage for all services. Once the configuration error was corrected, the DDoS continued to cause performance issues that primarily affected git.sr.ht, until the attack died down later yesterday afternoon.

The approximately 1-hour total outage of our AMS installation caused a severe network partition that disrupted systems hosted in other installations, notably our build workers. These issues were not understood until this morning, when the ensuing problems were corrected. However, builds submitted in the meanwhile were cancelled, and users will have to resubmit any builds from this period that still need to be run. You can do this by opening the affected build on the builds.sr.ht web UI and clicking “Resubmit”.

Additionally, there have been some issues updating our status page, which resulted in poor communication during the incident. An old outage notice was never properly closed, and during the outage updating the status page was a challenge – and a distraction. We elected to defer the solution until today to allow our engineers to enjoy what remains of their holiday weekends. The status page issues have now been resolved.

We apologize for the inconvenience.

(10:00 UTC — Apr 6)

Attacks have resumed.

DDoS attacks targetting git.sr.ht have resumed, disrupting performance and availability. We are looking into mitigations.

(15:30 UTC — Apr 6)

Attacks are ongoing.

We continue to experience a DDoS attack primarily targetting git.sr.ht. We are continuing to explore mitigations but in the meantime you can expect continued degraded performance on git.sr.ht, particularly with SSH operations.

(10:30 UTC — Apr 7)

Attacks now targeting builds.sr.ht.

After a brief pause the attack is now targeting builds.sr.ht. We are once more working on mitigations.

(15:30 UTC — Apr 8)

[Resolved] Planned maintenance on all services

Wed, 22 Oct 2025 11:00:00 +0000

The maintenance is complete. All services are operational again. Note that some chat.sr.ht upstream connections may take a while to be re-established due to rate limits.

(12:25 UTC — Oct 22)

The maintenance window is now open. Disruptions will occur across all services.

(11:00 UTC — Oct 22)

Planned maintenance on October 22nd will cause intermittent outages: Planned maintenance starting at 11:00 UTC will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes each. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take up to a few hours.

(10:00 UTC — Oct 16)

[Resolved] Planned maintenance on all services

Tue, 22 Jul 2025 09:00:00 +0000

The maintenance is complete. All services are restored.

(10:15 UTC — Jul 22)

The maintenance window is now open. Disruptions will occur across all services.

(09:00 UTC — Jul 22)

Planned maintenance on July 22nd will cause intermittent outages: Planned maintenance starting at 09:00 UTC will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes each. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take up to a few hours.

(07:30 UTC — Jul 2)

[Resolved] Planned maintenance on Tuesday

Tue, 11 Feb 2025 10:00:00 +0000

Maintenance completed. Access to all services is restored. (11:40 UTC — Feb 11)

Maintenance will take longer as expected. Due to unforeseen circumstances, the maintenance will take longer than anticipated. We are working to resolve the issue as quickly as possible. (11:00 UTC — Feb 11)

Maintenance is underway. The maintenance window is now open. (10:00 UTC — Feb 11)

Planned maintenance on February 11th will cause intermittent outages: Planned maintenance starting at 10AM UTC will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take no more than an hour.

Have a nice weekend! (10:50 UTC — Feb 6)

[Resolved] Planned maintenance on Monday

Mon, 25 Nov 2024 13:00:00 +0000

Maintenance work is completed. All services should be operating normally. Please let us know if you notice any lingering problems. (13:26 UTC — Nov 25)

The maintenance window is now open. (13:00 UTC — Nov 25)

Maintenance window rescheduled for 13:00 UTC. We have advanced the schedule by one hour to 13:00 UTC. (08:00 UTC — Nov 25)

Planned maintenance is scheduled for Monday, November 25th. On Monday at 13:00 UTC we will be rolling out some changes which will cause spurious errors and issues across services for the duration of the roll-out. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take no more than an hour.

Have a nice weekend! (14:00 UTC — Nov 22)

[Resolved] Upstream network outage

Wed, 30 Oct 2024 11:30:00 +0000

Upstream network issues appear resolved. We still know little of the cause but we know it broadly affected numerous providers in our service area. (14:00 UTC — Oct 30)

Upstream network issues are causing intermittent availability problems for SourceHut. An issue with our upstream network provider has affected user access to SourceHut. We do not have any further details or an ETA on a solution at this time. We expect the network to be intermittent until the upstream issue is resolved. (11:30 UTC — Oct 30)

[Resolved] Planned outage for August 5th

Mon, 05 Aug 2024 11:00:00 +0000

All planned maintenance has been completed. The maintenance window is now closed. Please notify support if you encounter any unexpected issues.

The maintenance window is now open. We will be opening our maintenance window in the next few minutes and begin rolling out upgrades.

A planned maintenance window is scheduled for August 5th. The maintenance window originally scheduled for August 1st has been rescheduled for Monday, August 5th at 11:00 UTC. This is expected to cause intermittent partial outages between 15 and 30 minutes affecting all services. Follow this page or #sr.ht on IRC for updates.

[Resolved] Planned outage for August 1st

Thu, 01 Aug 2024 11:00:00 +0000

The maintenance window has been cancelled. A new maintenance window will be planned in the next few days.

A planned maintenance window is scheduled for August 1st. At 11:00 UTC we will be rolling out upgrades across our fleet. This is expected to cause intermittent partial outages between 15 and 30 minutes affecting all services. Follow this page or #sr.ht on IRC for updates.

[Resolved] Build queues are backed up

Mon, 27 May 2024 08:00:00 +0000

Build queues are full and wait times are expected. Due to a minor issue with our build servers, build queues were unprocessed for a couple of hours this morning. The system is catching up with pending builds now, but wait times may be higher than normal for the next hour or so. (08:00 UTC — May 27)

[Resolved] Planned maintenance will require service interruptions

Wed, 20 Mar 2024 12:00:00 +0000

Planned maintenance is complete. We have completed all planned upgrades and restored all services to normal working order. Please let us know if you notice any lingering issues. Thank you for your patience. (15:15 UTC — Mar 20)

The maintenance window is open. Planned maintenance is now underway. (12:00 UTC — Mar 20)

Planned maintenance this Wednesday will require service interruptions. We are planning upgrades across our fleet on Wednesday, March 20th, which will require service interruptions to complete. Some services are expected to be briefly unavailable for up to 20 minutes each across the duration of the maintenance window. (16:00 UTC — Mar 18)

[Resolved] Network outage

Wed, 10 Jan 2024 08:50:00 +0000

Update 2024-01-17 10:30 UTC: full service has been restored.

Update 2024-01-16 19:45 UTC: builds.sr.ht is now running at full capacity.

Update 2024-01-16 18:00 UTC: builds.sr.ht is now available for full service. We are running it at half of the planned capacity, with the remainder coming online shortly, so you may experience longer queue times while waiting for builds to be processed.

Update 2024-01-16 10:30 UTC: Our priority for today is to bring builds.sr.ht and chat.sr.ht online, as well as address a number of less user-impacting tasks we have on our todo list.

Update 2024-01-15 22:15 UTC: Object storage for git.sr.ht (e.g. releases attached to git tags) is now available for read and write operations.

Update 2024-01-15 15:50 UTC: pages.sr.ht is now available for full service, albeit with degraded performance for publish operations.

Update 2024-01-15 08:20 UTC: hg.sr.ht is now available for full service. We are working on builds availability today.

Update 2024-01-15 07:45 UTC: We got in touch with hg.sr.ht’s community maintainer and put the finishing touches on it. It is now available in read-only mode.

Update 2024-01-14 14:45 UTC: The mail system is coming back online, and with it lists and todo are entering full service. We do not believe any emails were lost, queues should process normally and catch up on emails sent during outage intervals.

For users with questions about billing: the billing system has been shut off for the duration of the outage. Before we turn it on again, we are going to credit all paying users with free service for the duration of the outage period.

Update 2024-01-14 14:00 UTC: We have have restored git.sr.ht and man.sr.ht to full service.

Note that we are still working with our existing transit provider and are experimenting with a new solution for mitigating the DDoS, but we are not certain that this approach is reliable and we are still working on setting up a more permanent transit solution in the background.

Update 2024-01-14 08:50 UTC: We have partially restored service once again, and are working on restoring more services. lists and todo are operational from the web, but we have disabled the mail system for the time being.

Update 2024-01-13 14:17 UTC: We will need to find a new transit provider to mitigate the problem. We are in talks with a provider to address the problem and we will begin the engineering work shortly. Further updates to come as we have them. Thank you again for your patience.

Update 2024-01-13 13:00 UTC: Our new transit provider has notified us that the DDoS has followed us to our new network. They have deployed mitigations and we do not expect an interruption in service. Update: service is impacted, we are investigating further.

Note that our new transit solution utilizes end-to-end encryption such that traffic between you and SourceHut is received and processed directly by our colocated servers and is not handled in plaintext by third-parties.

Update 2024-01-13 12:40 UTC: Mail service to lists and todo has been restored, subject to DNS propagation delays. Emails queued during the outage should resume processing now – we do not believe any emails have been lost.

Update 2024-01-13 11:42 UTC: We have brought pages.sr.ht into service for read-only operations. All custom domains should be working soon, subject to DNS propagation delays, with the exception of custom domains using apex records (i.e. top level domains such as example.org rather than subdomain.example.org). Manual intervention is required for affected users.

We have established a temporary IP address for serving custom domains using apex records. Users can change their apex record to the following IP address to restore read-only pages service:

@   IN  A   141.95.4.185

Note that we may change this IP address in the future. You will be notified by email if later changes are required for your domain.

Update 2024-01-13 11:00 UTC: We have enabled read/write access to most services. hg is still read-only, and todo and lists process requests from the web but the mail system is still a work-in-progress.

Update 2024-01-13 09:26 UTC: Good morning. We have brought hg.sr.ht up in read-only mode and are working towards enabling read/write for available services today. We have also finished importing the diff from backup for git.sr.hg, and the diff for hg should finish soon.

Update 2024-01-12 20:49 UTC: We are now putting the finishing touches on our goals for today. We have 7 primary services up and running in read-only mode. An earlier issue with git clone/fetch was also fixed; this should be working again.

We have planned a lighter workload for this weekend; we need the rest. Our goal is to have hg.sr.ht online in read-only mode. We will try to get read/write service partially restored for these 7 services, plus hg, this weekend. chat.sr.ht, pages.sr.ht, and builds.sr.ht have special considerations for resumption of service which we will be planning for early next week.

Thank you for your patience, and have a good night.

Update 2024-01-12 16:38 UTC: todo and lists are coming online in read-only mode.

Update 2024-01-12 16:13 UTC: We are beginning to bring some services online. meta and git will become available as DNS propegates to your local nameserver.

My name is Drew, I’m the founder of SourceHut and one of three SourceHut staff members working on the outage, alongside my colleagues Simon and Conrad. As you have noticed, SourceHut is down. I offer my deepest apologies for this situation. We have made a name for ourselves for reliability, and this is the most severe and prolonged outage we have ever faced. We spend a lot of time planning to make sure this does not happen, and we failed. We have all hands on deck working the problem to restore service as soon as possible.

In our emergency planning models, we have procedures in place for many kinds of eventualities. What has happened this week is essentially our worst-case scenario: “what if the primary datacenter just disappeared tomorrow?” We ask this question of ourselves seriously, and make serious plans for what we’d do if this were to pass, and we are executing those plans now – though we had hoped that we would never have to.

I humbly ask for your patience and support as we deal with a very difficult situation, and, again, I offer my deepest apologies that this situation has come to pass.

What is happening?

At 06:30 UTC on January 10th, two days prior to the time of writing, a distributed denial of service attack (DDoS) began targetting SourceHut. We still do not know many details – we don’t know who they are or why they are targetting us, but we do know that they are targetting SourceHut specifically.

We deal with ordinary DDoS attacks in the normal course of operations, and we are generally able to mitigate them on our end. However, this is not an ordinary DDoS attack; the attacker posesses considerable resources and is operating at a scale beyond that which we have the means to mitigate ourselves. In response, before we could do much ourselves to understand or mitigate the problem, our upstream network provider null routed SourceHut entirely, rendering both the internet at large, and SourceHut staff, unable to reach our servers.

The primary datacenter, PHL, was affected by this problem. We rent colocation space from our PHL supplier, where we have our own servers installed. We purchase networking through our provider, who allocates us a block out of their AS, and who upstreams with Cogent, which is the upstream that ultimately black holed us. Unfortunately, our colocation provider went through two acquisitions in the past year, and we failed to notice that our account had been forgotten as they migrated between ticketing systems through one of these acquisitions. Thus unable to page them, we were initially forced to wait until their normal office hours began to contact them, 7 hours after the start of the incident.

When we did get them on the phone, our access to support ticketing was restored, they apologised profusely for the mistake, and we were able to work with them on restoring service and addressing the problems we were facing. This led to SourceHut’s availability being partially restored on the evening of January 10th, until the DDoS escalated in the early hours of January 11th, after which point our provider was forced to null route us again.

We have seen some collateral damage as well. You may have noticed that Hacker News was down on January 10th; we believe that was ultimately due to Cogent’s heavy handed approach to mitigating the DDoS targetting SourceHut (sorry, HN, glad you got it sorted). Last night, a non-profit free software forge, Codeberg, also became subject to a DDoS, which is still ongoing and may be caused by the same actors. This caused our status page to go offline – Codeberg has been kind enough to host it for us so that it’s reachable during an outage – we’re not sure if Codeberg was targetted because they hosted our status page or if this is part of a broader attack on free software forge platforms.

What are we doing about it?

We maintain three sites, PHL, FRE, and AMS. PHL is our primary and is offline, FRE is our backup site, and AMS is a research installation we eventually hoped to use to migrate our platform to European hosting. As we initially had no access whatsoever to PHL, we began restoring from backups to AMS to set up a parallel installation of SourceHut from scratch.

We have since received some assistance from our PHL provider in regaining access to our PHL servers out of band, which is speeding up affairs, but we do not expect to get PHL online soon and we are proceeding with the AMS installation for now.

The prognosis on user data loss is good. Our backups are working and regularly tested, the last full backup of git and hg was taken a few hours before the DDoS began, and we have out-of-band access to the live PHL servers where all changes which occured since the most recent backup are safely preserved. The database is replicated in real-time and was only seconds behind production before it went offline.

We have replicated the production database in AMS and started spinning up SourceHut services there: we have meta, todo, lists, paste, and the project hub fully operational against production data in our staging environment here. We are still working on the following services in order of priority:

git.sr.ht
hg.sr.ht
pages.sr.ht
chat.sr.ht
man.sr.ht
builds.sr.ht

These services, particularly git and hg, require large transfers of data across our networks to restore from backups, and will take some time. Chat does not require particularly large amounts of data to be managed, but has special networking concerns that we are addressing as well.

Our goal is to enable read-only access for the community as quickly as possible, then work on full read/write access following that. Object storage (used for git/hg releases, build artifacts, and SourceHut pages) presents a special set of problems; we are working on those separately. Finding suitable compute to run build jobs is another issue which requires special attention, but we have a plan for this as well.

One of our main concerns right now is finding a way of getting back online on a new network without the DDoS immediately following us there, and we have reason to believe that it will. A layer 3 DDoS like the one we are facing is complex and expensive to mitigate. We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means, but we are investigating other solutions which may be more affordable and have a few avenues for research today, though we cannot disclose too many details without risking alerting the attackers to our plans.

How you can help

What we need the most right now is your patience and understanding. Mitigating this sort of attack is a marathon, not a sprint, and we have to be careful not to overwork our staff, ensure we’re getting enough sleep, and so on – we are working as hard as we can. There are many people hard at work on this problem for you – I’d like to thank Simon and Conrad in particular for their work, as well as the datacenter and network operators upstream of us who are doing their best as well.

You can receive updates on this page, so long as we’re able to keep it online (low priority), as well as on Mastodon, where we are posting updates as well. This is also a good place to share your words of support and encouragement, as well as the #sr.ht IRC channel on Libera Chat. My inbox at sir@cmpwn.com is also working (not without some effort, I’ll add), if you wish to send your support or offer any resources that might help.

Thank you for your patience and support. We are working to make things right with you.

Note: at this point our status page went offline. We prepared a temporary status page; the text above has been imported from this status page.

The migration is proceeding. We are making good progress on the migration. We have a staging environment with production data (with no loss of user data) working with for the following services:

meta.sr.ht
sr.ht (project hub)
lists.sr.ht
todo.sr.ht
paste.sr.ht

We will strive to have these services available to you soon. We are still working on the following services:

git.sr.ht
hg.sr.ht
chat.sr.ht
pages.sr.ht
builds.sr.ht
man.sr.ht

These are currently being restored from a backup taken several hours prior the start of this incident, two days ago, which will require several more hours to complete. Following this, we have a procedure planned which will restore any data changed in the two-day window since this backup was taken.

We are prioritizing the following services, in order:

git repositories
Mercurial repositories
Object storage – git/hg releases, build artifacts, and SourceHut pages
chat.sr.ht logs
builds.sr.ht logs

The object storage may take as much as a few days to restore to full service; however, we may have read-only service working sooner.

We do not anticipate that we will have builds fully operational until we provision more compute to handle user builds, which may take another day or two.

Thank you for your patience. (18:49 UTC — Jan 11)

PHL service unreliable. The network to our PHL installation is unreliable. When available, it is configured in a read-only mode. We are still proceeding with the migration for the time being. (13:43 UTC — Jan 11)

PHL network restored. Cogent removed the black hole a few minutes ago, restoring access to our PHL installation. However, we are partway through our migration and have placed PHL into a read-only configuration for the time being. Further updates to come. (13:19 UTC — Jan 11)

Migration is in progress. We are partway through setting up a new installation in one of our secondary datacenters. We do not have an ETA but work is proceeding apace. (11:32 UTC — Jan 11)

We are preparing a migration. We do not see a near-term resolution being possible with our upstream network provider and we are preparing to restore service from backups in a new installation. It will take a while, but we are all-hands to address the issue. (08:48 UTC — Jan 11)

Cogent strikes again. They broke it again this morning. (07:00 UTC — Jan 11)

Service is partially restored. All services save for the project hub are now operational. Work proceeds to finish restoring all services. (18:23 UTC — Jan 10)

Root cause identified, mitigations underway. We have received another update from our NOC. Their upstream provider black holed our ASN to mitigate a DDoS coming into our network by hijacking our BGP routes to point to their black hole ASN. Untangling this requires some work, but we should be coming partially online as the network is reconfigured shortly. Full restoration of service may require some time still. (16:17 UTC — Jan 10)

Investigation still ongoing. We just received an update from the NOC. They’re working the issue but there is no ETA on the resolution. “The team is working on the problem as our sole priority”. (15:04 UTC — Jan 10)

NOC is investigating the issue. We got in touch with network operations and the issue is under investigation. (14:11 UTC — Jan 10)

We are unable to reach our emergency NOC. We have been attempting to reach the emergency datacenter operations contacts for the past hour without success; there is an issue with their emergency ticketing system. We will have to wait until their non-emergency line is available, which should be at 9 AM EST (ETA 4 hours). (09:56 UTC — Jan 10)

Issues with our upstream network provider is causing intermittent outages. We are having an issue with our upstream network provider which is causing intermittent outages. We are investigating the issue. (08:50 UTC — Jan 10)

[Resolved] Ongoing DDoS

Thu, 20 Apr 2023 18:10:00 +0000

The services are online and working. We have been monitoring the mitigations we put in place last night and, though we’re still tying off some loose ends, all services should be available for users. (07:00 UTC — Apr 21)

A distributed denial of service (DDoS) attack on sr.ht is underway. We have identified the source of the networking issues as a denial of service attack and we are deploying a mitigation. (19:10 UTC — Apr 20)

Networking issues caused an unplanned outage. Some networking issues are causing degraded availability for all our services. (18:10 UTC — Apr 20)

[Resolved] Planned outage

Wed, 22 Mar 2023 14:00:00 +0000

Planned maintenance is complete. Normal service is restored. Thank you for your patience. (16:00 UTC — Mar 22)

Maintenance is underway The maintenance window is now open. (14:00 UTC — Mar 22)

A planned outage is scheduled for March 22th. At 14:00 UTC, we plan to perform system upgrades across our fleet. This will cause intermittent outages which are expected to last 15-20 minutes each, over the course of 3 to 4 hours in total.

[Resolved] Unreachable database host

Wed, 11 Jan 2023 13:38:00 +0000

Service restored. We have migrated the database to a temporary host and brought our services back online. We had to use one of our build servers to host it for the time being, so our build system is operating at half its normal capacity – you may notice longer-than-usual build times while we roll out a permanent solution.

Thank you for your patience. (17:00 UTC — Jan 11)

Investigating. We have a technician on-site and are diagnosing the issue further. (15:47 UTC — Jan 11)

Unreachable host causing disruption of multiple services. Our database host has suddenly become unreachable, causing all services requiring it to become unavailable. We are currently working with people on location to determine the exact issue. (14:00 UTC — Jan 11)

[Resolved] Planned outage for high-risk deployment

Wed, 26 Oct 2022 09:00:00 +0000

Service has been restored. Webhooks are now being processed and the backlog has been completed. (00:00 UTC — Oct 27)

Service has mostly been restored. We are still running some migrations which must complete before we re-enable webhooks, but most services are functional. (13:00 UTC — Oct 26)

Maintenance is underway. The maintenance window is now open. (09:00 UTC — Oct 26)

This maintenance window has been rescheduled. It will now take place on Wednesday, October 26th, at 09:00 UTC.

An upcoming software deployment will require a planned outage to complete. On Wednesday, October 26th, we are rolling out high-risk software changes. To mitigate this risk, the services will be shut down and a snapshot taken, then brought back online one at a time as we apply the migrations. We anticipate that this process will take between 2 and 3 hours, during which time intermittent service availability is expected. (09:00 UTC — Oct 18)

[Resolved] Planned maintenance will cause major outages

Mon, 28 Feb 2022 15:00:00 +0000

Planned maintenance is complete. Normal service is restored. Thank you for your patience. (16:30 UTC — Feb 28)

Planned maintenance is underway. All SourceHut services will be affected. Barring unforseen circumstances, the maintenance window is scheduled for two hours. (15:00 UTC — Feb 28)

Phase three of our Alpine Linux 3.15 roll-out will cause major service disruptions. We will be rolling out Alpine Linux 3.15 to more of our hosts on Monday the 28th at 15:00 UTC.

This upgrade is expected to cause a full outage of all services throughout the two-hour maintenance window. (11:00 UTC — Feb 14)

[Resolved] Planned maintenance will cause major outages

Mon, 21 Feb 2022 15:00:00 +0000

Maintenance complete. We have completed all maintenance operations planned for today and service is restored to normal. (17:15 UTC — Feb 21)

Planned maintenance may exceed the scheduled maintenance window. We are still working on the roll-out and now expect to complete the migration before 18:00 UTC. (17:00 UTC — Feb 21)

Planned maintenance is underway. Expect intermittent outages for all services. (15:00 UTC — Feb 21)

Phase two of our Alpine Linux 3.15 roll-out will cause major service disruptions. We will be rolling out Alpine Linux 3.15 to more of our hosts on Monday the 21st at 15:00 UTC. This upgrade is expected to cause major service disruptions lasting between 5 and 15 minutes each; generally one service at a time will be affected.

The maintenance window is two hours. Following the completion of this maintenance, we will announce the window for phase three of the roll-out. (11:00 UTC — Feb 14)

[Resolved] Planned maintenance will cause minor outages

Mon, 14 Feb 2022 10:00:00 +0000

Today’s scheduled maintenance is complete. We have completed our maintenance work for today, and all services are restored to normal operation. Thank you for your patience. (10:45 UTC — Feb 14)

The migration is underway. We have started this migration work and expect to complete it no later than 12:00 UTC. (10:00 UTC — Feb 14)

Phase one of our Alpine Linux 3.15 roll-out will cause minor service disruptions. We will be rolling out Alpine Linux 3.15 to some of hosts on Monday the 14th at 10:00 UTC. This will disrupt blob storage (causing artifacts on builds.sr.ht and git.sr.ht to become unavailable) and will temporarily reduce build throughput (possibly causing longer waits for builds to start executing). Additionally, metrics.sr.ht will be briefly unavailable.

The maintenance window is two hours. Following the completion of this maintenance, we will announce the window for phase two of the roll-out, which will be more disruptive. (10:00 UTC — Feb 10)

[Resolved] Planned outage for all services

Mon, 16 Aug 2021 13:00:00 +0000

Planned maintenance is complete: We have finished upgrading all of the affected hosts and all services are restored to normal. (14:20 UTC — Aug 16)

Planned maintenance is ongoing: Our maintenance window has opened and we have started our work. (13:00 UTC — Aug 16)

Planned maintenance on August 16th will cause intermittent outages: We are planning the second (and last phase) of the maintenance which began on August 3rd. This will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. The total maintenance period should last less than 2 hours. (13:00 UTC — Aug 16)

[Resolved] Planned outage for all services

Tue, 03 Aug 2021 08:00:00 +0000

Maintenance complete: The issue with pages.sr.ht has been resolved and all services are now available. (10:30 UTC — Aug 3)

Maintenance mostly complete but pages.sr.ht still pending: We have completed maintenance on all services, which can now be expected to be stable, with the exception of pages.sr.ht. We have encountered an issue during the pages.sr.ht upgrades and are addressing it now. (10:00 UTC — Aug 3)

Planned maintenance on August 3rd will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. The total maintenance period should last less than 2 hours. (09:00 UTC — Aug 3)

[Resolved] Planned outage for all services

Mon, 08 Feb 2021 15:00:00 +0000

Planned maintenance has been completed. (16:30 UTC — Feb 8)

Planned maintenance is now underway. (15:00 UTC — Feb 8)

Planned maintenance on February 8th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. (15:00 UTC — Feb 8)

[Resolved] Network outage

Fri, 08 Jan 2021 09:03:00 +0000

An issue with our upstream ISP caused an outage. Our ISP decided to ring in the new year by unplugging us for 12 minutes last night. Sorry about that. (09:03 UTC — Jan 8)

[Resolved] Planned outage for all services

Mon, 15 Jun 2020 18:00:00 +0000

Planned maintenance is complete. (19:07 UTC — Jun 15)

Planned maintenance is underway. (18:00 UTC — Jun 15)

Planned maintenance on June 15th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. (18:00 UTC — Jun 15)

[Resolved] Planned outage for all services

Wed, 11 Mar 2020 16:00:00 +0000

Planned maintenance is complete. There may be some residual issues. (18:00 UTC — Mar 11)

Planned maintenance is underway (16:00 UTC — Mar 11)

Planned maintenance on March 11th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes. (19:00 UTC — Mar 5)

[Resolved] Planned outage for all services

Wed, 08 Jan 2020 16:00:00 +0000

Maintenance complete Service has been restored. (16:30 UTC — Jan 8)

Maintenance has begun: This maintenance is now underway. (16:00 UTC — Jan 8)

Planned maintenance on January 8th will cause a total service outage: We have scheduled maintenance for next Wednesday at 16:00 UTC to perform normal database maintenance. We expect a total outage of all services which should last no more than 2 hours. (16:00 UTC — Jan 8)

[Resolved] Intermittent outages during maintenance

Sat, 30 Nov 2019 15:00:00 +0000

Deployments complete: The deployment has been completed for all services and no issues have been found. Service should be operating normally. (15:45 UTC — Nov 30)

Ongoing deployment causing service disruption: We are currently deploying a major change in the Sourcehut architecture, which is likely to cause intermittent outages during deployment as services are rotated onto the new system and oversights are discovered and resolved. (15:00 UTC — Nov 30)

[Resolved] Major DNS outage

Sun, 29 Sep 2019 18:00:00 +0000

Resolved The issue has been fixed. It may take time for DNS updates to propegate to your local server. (21:45 UTC — Sep 29)

Working on the issue We have reached support and they’ve rolled back the changes. We’re monitoring our services to make sure the trasition occurs smoothly. (21:00 UTC — Sep 29)

No response We are still waiting for an update from our registrar. (20:00 UTC — Sep 29)

Major DNS outage: While deploying IPv6 support across Sourcehut, we submitted a routine request to our domain registrar to add IPv6 glue records for our nameservers. In the course of doing this, the registrar also removed the IPv4 glue records. We are attempting to reach them for assistance. Services are severely disrupted in the meantime. (10:00 UTC — Sep 29)

[Resolved] Planned maintenance on sr.ht

Wed, 01 May 2019 12:00:00 +0000

Upcoming planned maintenance: We have an upcoming planned database outage on May 10th. This will cause a temporary service disruption for all sr.ht services. The outage is expected to last less than 2 hours. (12:00 UTC — May 10)

[Resolved] Planned maintenance on sr.ht

Mon, 18 Feb 2019 12:00:00 +0000

Maintenance complete: Everything appears to be in good working order. (20:00 UTC — Feb 18)

Upcoming planned maintenance: We’re doing planned maintenance on Monday that will cause sporadic outages throughout the day as our hosts are moved into a larger rack. In exchange for your patience, we’ll be installing a much more powerful host for builds.sr.ht builds at the same time. (12:00 UTC — Feb 18)

[Resolved] builds.sr.ht outage

Wed, 13 Feb 2019 15:11:00 +0000

Full service restored DNS udpates should be well and propegated by now, and the host is confirmed to be in working order. Apologies for the disruption. (16:42 UTC — Feb 13)

Rolling back migration There are other problems with the new host - DNS is being rolled back to the old host and another attempt to migrate will be taken at a later date. If you updated your /etc/hosts entry, please reset it to the old host: 45.56.100.217.

DNS error: The builds.sr.ht DNS records were incorrectly updated to the wrong IP address, resulting in an outage which will last for the duration of the TTL (approximately 1 hour). A workaround is to add builds.sr.ht to your /etc/hosts file: the correct IP address is 173.195.146.148. Please set a reminder to remove this entry, as future DNS updates may cause local outages for you if you do not. (15:12 UTC — Feb 13)

[Resolved] Planned login outage

Sat, 29 Dec 2018 01:00:00 +0000

Maintenance complete: We have completed this period of maintenance, sr.ht has been restored to full service. (03:00 UTC — Dec 29)

Partial outage during planned maintenance: A planned service upgrade requires logins to be intermittedly for some sr.ht services. Users who are already logged in should not experience a disruption, though minor issues here and there may occur. (01:00 UTC — Dec 29)

[Resolved] Service disruption

Sat, 24 Nov 2018 19:39:00 +0000

Debian & FreeBSD builds now available: Full service has been restored. (03:54 UTC — Nov 27)

Arch Linux builds now available: Arch is available and Debian is coming. (18:01 UTC — Nov 25)

Alpine builds now available: The next step is to start pulling build images down to the new build runner. Currently Alpine is available and Arch is being worked on next. (17:01 UTC — Nov 25)

Restoring builds service: Work is ongoing to provision a new build host. Service is expected to be restored within a few hours. (15:23 UTC — Nov 25)

Temporary database server provisioned: Full service has been restored to all services except for builds.sr.ht, which requires the provisioning of a new build slave. Until then no builds will be run. (22:34 UTC — Nov 24)

Replacing the database server: The database server has been taken out of service for maintenance, and in the meantime I’m provisioning a replacement. I believe that there has been no data loss. (22:06 UTC — Nov 24)

Reopened: Failures are back up. Investigating. (21:03 UTC — Nov 24)

Resolved: The issue has been resolved, and we’re monitoring the system for more information. (20:42 UTC — Nov 24)

Investigating: The same database server causing issues yesterday is causing more problems today. Investigating. (19:40 UTC — Nov 24)

[Resolved] Database RAID failure

Fri, 23 Nov 2018 18:00:00 +0000

Service restored: The database server has been returned to service. (23:27 UTC — Nov 23)

Restoring service: The RAID rebuild completed successfully and the server is on its way back to the datacenter to be entered back into service. (22:25 UTC — Nov 23)

Resolving: An issue with one of the hard drives on one of our database servers was discovered, resulting in a total outage for this server. The RAID array is being rebuilt and service is expected to be restored within a few hours. (20:00 UTC — Nov 23)

Investigating: One of our database servers has gone offline and is unresponsive. The issue will require manual intervention at the datacenter, but the NOC is slow to respond during the holiday weekend. Once we gain access to the server, an update will be posted. (18:00 UTC — Nov 23)