tag:status.northflank.com,2005:/history Northflank Status - Incident history 2026-08-05T13:27:13.111+00:00 Northflank tag:status.northflank.com,2005:Incident/cmsg4fvtt034m1ao3nowjoker 2026-08-05T13:27:13.111+00:00 2026-08-05T13:27:13.111+00:00 Workloads Failing to Start Across Multiple Regions Type: Incident

Duration: 1 hour and 1 minute

Affected Components: Addons, Jobs, Services

Aug 5, 13:27:13 GMT+0
Investigating - We are seeing partial start up issues for workloads across multiple regions. We are currently investigating this incident..

Aug 5, 13:42:39 GMT+0
Monitoring - We have identified the issue and applied a mitigation. Previously stuck workloads are now starting successfully across affected regions. We are monitoring recovery and will confirm once all workloads are fully operational..

Aug 5, 14:27:54 GMT+0
Resolved - All affected workloads are now fully operational. .

]]> tag:status.northflank.com,2005:Incident/cmrw1sp1g00870smo8ztfcsfn 2026-07-22T12:17:46.396+00:00 2026-07-22T12:17:46.396+00:00 Investigating node stability issues in the US - East region Type: Incident

Duration: 1 hour and 22 minutes

Affected Components: Addons, Jobs, Services

Jul 22, 12:17:46 GMT+0
Investigating - We have noticed increased failure rates of nodes in the US - East region and are investigating the issue..

Jul 22, 13:11:06 GMT+0
Monitoring - We are seeing recovery across compute and storage. We are working to get the remaining workloads back online..

Jul 22, 13:39:58 GMT+0
Resolved - We have recovered affected workloads in the region and there are no more on-going node failures. The root cause was an outage on the Google Cloud Platform in us-east4 which hosts Northflank's US - East region..

]]> tag:status.northflank.com,2005:Incident/cmrujwsz307ud0kqm1nugymth 2026-07-21T11:09:21.442+00:00 2026-07-21T11:09:21.442+00:00 Active DDoS Attack in US - Central Type: Incident

Duration: 2 hours and 49 minutes

Affected Components: Networking

Jul 21, 11:09:21 GMT+0
Identified - We are currently experiencing a DDoS attack in the US - Central region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..

Jul 21, 11:20:15 GMT+0
Monitoring - We have implemented mitigations are monitoring the situation..

Jul 21, 11:41:41 GMT+0
Resolved - The DDoS attack has stopped, we will continue to monitor the affected region..

Jul 21, 13:23:09 GMT+0
Identified - We are currently experiencing a DDoS attack in the US - Central region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..

Jul 21, 13:43:46 GMT+0
Monitoring - We have implemented mitigations are monitoring the situation..

Jul 21, 13:58:28 GMT+0
Resolved - This incident has been resolved. We will continue to monitor the affected region..

]]> tag:status.northflank.com,2005:Incident/cmqjv1jt601p42okxsc6cjelx 2026-06-18T18:55:48.117+00:00 2026-06-18T18:55:48.117+00:00 Let's Encrypt Certificate Generation Issues Type: Incident

Duration: 4 hours and 40 minutes

Affected Components: Certificates

Jun 18, 18:55:48 GMT+0
Investigating - We are currently observing delays in certificate generation due to an ongoing incident with Let's Encrypt: <https://letsencrypt.status.io/> This will affect provisioning of new addons with TLS enabled, the provisioning of BYOC clusters as well as domain certification provisioning..

Jun 18, 23:02:45 GMT+0
Monitoring - No recent API errors experienced and addon provisioning is working as expected. We're still closely monitoring Let's Encrypt API responses..

Jun 18, 23:36:05 GMT+0
Resolved - No more certificate creation errors have been observed from Let's Encrypt. This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmq896fck00lcqopy8micgt3z 2026-06-10T15:58:15.402+00:00 2026-06-10T15:58:15.402+00:00 Issues accessing app.northflank.com Type: Incident

Duration: 17 minutes

Affected Components: Northflank App

Jun 10, 15:58:15 GMT+0
Investigating - We are currently investigating this incident..

Jun 10, 16:03:23 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Users should now be able to log in and access the UI. Running workloads were unaffected..

Jun 10, 16:14:52 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmq7v8suo0762p4uz2pxqysg8 2026-06-10T08:00:23.675+00:00 2026-06-10T08:00:23.675+00:00 Europe - West (London): Intermittent connectivity issues impacting Postgres cluster management Type: Incident

Duration: 6 hours and 23 minutes

Affected Components: Addons

Jun 10, 08:00:23 GMT+0
Investigating - We are currently investigating this incident. You may see logs in Postgres with the following error message: ``` ERROR: Error communicating with DCS ```.

Jun 10, 09:59:53 GMT+0
Monitoring - We have observed a recovery starting about 20 minutes ago. Currently, Postgres addons are stable..

Jun 10, 10:19:44 GMT+0
Identified - We saw a period of recovery between 09:45 and 10:04 UTC, but the issue has since recurred. This is tied to an ongoing incident affecting our cloud service provider in the region. We are continuing to monitor and will share further updates as we have them.

Jun 10, 11:53:12 GMT+0
Monitoring - We have not observed any further issues since 11:05 UTC. We are awaiting confirmation from our cloud provider that the issue is fully resolved on their side, and we will update this status accordingly..

Jun 10, 14:23:08 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmp46smud00byp6qva4zwgzm6 2026-05-13T15:00:45.533+00:00 2026-05-13T15:00:45.533+00:00 Europe - West (London): New resources not able to be created Type: Incident

Duration: 1 hour and 26 minutes

Affected Components: Addons, Jobs, Services

May 13, 15:00:45 GMT+0
Investigating - We are currently investigating this incident. Existing running workloads are unaffected..

May 13, 15:33:29 GMT+0
Monitoring - We are seeing an improvement; workloads are starting to be created. Connectivity has been restored, and we are continuing to monitor the situation..

May 13, 16:27:13 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmp05l0ln00fo96bh1a4pufch 2026-05-10T18:30:00.000+00:00 2026-05-10T18:30:00.000+00:00 Active DDoS Attack in Europe - West - Netherlands Type: Incident

Duration: 2 hours and 2 minutes

Affected Components: Addons, Services

May 10, 18:30:00 GMT+0
Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..

May 10, 20:10:07 GMT+0
Monitoring - We have put in place mitigations and are monitoring the affected region..

May 10, 20:31:56 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmoxb63od01es6n0u18am5oyu 2026-05-08T19:00:00.000+00:00 2026-05-08T19:00:00.000+00:00 Let's Encrypt Certificate Generation Issues Type: Incident

Duration: 2 hours and 36 minutes

Affected Components: Certificates

May 8, 19:00:00 GMT+0
Identified - [Let's Encrypt is currently experiencing a partial service disruption](https://letsencrypt.status.io/) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons, domains and BYOC clusters..

May 8, 21:36:09 GMT+0
Resolved - Certificate generation is functioning normally again. We will continue to monitor the situation..

]]> tag:status.northflank.com,2005:Incident/cmnpp03h1006jd0s28d2teq25 2026-04-08T06:15:00.000+00:00 2026-04-08T06:15:00.000+00:00 Addon and volume availability issues in US - Central Type: Incident

Duration: 52 minutes

Affected Components: Addons, Services

Apr 8, 06:15:00 GMT+0
Identified - We are aware of an issue with a node affecting some stateful workloads in US - Central. We are working on a fix for this incident..

Apr 8, 07:07:27 GMT+0
Resolved - We have recovered the node, and all workloads are now running again..

]]> tag:status.northflank.com,2005:Incident/cmn651uh000a880hcl3ox9cxf 2026-03-25T14:28:04.826+00:00 2026-03-25T14:28:04.826+00:00 Active DDoS Attack in Europe - West - Netherlands Type: Incident

Duration: 3 hours and 21 minutes

Affected Components: Addons, Jobs, Services

Mar 25, 14:28:04 GMT+0
Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..

Mar 25, 16:01:00 GMT+0
Monitoring - The region is currently stable. We are still actively mitigating the ongoing attack..

Mar 25, 17:48:48 GMT+0
Resolved - The DDoS attack has stopped we will continue to monitor the affected region..

]]> tag:status.northflank.com,2005:Incident/cmmth1qit0cwahxt9crdwuuo3 2026-03-16T17:42:54.132+00:00 2026-03-16T18:16:28.120+00:00 App UI is experiencing loading issues Type: Incident

Duration: 34 minutes

Affected Components: Northflank App

Mar 16, 18:16:28 GMT+0
Resolved - This incident has been resolved..

Mar 16, 18:15:58 GMT+0
Monitoring - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation..

Mar 16, 17:42:54 GMT+0
Identified - We are investing an incident where the UI is failing to load correctly..

]]> tag:status.northflank.com,2005:Incident/cmmj4y7ct01zw1jix7jv6r4vd 2026-03-09T12:06:32.914+00:00 2026-03-09T12:06:32.914+00:00 Degraded performance in US Central region Type: Incident

Duration: 29 minutes

Affected Components: Networking, Addons, Services

Mar 9, 12:06:32 GMT+0
Investigating - We are currently investigating this incident..

Mar 9, 12:35:56 GMT+0
Resolved - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads..

]]> tag:status.northflank.com,2005:Incident/cmltl7z5d002trjwmeqyabys9 2026-02-19T15:00:00.487+00:00 2026-02-19T15:00:00.487+00:00 Issues with builds using the Heroku 24 Buildpack Type: Incident

Duration: 1 hour and 34 minutes

Affected Components: Builds

Feb 19, 15:00:00 GMT+0
Identified - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released..

Feb 19, 16:34:09 GMT+0
Resolved - We have released a fix for the issue..

]]> tag:status.northflank.com,2005:Incident/cmj1i24qr0106uhdob8a4bf2w 2025-12-11T13:54:32.305+00:00 2025-12-11T13:54:32.305+00:00 Node infrastructure stability - London Region Type: Incident

Duration: 39 minutes

Affected Components: Networking, , Addons, Jobs, Services, Northflank Platform →

Dec 11, 13:54:32 GMT+0
Identified - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation..

Dec 11, 14:13:28 GMT+0
Monitoring - The mitigation is in place and the region is operating as expected..

Dec 11, 14:33:03 GMT+0
Resolved - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution..

]]> tag:status.northflank.com,2005:Incident/cmfyn4drv02axox9ea8otzvu7 2025-09-24T23:53:50.567+00:00 2025-09-24T23:53:50.567+00:00 Docker Hub is experiencing issues Type: Incident

Duration: 1 hour and 44 minutes

Affected Components: Addons, Jobs, Services

Sep 24, 23:53:50 GMT+0
Identified - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. <https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9>.

Sep 25, 01:11:08 GMT+0
Monitoring - We are observing that image pull requests for Docker Hub images are now succeeding..

Sep 25, 01:38:09 GMT+0
Resolved - Dockerhub is fully operational and image pulling is now working as expected..

]]> tag:status.northflank.com,2005:Incident/cmfmovpks003r739gifi0hnsh 2025-09-16T15:09:51.058+00:00 2025-09-16T15:09:51.058+00:00 Delay in DNS propagation Type: Incident

Duration: 18 hours and 29 minutes

Affected Components: Addons, Services, Certificates

Sep 16, 15:09:51 GMT+0
Monitoring - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: <https://status.ai-apps-comms.ibm.com/ns1connect>.

Sep 17, 09:39:01 GMT+0
Resolved - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour..

]]> tag:status.northflank.com,2005:Incident/cmernfevf002amj1q4pgwr3xn 2025-08-25T21:48:19.642+00:00 2025-08-25T21:48:19.642+00:00 Issue affecting local disk caching for builds running on Northflank's infrastructure Type: Incident

Duration: 1 hour and 17 minutes

Affected Components: Builds

Aug 25, 21:48:19 GMT+0
Identified - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this..

Aug 25, 22:45:24 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 25, 23:04:54 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmdy7qu4001ym91lf2txeit0h 2025-08-05T07:23:59.728+00:00 2025-08-05T07:23:59.728+00:00 Log & Metric Ingestion and Query Outage Type: Incident

Duration: 1 hour

Affected Components: Logs and Metrics

Aug 5, 07:23:59 GMT+0
Investigating - We are currently investigating this incident..

Aug 5, 07:55:18 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 5, 08:23:56 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.northflank.com,2005:Incident/cmddhhx1500cbb87xjzw4t6ee 2025-07-21T19:13:49.958+00:00 2025-07-21T19:13:49.958+00:00 Let's Encrypt Certificate Generation Issues Type: Incident

Duration: 11 hours and 31 minutes

Affected Components: Certificates

Jul 21, 19:13:49 GMT+0
Monitoring - [Let's Encrypt is currently experiencing an outage](https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/687e8d62b8a4e804fad85799) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains..

Jul 22, 06:44:30 GMT+0
Resolved - Certificate generation is functioning without issue. We will continue to monitor the situation..

]]> tag:status.northflank.com,2005:Incident/cmbtqad4j000l6didp4d0xkfy 2025-06-12T18:00:00.000+00:00 2025-06-12T20:40:23.833+00:00 Issues with Google Cloud platform - Affecting multiple services Type: Incident

Duration: 2 hours and 45 minutes

Affected Components: Builds, Addons, Logs and Metrics

Jun 12, 20:40:23 GMT+0
Monitoring - We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: \* GCP BYOC cluster processing.

Jun 12, 18:00:00 GMT+0
Identified - We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected..

Jun 12, 19:21:45 GMT+0
Monitoring - Google Cloud has acknowledged the incident. For detailed information, please visit: <https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW> Most services have been restored to normal operation. However, we are still experiencing the following issues: * Addon creation in the US - Central region * Addon backups in the US - Central region * GCP BYOC cluster processing We will continue monitoring the situation and provide updates as they become available..

Jun 12, 20:44:39 GMT+0
Resolved - GCP BYOC cluster processing is now operational..

]]> tag:status.northflank.com,2005:Incident/cmbsjf79f0025k18t40kcbyb0 2025-06-11T22:00:00.000+00:00 2025-06-11T22:00:00.000+00:00 Metrics ingestion and query outage Type: Incident

Duration: 2 hours and 17 minutes

Affected Components: Logs and Metrics

Jun 11, 22:00:00 GMT+0
Identified - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results..

Jun 11, 23:16:34 GMT+0
Monitoring - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up..

Jun 12, 00:16:31 GMT+0
Resolved - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future..

]]> tag:status.northflank.com,2005:Incident/cmay5h4pg0052zbome4c6m5q6 2025-05-21T16:04:00.000+00:00 2025-05-21T16:04:00.000+00:00 Infrastructure stability issues Type: Incident

Duration: 1 hour and 10 minutes

Affected Components: Networking, Addons, Services

May 21, 16:04:00 GMT+0
Monitoring - .

May 21, 17:13:33 GMT+0
Resolved - **We sincerely apologise for the outage you experienced.** **What happened:** Yesterday, we rolled out an update to support ARM architecture across our platform. As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas. This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers. **Timeline of events (UTC):** * **15:36** — Northflank engineers were alerted to failing health checks on customer endpoints. * **15:41** — Elevated 503 errors were detected from a specific cluster. * **15:42** — An internal incident was formally declared. * **15:57** — Replacement infrastructure began provisioning to recover failed nodes. * **16:32** — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention. Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again. Will Stewart CEO @ Northflank.

]]> tag:status.northflank.com,2005:Incident/cm391ojva000bzr1411i47iz0 2024-11-08T18:02:57.501+00:00 2024-11-08T19:20:22.960+00:00 US - Central - Inability to Create Disk Snapshots Type: Incident

Duration: 1 hour and 17 minutes

Affected Components: Addons

Nov 8, 19:20:22 GMT+0
Resolved - This incident has been resolved..

Nov 8, 18:02:57 GMT+0
Monitoring - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region.

]]> tag:status.northflank.com,2005:Incident/cm252qf810036lb09qejd495o 2024-10-11T18:41:37.083+00:00 2024-10-11T18:41:37.083+00:00 DNS Propagation Delay Type: Incident

Duration: 1 hour and 27 minutes

Affected Components: Services, Addons, Certificates

Oct 11, 18:41:37 GMT+0
Monitoring - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. <https://ns1connect.trust.pagerduty.com/posts/details/PK35B1W>.

Oct 11, 20:09:04 GMT+0
Resolved - This incident has been resolved..

]]>