Northflank Status - Incident history https://status.northflank.com Northflank Wed, 5 Aug 2026 13:27:13 +0000 Workloads Failing to Start Across Multiple Regions Type: Incident Duration: 1 hour and 1 minute Affected Components: Addons, Jobs, Services Aug 5, 13:27:13 GMT+0 - Investigating - We are seeing partial start up issues for workloads across multiple regions. We are currently investigating this incident. Aug 5, 13:42:39 GMT+0 - Monitoring - We have identified the issue and applied a mitigation. Previously stuck workloads are now starting successfully across affected regions. We are monitoring recovery and will confirm once all workloads are fully operational. Aug 5, 14:27:54 GMT+0 - Resolved - All affected workloads are now fully operational. Type: Incident

Duration: 1 hour and 1 minute

Affected Components: , ,

<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>13:27:13</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing partial start up issues for workloads across multiple regions. We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>13:42:39</var> GMT+0</small><br><strong>Monitoring</strong> - We have identified the issue and applied a mitigation. Previously stuck workloads are now starting successfully across affected regions. We are monitoring recovery and will confirm once all workloads are fully operational..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>14:27:54</var> GMT+0</small><br><strong>Resolved</strong> - All affected workloads are now fully operational. .</p> ]]> Wed, 5 Aug 2026 13:27:13 +0000 https://status.northflank.com/incident/cmsg4fvtt034m1ao3nowjoker https://status.northflank.com/incident/cmsg4fvtt034m1ao3nowjoker Investigating node stability issues in the US - East region Type: Incident Duration: 1 hour and 22 minutes Affected Components: Addons, Jobs, Services Jul 22, 12:17:46 GMT+0 - Investigating - We have noticed increased failure rates of nodes in the US - East region and are investigating the issue. Jul 22, 13:11:06 GMT+0 - Monitoring - We are seeing recovery across compute and storage. We are working to get the remaining workloads back online. Jul 22, 13:39:58 GMT+0 - Resolved - We have recovered affected workloads in the region and there are no more on-going node failures. The root cause was an outage on the Google Cloud Platform in us-east4 which hosts Northflank's US - East region. Type: Incident

Duration: 1 hour and 22 minutes

Affected Components: , ,

<p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>12:17:46</var> GMT+0</small><br><strong>Investigating</strong> - We have noticed increased failure rates of nodes in the US - East region and are investigating the issue..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>13:11:06</var> GMT+0</small><br><strong>Monitoring</strong> - We are seeing recovery across compute and storage. We are working to get the remaining workloads back online..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>13:39:58</var> GMT+0</small><br><strong>Resolved</strong> - We have recovered affected workloads in the region and there are no more on-going node failures. The root cause was an outage on the Google Cloud Platform in us-east4 which hosts Northflank's US - East region..</p> ]]> Wed, 22 Jul 2026 12:17:46 +0000 https://status.northflank.com/incident/cmrw1sp1g00870smo8ztfcsfn https://status.northflank.com/incident/cmrw1sp1g00870smo8ztfcsfn Active DDoS Attack in US - Central Type: Incident Duration: 2 hours and 49 minutes Affected Components: Networking Jul 21, 11:09:21 GMT+0 - Identified - We are currently experiencing a DDoS attack in the US - Central region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations. Jul 21, 11:20:15 GMT+0 - Monitoring - We have implemented mitigations are monitoring the situation. Jul 21, 11:41:41 GMT+0 - Resolved - The DDoS attack has stopped, we will continue to monitor the affected region. Jul 21, 13:23:09 GMT+0 - Identified - We are currently experiencing a DDoS attack in the US - Central region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations. Jul 21, 13:43:46 GMT+0 - Monitoring - We have implemented mitigations are monitoring the situation. Jul 21, 13:58:28 GMT+0 - Resolved - This incident has been resolved. We will continue to monitor the affected region. Type: Incident

Duration: 2 hours and 49 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>11:09:21</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a DDoS attack in the US - Central region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>11:20:15</var> GMT+0</small><br><strong>Monitoring</strong> - We have implemented mitigations are monitoring the situation..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>11:41:41</var> GMT+0</small><br><strong>Resolved</strong> - The DDoS attack has stopped, we will continue to monitor the affected region..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>13:23:09</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a DDoS attack in the US - Central region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>13:43:46</var> GMT+0</small><br><strong>Monitoring</strong> - We have implemented mitigations are monitoring the situation..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>13:58:28</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. We will continue to monitor the affected region..</p> ]]> Tue, 21 Jul 2026 11:09:21 +0000 https://status.northflank.com/incident/cmrujwsz307ud0kqm1nugymth https://status.northflank.com/incident/cmrujwsz307ud0kqm1nugymth Let's Encrypt Certificate Generation Issues Type: Incident Duration: 4 hours and 40 minutes Affected Components: Certificates Jun 18, 18:55:48 GMT+0 - Investigating - We are currently observing delays in certificate generation due to an ongoing incident with Let's Encrypt: <https://letsencrypt.status.io/> This will affect provisioning of new addons with TLS enabled, the provisioning of BYOC clusters as well as domain certification provisioning. Jun 18, 23:02:45 GMT+0 - Monitoring - No recent API errors experienced and addon provisioning is working as expected. We're still closely monitoring Let's Encrypt API responses. Jun 18, 23:36:05 GMT+0 - Resolved - No more certificate creation errors have been observed from Let's Encrypt. This incident has been resolved. Type: Incident

Duration: 4 hours and 40 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 18</var>, <var data-var='time'>18:55:48</var> GMT+0</small><br><strong>Investigating</strong> - We are currently observing delays in certificate generation due to an ongoing incident with Let's Encrypt: <https://letsencrypt.status.io/> This will affect provisioning of new addons with TLS enabled, the provisioning of BYOC clusters as well as domain certification provisioning..</p> <p><small>Jun <var data-var='date'> 18</var>, <var data-var='time'>23:02:45</var> GMT+0</small><br><strong>Monitoring</strong> - No recent API errors experienced and addon provisioning is working as expected. We're still closely monitoring Let's Encrypt API responses..</p> <p><small>Jun <var data-var='date'> 18</var>, <var data-var='time'>23:36:05</var> GMT+0</small><br><strong>Resolved</strong> - No more certificate creation errors have been observed from Let's Encrypt. This incident has been resolved..</p> ]]> Thu, 18 Jun 2026 18:55:48 +0000 https://status.northflank.com/incident/cmqjv1jt601p42okxsc6cjelx https://status.northflank.com/incident/cmqjv1jt601p42okxsc6cjelx Issues accessing app.northflank.com Type: Incident Duration: 17 minutes Affected Components: Northflank App Jun 10, 15:58:15 GMT+0 - Investigating - We are currently investigating this incident. Jun 10, 16:03:23 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Users should now be able to log in and access the UI. Running workloads were unaffected. Jun 10, 16:14:52 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 17 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>15:58:15</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>16:03:23</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result. Users should now be able to log in and access the UI. Running workloads were unaffected..</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>16:14:52</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 10 Jun 2026 15:58:15 +0000 https://status.northflank.com/incident/cmq896fck00lcqopy8micgt3z https://status.northflank.com/incident/cmq896fck00lcqopy8micgt3z Europe - West (London): Intermittent connectivity issues impacting Postgres cluster management Type: Incident Duration: 6 hours and 23 minutes Affected Components: Addons Jun 10, 08:00:23 GMT+0 - Investigating - We are currently investigating this incident. You may see logs in Postgres with the following error message: ``` ERROR: Error communicating with DCS ``` Jun 10, 09:59:53 GMT+0 - Monitoring - We have observed a recovery starting about 20 minutes ago. Currently, Postgres addons are stable. Jun 10, 10:19:44 GMT+0 - Identified - We saw a period of recovery between 09:45 and 10:04 UTC, but the issue has since recurred. This is tied to an ongoing incident affecting our cloud service provider in the region. We are continuing to monitor and will share further updates as we have them Jun 10, 11:53:12 GMT+0 - Monitoring - We have not observed any further issues since 11:05 UTC. We are awaiting confirmation from our cloud provider that the issue is fully resolved on their side, and we will update this status accordingly. Jun 10, 14:23:08 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 6 hours and 23 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>08:00:23</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. You may see logs in Postgres with the following error message: ``` ERROR: Error communicating with DCS ```.</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>09:59:53</var> GMT+0</small><br><strong>Monitoring</strong> - We have observed a recovery starting about 20 minutes ago. Currently, Postgres addons are stable..</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>10:19:44</var> GMT+0</small><br><strong>Identified</strong> - We saw a period of recovery between 09:45 and 10:04 UTC, but the issue has since recurred. This is tied to an ongoing incident affecting our cloud service provider in the region. We are continuing to monitor and will share further updates as we have them.</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>11:53:12</var> GMT+0</small><br><strong>Monitoring</strong> - We have not observed any further issues since 11:05 UTC. We are awaiting confirmation from our cloud provider that the issue is fully resolved on their side, and we will update this status accordingly..</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>14:23:08</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 10 Jun 2026 08:00:23 +0000 https://status.northflank.com/incident/cmq7v8suo0762p4uz2pxqysg8 https://status.northflank.com/incident/cmq7v8suo0762p4uz2pxqysg8 Europe - West (London): New resources not able to be created Type: Incident Duration: 1 hour and 26 minutes Affected Components: Addons, Jobs, Services May 13, 15:00:45 GMT+0 - Investigating - We are currently investigating this incident. Existing running workloads are unaffected. May 13, 15:33:29 GMT+0 - Monitoring - We are seeing an improvement; workloads are starting to be created. Connectivity has been restored, and we are continuing to monitor the situation. May 13, 16:27:13 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 26 minutes

Affected Components: , ,

<p><small>May <var data-var='date'> 13</var>, <var data-var='time'>15:00:45</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. Existing running workloads are unaffected..</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>15:33:29</var> GMT+0</small><br><strong>Monitoring</strong> - We are seeing an improvement; workloads are starting to be created. Connectivity has been restored, and we are continuing to monitor the situation..</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>16:27:13</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 13 May 2026 15:00:45 +0000 https://status.northflank.com/incident/cmp46smud00byp6qva4zwgzm6 https://status.northflank.com/incident/cmp46smud00byp6qva4zwgzm6 Active DDoS Attack in Europe - West - Netherlands Type: Incident Duration: 2 hours and 2 minutes Affected Components: Addons, Services May 10, 18:30:00 GMT+0 - Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations. May 10, 20:10:07 GMT+0 - Monitoring - We have put in place mitigations and are monitoring the affected region. May 10, 20:31:56 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 hours and 2 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 10</var>, <var data-var='time'>18:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..</p> <p><small>May <var data-var='date'> 10</var>, <var data-var='time'>20:10:07</var> GMT+0</small><br><strong>Monitoring</strong> - We have put in place mitigations and are monitoring the affected region..</p> <p><small>May <var data-var='date'> 10</var>, <var data-var='time'>20:31:56</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 10 May 2026 18:30:00 +0000 https://status.northflank.com/incident/cmp05l0ln00fo96bh1a4pufch https://status.northflank.com/incident/cmp05l0ln00fo96bh1a4pufch Let's Encrypt Certificate Generation Issues Type: Incident Duration: 2 hours and 36 minutes Affected Components: Certificates May 8, 19:00:00 GMT+0 - Identified - [Let's Encrypt is currently experiencing a partial service disruption](https://letsencrypt.status.io/) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons, domains and BYOC clusters. May 8, 21:36:09 GMT+0 - Resolved - Certificate generation is functioning normally again. We will continue to monitor the situation. Type: Incident

Duration: 2 hours and 36 minutes

Affected Components:

<p><small>May <var data-var='date'> 8</var>, <var data-var='time'>19:00:00</var> GMT+0</small><br><strong>Identified</strong> - [Let's Encrypt is currently experiencing a partial service disruption](https://letsencrypt.status.io/) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons, domains and BYOC clusters..</p> <p><small>May <var data-var='date'> 8</var>, <var data-var='time'>21:36:09</var> GMT+0</small><br><strong>Resolved</strong> - Certificate generation is functioning normally again. We will continue to monitor the situation..</p> ]]> Fri, 8 May 2026 19:00:00 +0000 https://status.northflank.com/incident/cmoxb63od01es6n0u18am5oyu https://status.northflank.com/incident/cmoxb63od01es6n0u18am5oyu Addon and volume availability issues in US - Central Type: Incident Duration: 52 minutes Affected Components: Addons, Services Apr 8, 06:15:00 GMT+0 - Identified - We are aware of an issue with a node affecting some stateful workloads in US - Central. We are working on a fix for this incident. Apr 8, 07:07:27 GMT+0 - Resolved - We have recovered the node, and all workloads are now running again. Type: Incident

Duration: 52 minutes

Affected Components: ,

<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>06:15:00</var> GMT+0</small><br><strong>Identified</strong> - We are aware of an issue with a node affecting some stateful workloads in US - Central. We are working on a fix for this incident..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>07:07:27</var> GMT+0</small><br><strong>Resolved</strong> - We have recovered the node, and all workloads are now running again..</p> ]]> Wed, 8 Apr 2026 06:15:00 +0000 https://status.northflank.com/incident/cmnpp03h1006jd0s28d2teq25 https://status.northflank.com/incident/cmnpp03h1006jd0s28d2teq25 Active DDoS Attack in Europe - West - Netherlands Type: Incident Duration: 3 hours and 21 minutes Affected Components: Addons, Jobs, Services Mar 25, 14:28:04 GMT+0 - Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations. Mar 25, 16:01:00 GMT+0 - Monitoring - The region is currently stable. We are still actively mitigating the ongoing attack. Mar 25, 17:48:48 GMT+0 - Resolved - The DDoS attack has stopped we will continue to monitor the affected region. Type: Incident

Duration: 3 hours and 21 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>14:28:04</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..</p> <p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>16:01:00</var> GMT+0</small><br><strong>Monitoring</strong> - The region is currently stable. We are still actively mitigating the ongoing attack..</p> <p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>17:48:48</var> GMT+0</small><br><strong>Resolved</strong> - The DDoS attack has stopped we will continue to monitor the affected region..</p> ]]> Wed, 25 Mar 2026 14:28:04 +0000 https://status.northflank.com/incident/cmn651uh000a880hcl3ox9cxf https://status.northflank.com/incident/cmn651uh000a880hcl3ox9cxf App UI is experiencing loading issues Type: Incident Duration: 34 minutes Affected Components: Northflank App Mar 16, 18:16:28 GMT+0 - Resolved - This incident has been resolved. Mar 16, 18:15:58 GMT+0 - Monitoring - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation. Mar 16, 17:42:54 GMT+0 - Identified - We are investing an incident where the UI is failing to load correctly. Type: Incident

Duration: 34 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>18:16:28</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>18:15:58</var> GMT+0</small><br><strong>Monitoring</strong> - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation..</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>17:42:54</var> GMT+0</small><br><strong>Identified</strong> - We are investing an incident where the UI is failing to load correctly..</p> ]]> Mon, 16 Mar 2026 17:42:54 +0000 https://status.northflank.com/incident/cmmth1qit0cwahxt9crdwuuo3 https://status.northflank.com/incident/cmmth1qit0cwahxt9crdwuuo3 Degraded performance in US Central region Type: Incident Duration: 29 minutes Affected Components: Networking, Addons, Services Mar 9, 12:06:32 GMT+0 - Investigating - We are currently investigating this incident. Mar 9, 12:35:56 GMT+0 - Resolved - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads. Type: Incident

Duration: 29 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>12:06:32</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>12:35:56</var> GMT+0</small><br><strong>Resolved</strong> - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads..</p> ]]> Mon, 9 Mar 2026 12:06:32 +0000 https://status.northflank.com/incident/cmmj4y7ct01zw1jix7jv6r4vd https://status.northflank.com/incident/cmmj4y7ct01zw1jix7jv6r4vd Issues with builds using the Heroku 24 Buildpack Type: Incident Duration: 1 hour and 34 minutes Affected Components: Builds Feb 19, 15:00:00 GMT+0 - Identified - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released. Feb 19, 16:34:09 GMT+0 - Resolved - We have released a fix for the issue. Type: Incident

Duration: 1 hour and 34 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>16:34:09</var> GMT+0</small><br><strong>Resolved</strong> - We have released a fix for the issue..</p> ]]> Thu, 19 Feb 2026 15:00:00 +0000 https://status.northflank.com/incident/cmltl7z5d002trjwmeqyabys9 https://status.northflank.com/incident/cmltl7z5d002trjwmeqyabys9 Node infrastructure stability - London Region Type: Incident Duration: 39 minutes Affected Components: Networking, , Addons, Jobs, Services, Northflank Platform → Dec 11, 13:54:32 GMT+0 - Identified - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation. Dec 11, 14:13:28 GMT+0 - Monitoring - The mitigation is in place and the region is operating as expected. Dec 11, 14:33:03 GMT+0 - Resolved - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution. Type: Incident

Duration: 39 minutes

Affected Components: , , , ,

<p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>13:54:32</var> GMT+0</small><br><strong>Identified</strong> - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:13:28</var> GMT+0</small><br><strong>Monitoring</strong> - The mitigation is in place and the region is operating as expected..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:33:03</var> GMT+0</small><br><strong>Resolved</strong> - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution..</p> ]]> Thu, 11 Dec 2025 13:54:32 +0000 https://status.northflank.com/incident/cmj1i24qr0106uhdob8a4bf2w https://status.northflank.com/incident/cmj1i24qr0106uhdob8a4bf2w Docker Hub is experiencing issues Type: Incident Duration: 1 hour and 44 minutes Affected Components: Addons, Jobs, Services Sep 24, 23:53:50 GMT+0 - Identified - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. <https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9> Sep 25, 01:11:08 GMT+0 - Monitoring - We are observing that image pull requests for Docker Hub images are now succeeding. Sep 25, 01:38:09 GMT+0 - Resolved - Dockerhub is fully operational and image pulling is now working as expected. Type: Incident

Duration: 1 hour and 44 minutes

Affected Components: , ,

<p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>23:53:50</var> GMT+0</small><br><strong>Identified</strong> - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. <https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9>.</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>01:11:08</var> GMT+0</small><br><strong>Monitoring</strong> - We are observing that image pull requests for Docker Hub images are now succeeding..</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>01:38:09</var> GMT+0</small><br><strong>Resolved</strong> - Dockerhub is fully operational and image pulling is now working as expected..</p> ]]> Wed, 24 Sep 2025 23:53:50 +0000 https://status.northflank.com/incident/cmfyn4drv02axox9ea8otzvu7 https://status.northflank.com/incident/cmfyn4drv02axox9ea8otzvu7 Delay in DNS propagation Type: Incident Duration: 18 hours and 29 minutes Affected Components: Addons, Services, Certificates Sep 16, 15:09:51 GMT+0 - Monitoring - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: <https://status.ai-apps-comms.ibm.com/ns1connect> Sep 17, 09:39:01 GMT+0 - Resolved - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour. Type: Incident

Duration: 18 hours and 29 minutes

Affected Components: , ,

<p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>15:09:51</var> GMT+0</small><br><strong>Monitoring</strong> - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: <https://status.ai-apps-comms.ibm.com/ns1connect>.</p> <p><small>Sep <var data-var='date'> 17</var>, <var data-var='time'>09:39:01</var> GMT+0</small><br><strong>Resolved</strong> - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour..</p> ]]> Tue, 16 Sep 2025 15:09:51 +0000 https://status.northflank.com/incident/cmfmovpks003r739gifi0hnsh https://status.northflank.com/incident/cmfmovpks003r739gifi0hnsh Issue affecting local disk caching for builds running on Northflank's infrastructure Type: Incident Duration: 1 hour and 17 minutes Affected Components: Builds Aug 25, 21:48:19 GMT+0 - Identified - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this. Aug 25, 22:45:24 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 25, 23:04:54 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 17 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>21:48:19</var> GMT+0</small><br><strong>Identified</strong> - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this..</p> <p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>22:45:24</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>23:04:54</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 25 Aug 2025 21:48:19 +0000 https://status.northflank.com/incident/cmernfevf002amj1q4pgwr3xn https://status.northflank.com/incident/cmernfevf002amj1q4pgwr3xn Log & Metric Ingestion and Query Outage Type: Incident Duration: 1 hour Affected Components: Logs and Metrics Aug 5, 07:23:59 GMT+0 - Investigating - We are currently investigating this incident. Aug 5, 07:55:18 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 5, 08:23:56 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour

Affected Components:

<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>07:23:59</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>07:55:18</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>08:23:56</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 5 Aug 2025 07:23:59 +0000 https://status.northflank.com/incident/cmdy7qu4001ym91lf2txeit0h https://status.northflank.com/incident/cmdy7qu4001ym91lf2txeit0h Let's Encrypt Certificate Generation Issues Type: Incident Duration: 11 hours and 31 minutes Affected Components: Certificates Jul 21, 19:13:49 GMT+0 - Monitoring - [Let's Encrypt is currently experiencing an outage](https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/687e8d62b8a4e804fad85799) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains. Jul 22, 06:44:30 GMT+0 - Resolved - Certificate generation is functioning without issue. We will continue to monitor the situation. Type: Incident

Duration: 11 hours and 31 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>19:13:49</var> GMT+0</small><br><strong>Monitoring</strong> - [Let's Encrypt is currently experiencing an outage](https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/687e8d62b8a4e804fad85799) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>06:44:30</var> GMT+0</small><br><strong>Resolved</strong> - Certificate generation is functioning without issue. We will continue to monitor the situation..</p> ]]> Mon, 21 Jul 2025 19:13:49 +0000 https://status.northflank.com/incident/cmddhhx1500cbb87xjzw4t6ee https://status.northflank.com/incident/cmddhhx1500cbb87xjzw4t6ee Issues with Google Cloud platform - Affecting multiple services Type: Incident Duration: 2 hours and 45 minutes Affected Components: Builds, Addons, Logs and Metrics Jun 12, 20:40:23 GMT+0 - Monitoring - We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: \* GCP BYOC cluster processing Jun 12, 18:00:00 GMT+0 - Identified - We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected. Jun 12, 19:21:45 GMT+0 - Monitoring - Google Cloud has acknowledged the incident. For detailed information, please visit: <https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW> Most services have been restored to normal operation. However, we are still experiencing the following issues: * Addon creation in the US - Central region * Addon backups in the US - Central region * GCP BYOC cluster processing We will continue monitoring the situation and provide updates as they become available. Jun 12, 20:44:39 GMT+0 - Resolved - GCP BYOC cluster processing is now operational. Type: Incident

Duration: 2 hours and 45 minutes

Affected Components: , ,

<p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>20:40:23</var> GMT+0</small><br><strong>Monitoring</strong> - We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: \* GCP BYOC cluster processing.</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Identified</strong> - We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>19:21:45</var> GMT+0</small><br><strong>Monitoring</strong> - Google Cloud has acknowledged the incident. For detailed information, please visit: <https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW> Most services have been restored to normal operation. However, we are still experiencing the following issues: * Addon creation in the US - Central region * Addon backups in the US - Central region * GCP BYOC cluster processing We will continue monitoring the situation and provide updates as they become available..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>20:44:39</var> GMT+0</small><br><strong>Resolved</strong> - GCP BYOC cluster processing is now operational..</p> ]]> Thu, 12 Jun 2025 18:00:00 +0000 https://status.northflank.com/incident/cmbtqad4j000l6didp4d0xkfy https://status.northflank.com/incident/cmbtqad4j000l6didp4d0xkfy Metrics ingestion and query outage Type: Incident Duration: 2 hours and 17 minutes Affected Components: Logs and Metrics Jun 11, 22:00:00 GMT+0 - Identified - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results. Jun 11, 23:16:34 GMT+0 - Monitoring - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up. Jun 12, 00:16:31 GMT+0 - Resolved - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future. Type: Incident

Duration: 2 hours and 17 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Identified</strong> - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results..</p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>23:16:34</var> GMT+0</small><br><strong>Monitoring</strong> - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>00:16:31</var> GMT+0</small><br><strong>Resolved</strong> - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future..</p> ]]> Wed, 11 Jun 2025 22:00:00 +0000 https://status.northflank.com/incident/cmbsjf79f0025k18t40kcbyb0 https://status.northflank.com/incident/cmbsjf79f0025k18t40kcbyb0 Infrastructure stability issues Type: Incident Duration: 1 hour and 10 minutes Affected Components: Networking, Addons, Services May 21, 16:04:00 GMT+0 - Monitoring - May 21, 17:13:33 GMT+0 - Resolved - **We sincerely apologise for the outage you experienced.** **What happened:** Yesterday, we rolled out an update to support ARM architecture across our platform. As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas. This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers. **Timeline of events (UTC):** * **15:36** — Northflank engineers were alerted to failing health checks on customer endpoints. * **15:41** — Elevated 503 errors were detected from a specific cluster. * **15:42** — An internal incident was formally declared. * **15:57** — Replacement infrastructure began provisioning to recover failed nodes. * **16:32** — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention. Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again. Will Stewart CEO @ Northflank Type: Incident

Duration: 1 hour and 10 minutes

Affected Components: , ,

<p><small>May <var data-var='date'> 21</var>, <var data-var='time'>16:04:00</var> GMT+0</small><br><strong>Monitoring</strong> - .</p> <p><small>May <var data-var='date'> 21</var>, <var data-var='time'>17:13:33</var> GMT+0</small><br><strong>Resolved</strong> - **We sincerely apologise for the outage you experienced.** **What happened:** Yesterday, we rolled out an update to support ARM architecture across our platform. As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas. This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers. **Timeline of events (UTC):** * **15:36** — Northflank engineers were alerted to failing health checks on customer endpoints. * **15:41** — Elevated 503 errors were detected from a specific cluster. * **15:42** — An internal incident was formally declared. * **15:57** — Replacement infrastructure began provisioning to recover failed nodes. * **16:32** — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention. Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again. Will Stewart CEO @ Northflank.</p> ]]> Wed, 21 May 2025 16:04:00 +0000 https://status.northflank.com/incident/cmay5h4pg0052zbome4c6m5q6 https://status.northflank.com/incident/cmay5h4pg0052zbome4c6m5q6 US - Central - Inability to Create Disk Snapshots Type: Incident Duration: 1 hour and 17 minutes Affected Components: Addons Nov 8, 19:20:22 GMT+0 - Resolved - This incident has been resolved. Nov 8, 18:02:57 GMT+0 - Monitoring - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region Type: Incident

Duration: 1 hour and 17 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>19:20:22</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>18:02:57</var> GMT+0</small><br><strong>Monitoring</strong> - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region.</p> ]]> Fri, 8 Nov 2024 18:02:57 +0000 https://status.northflank.com/incident/cm391ojva000bzr1411i47iz0 https://status.northflank.com/incident/cm391ojva000bzr1411i47iz0 DNS Propagation Delay Type: Incident Duration: 1 hour and 27 minutes Affected Components: Services, Addons, Certificates Oct 11, 18:41:37 GMT+0 - Monitoring - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. <https://ns1connect.trust.pagerduty.com/posts/details/PK35B1W> Oct 11, 20:09:04 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 27 minutes

Affected Components: , ,

<p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>18:41:37</var> GMT+0</small><br><strong>Monitoring</strong> - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. <https://ns1connect.trust.pagerduty.com/posts/details/PK35B1W>.</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>20:09:04</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 11 Oct 2024 18:41:37 +0000 https://status.northflank.com/incident/cm252qf810036lb09qejd495o https://status.northflank.com/incident/cm252qf810036lb09qejd495o