Northflank Status - Incident history https://status.northflank.com Northflank Wed, 25 Mar 2026 14:28:04 +0000 Active DDoS Attack in Europe - West - Netherlands Type: Incident Duration: 3 hours and 21 minutes Affected Components: Addons, Jobs, Services Mar 25, 14:28:04 GMT+0 - Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations. Mar 25, 16:01:00 GMT+0 - Monitoring - The region is currently stable. We are still actively mitigating the ongoing attack. Mar 25, 17:48:48 GMT+0 - Resolved - The DDoS attack has stopped we will continue to monitor the affected region. Type: Incident

Duration: 3 hours and 21 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>14:28:04</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..</p> <p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>16:01:00</var> GMT+0</small><br><strong>Monitoring</strong> - The region is currently stable. We are still actively mitigating the ongoing attack..</p> <p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>17:48:48</var> GMT+0</small><br><strong>Resolved</strong> - The DDoS attack has stopped we will continue to monitor the affected region..</p> ]]> Wed, 25 Mar 2026 14:28:04 +0000 https://status.northflank.com/incident/cmn651uh000a880hcl3ox9cxf https://status.northflank.com/incident/cmn651uh000a880hcl3ox9cxf App UI is experiencing loading issues Type: Incident Duration: 34 minutes Affected Components: Northflank App Mar 16, 18:16:28 GMT+0 - Resolved - This incident has been resolved. Mar 16, 18:15:58 GMT+0 - Monitoring - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation. Mar 16, 17:42:54 GMT+0 - Identified - We are investing an incident where the UI is failing to load correctly. Type: Incident

Duration: 34 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>18:16:28</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>18:15:58</var> GMT+0</small><br><strong>Monitoring</strong> - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation..</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>17:42:54</var> GMT+0</small><br><strong>Identified</strong> - We are investing an incident where the UI is failing to load correctly..</p> ]]> Mon, 16 Mar 2026 17:42:54 +0000 https://status.northflank.com/incident/cmmth1qit0cwahxt9crdwuuo3 https://status.northflank.com/incident/cmmth1qit0cwahxt9crdwuuo3 Degraded performance in US Central region Type: Incident Duration: 29 minutes Affected Components: Networking, Addons, Services Mar 9, 12:06:32 GMT+0 - Investigating - We are currently investigating this incident. Mar 9, 12:35:56 GMT+0 - Resolved - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads. Type: Incident

Duration: 29 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>12:06:32</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>12:35:56</var> GMT+0</small><br><strong>Resolved</strong> - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads..</p> ]]> Mon, 9 Mar 2026 12:06:32 +0000 https://status.northflank.com/incident/cmmj4y7ct01zw1jix7jv6r4vd https://status.northflank.com/incident/cmmj4y7ct01zw1jix7jv6r4vd Issues with builds using the Heroku 24 Buildpack Type: Incident Duration: 1 hour and 34 minutes Affected Components: Builds Feb 19, 15:00:00 GMT+0 - Identified - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released. Feb 19, 16:34:09 GMT+0 - Resolved - We have released a fix for the issue. Type: Incident

Duration: 1 hour and 34 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>16:34:09</var> GMT+0</small><br><strong>Resolved</strong> - We have released a fix for the issue..</p> ]]> Thu, 19 Feb 2026 15:00:00 +0000 https://status.northflank.com/incident/cmltl7z5d002trjwmeqyabys9 https://status.northflank.com/incident/cmltl7z5d002trjwmeqyabys9 Node infrastructure stability - London Region Type: Incident Duration: 39 minutes Affected Components: Networking, , Addons, Jobs, Services, Northflank Platform → Dec 11, 13:54:32 GMT+0 - Identified - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation. Dec 11, 14:13:28 GMT+0 - Monitoring - The mitigation is in place and the region is operating as expected. Dec 11, 14:33:03 GMT+0 - Resolved - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution. Type: Incident

Duration: 39 minutes

Affected Components: , , , ,

<p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>13:54:32</var> GMT+0</small><br><strong>Identified</strong> - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:13:28</var> GMT+0</small><br><strong>Monitoring</strong> - The mitigation is in place and the region is operating as expected..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:33:03</var> GMT+0</small><br><strong>Resolved</strong> - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution..</p> ]]> Thu, 11 Dec 2025 13:54:32 +0000 https://status.northflank.com/incident/cmj1i24qr0106uhdob8a4bf2w https://status.northflank.com/incident/cmj1i24qr0106uhdob8a4bf2w Docker Hub is experiencing issues Type: Incident Duration: 1 hour and 44 minutes Affected Components: Addons, Jobs, Services Sep 24, 23:53:50 GMT+0 - Identified - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. <https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9> Sep 25, 01:11:08 GMT+0 - Monitoring - We are observing that image pull requests for Docker Hub images are now succeeding. Sep 25, 01:38:09 GMT+0 - Resolved - Dockerhub is fully operational and image pulling is now working as expected. Type: Incident

Duration: 1 hour and 44 minutes

Affected Components: , ,

<p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>23:53:50</var> GMT+0</small><br><strong>Identified</strong> - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. <https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9>.</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>01:11:08</var> GMT+0</small><br><strong>Monitoring</strong> - We are observing that image pull requests for Docker Hub images are now succeeding..</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>01:38:09</var> GMT+0</small><br><strong>Resolved</strong> - Dockerhub is fully operational and image pulling is now working as expected..</p> ]]> Wed, 24 Sep 2025 23:53:50 +0000 https://status.northflank.com/incident/cmfyn4drv02axox9ea8otzvu7 https://status.northflank.com/incident/cmfyn4drv02axox9ea8otzvu7 Delay in DNS propagation Type: Incident Duration: 18 hours and 29 minutes Affected Components: Addons, Services, Certificates Sep 16, 15:09:51 GMT+0 - Monitoring - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: <https://status.ai-apps-comms.ibm.com/ns1connect> Sep 17, 09:39:01 GMT+0 - Resolved - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour. Type: Incident

Duration: 18 hours and 29 minutes

Affected Components: , ,

<p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>15:09:51</var> GMT+0</small><br><strong>Monitoring</strong> - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: <https://status.ai-apps-comms.ibm.com/ns1connect>.</p> <p><small>Sep <var data-var='date'> 17</var>, <var data-var='time'>09:39:01</var> GMT+0</small><br><strong>Resolved</strong> - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour..</p> ]]> Tue, 16 Sep 2025 15:09:51 +0000 https://status.northflank.com/incident/cmfmovpks003r739gifi0hnsh https://status.northflank.com/incident/cmfmovpks003r739gifi0hnsh Issue affecting local disk caching for builds running on Northflank's infrastructure Type: Incident Duration: 1 hour and 17 minutes Affected Components: Builds Aug 25, 21:48:19 GMT+0 - Identified - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this. Aug 25, 22:45:24 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 25, 23:04:54 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 17 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>21:48:19</var> GMT+0</small><br><strong>Identified</strong> - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this..</p> <p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>22:45:24</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>23:04:54</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 25 Aug 2025 21:48:19 +0000 https://status.northflank.com/incident/cmernfevf002amj1q4pgwr3xn https://status.northflank.com/incident/cmernfevf002amj1q4pgwr3xn Log & Metric Ingestion and Query Outage Type: Incident Duration: 1 hour Affected Components: Logs and Metrics Aug 5, 07:23:59 GMT+0 - Investigating - We are currently investigating this incident. Aug 5, 07:55:18 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 5, 08:23:56 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour

Affected Components:

<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>07:23:59</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>07:55:18</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>08:23:56</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 5 Aug 2025 07:23:59 +0000 https://status.northflank.com/incident/cmdy7qu4001ym91lf2txeit0h https://status.northflank.com/incident/cmdy7qu4001ym91lf2txeit0h Let's Encrypt Certificate Generation Issues Type: Incident Duration: 11 hours and 31 minutes Affected Components: Certificates Jul 21, 19:13:49 GMT+0 - Monitoring - [Let's Encrypt is currently experiencing an outage](https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/687e8d62b8a4e804fad85799) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains. Jul 22, 06:44:30 GMT+0 - Resolved - Certificate generation is functioning without issue. We will continue to monitor the situation. Type: Incident

Duration: 11 hours and 31 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>19:13:49</var> GMT+0</small><br><strong>Monitoring</strong> - [Let's Encrypt is currently experiencing an outage](https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/687e8d62b8a4e804fad85799) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>06:44:30</var> GMT+0</small><br><strong>Resolved</strong> - Certificate generation is functioning without issue. We will continue to monitor the situation..</p> ]]> Mon, 21 Jul 2025 19:13:49 +0000 https://status.northflank.com/incident/cmddhhx1500cbb87xjzw4t6ee https://status.northflank.com/incident/cmddhhx1500cbb87xjzw4t6ee Issues with Google Cloud platform - Affecting multiple services Type: Incident Duration: 2 hours and 45 minutes Affected Components: Builds, Addons, Logs and Metrics Jun 12, 20:40:23 GMT+0 - Monitoring - We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: \* GCP BYOC cluster processing Jun 12, 18:00:00 GMT+0 - Identified - We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected. Jun 12, 19:21:45 GMT+0 - Monitoring - Google Cloud has acknowledged the incident. For detailed information, please visit: <https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW> Most services have been restored to normal operation. However, we are still experiencing the following issues: * Addon creation in the US - Central region * Addon backups in the US - Central region * GCP BYOC cluster processing We will continue monitoring the situation and provide updates as they become available. Jun 12, 20:44:39 GMT+0 - Resolved - GCP BYOC cluster processing is now operational. Type: Incident

Duration: 2 hours and 45 minutes

Affected Components: , ,

<p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>20:40:23</var> GMT+0</small><br><strong>Monitoring</strong> - We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: \* GCP BYOC cluster processing.</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Identified</strong> - We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>19:21:45</var> GMT+0</small><br><strong>Monitoring</strong> - Google Cloud has acknowledged the incident. For detailed information, please visit: <https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW> Most services have been restored to normal operation. However, we are still experiencing the following issues: * Addon creation in the US - Central region * Addon backups in the US - Central region * GCP BYOC cluster processing We will continue monitoring the situation and provide updates as they become available..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>20:44:39</var> GMT+0</small><br><strong>Resolved</strong> - GCP BYOC cluster processing is now operational..</p> ]]> Thu, 12 Jun 2025 18:00:00 +0000 https://status.northflank.com/incident/cmbtqad4j000l6didp4d0xkfy https://status.northflank.com/incident/cmbtqad4j000l6didp4d0xkfy Metrics ingestion and query outage Type: Incident Duration: 2 hours and 17 minutes Affected Components: Logs and Metrics Jun 11, 22:00:00 GMT+0 - Identified - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results. Jun 11, 23:16:34 GMT+0 - Monitoring - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up. Jun 12, 00:16:31 GMT+0 - Resolved - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future. Type: Incident

Duration: 2 hours and 17 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Identified</strong> - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results..</p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>23:16:34</var> GMT+0</small><br><strong>Monitoring</strong> - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>00:16:31</var> GMT+0</small><br><strong>Resolved</strong> - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future..</p> ]]> Wed, 11 Jun 2025 22:00:00 +0000 https://status.northflank.com/incident/cmbsjf79f0025k18t40kcbyb0 https://status.northflank.com/incident/cmbsjf79f0025k18t40kcbyb0 Infrastructure stability issues Type: Incident Duration: 1 hour and 10 minutes Affected Components: Networking, Addons, Services May 21, 16:04:00 GMT+0 - Monitoring - May 21, 17:13:33 GMT+0 - Resolved - **We sincerely apologise for the outage you experienced.** **What happened:** Yesterday, we rolled out an update to support ARM architecture across our platform. As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas. This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers. **Timeline of events (UTC):** * **15:36** — Northflank engineers were alerted to failing health checks on customer endpoints. * **15:41** — Elevated 503 errors were detected from a specific cluster. * **15:42** — An internal incident was formally declared. * **15:57** — Replacement infrastructure began provisioning to recover failed nodes. * **16:32** — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention. Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again. Will Stewart CEO @ Northflank Type: Incident

Duration: 1 hour and 10 minutes

Affected Components: , ,

<p><small>May <var data-var='date'> 21</var>, <var data-var='time'>16:04:00</var> GMT+0</small><br><strong>Monitoring</strong> - .</p> <p><small>May <var data-var='date'> 21</var>, <var data-var='time'>17:13:33</var> GMT+0</small><br><strong>Resolved</strong> - **We sincerely apologise for the outage you experienced.** **What happened:** Yesterday, we rolled out an update to support ARM architecture across our platform. As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas. This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers. **Timeline of events (UTC):** * **15:36** — Northflank engineers were alerted to failing health checks on customer endpoints. * **15:41** — Elevated 503 errors were detected from a specific cluster. * **15:42** — An internal incident was formally declared. * **15:57** — Replacement infrastructure began provisioning to recover failed nodes. * **16:32** — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention. Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again. Will Stewart CEO @ Northflank.</p> ]]> Wed, 21 May 2025 16:04:00 +0000 https://status.northflank.com/incident/cmay5h4pg0052zbome4c6m5q6 https://status.northflank.com/incident/cmay5h4pg0052zbome4c6m5q6 US - Central - Inability to Create Disk Snapshots Type: Incident Duration: 1 hour and 17 minutes Affected Components: Addons Nov 8, 19:20:22 GMT+0 - Resolved - This incident has been resolved. Nov 8, 18:02:57 GMT+0 - Monitoring - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region Type: Incident

Duration: 1 hour and 17 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>19:20:22</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>18:02:57</var> GMT+0</small><br><strong>Monitoring</strong> - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region.</p> ]]> Fri, 8 Nov 2024 18:02:57 +0000 https://status.northflank.com/incident/cm391ojva000bzr1411i47iz0 https://status.northflank.com/incident/cm391ojva000bzr1411i47iz0 DNS Propagation Delay Type: Incident Duration: 1 hour and 27 minutes Affected Components: Services, Addons, Certificates Oct 11, 18:41:37 GMT+0 - Monitoring - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. <https://ns1connect.trust.pagerduty.com/posts/details/PK35B1W> Oct 11, 20:09:04 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 27 minutes

Affected Components: , ,

<p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>18:41:37</var> GMT+0</small><br><strong>Monitoring</strong> - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. <https://ns1connect.trust.pagerduty.com/posts/details/PK35B1W>.</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>20:09:04</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 11 Oct 2024 18:41:37 +0000 https://status.northflank.com/incident/cm252qf810036lb09qejd495o https://status.northflank.com/incident/cm252qf810036lb09qejd495o Metrics ingestion partial outage Type: Incident Affected Components: Logs and Metrics Jul 16, 10:46:00 GMT+0 - Resolved - Metrics experienced ingestion issues between 10:46AM -> 11:30AM UTC We implemented a fix and are currently monitoring the result. Type: Incident

Affected Components:

<p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>10:46:00</var> GMT+0</small><br><strong>Resolved</strong> - Metrics experienced ingestion issues between 10:46AM -> 11:30AM UTC We implemented a fix and are currently monitoring the result..</p> ]]> Tue, 16 Jul 2024 10:46:00 +0000 https://status.northflank.com/incident/clyocih3y146600h2n4yyz8ap39 https://status.northflank.com/incident/clyocih3y146600h2n4yyz8ap39 Partial Logs Outage Type: Incident Affected Components: Logs and Metrics Jul 16, 04:00:00 GMT+0 - Resolved - Engineers identified an incident with Northflank's logging backend between 0400UTC and 0700UTC. The problem has been resolved. The impact is log loss between this timeframe. We are still investigating the cause of the issue and will keep you updated and whether logs can be recovered. Type: Incident

Affected Components:

<p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Resolved</strong> - Engineers identified an incident with Northflank's logging backend between 0400UTC and 0700UTC. The problem has been resolved. The impact is log loss between this timeframe. We are still investigating the cause of the issue and will keep you updated and whether logs can be recovered..</p> ]]> Tue, 16 Jul 2024 04:00:00 +0000 https://status.northflank.com/incident/clyoyh9qz173514icoqudznap8n https://status.northflank.com/incident/clyoyh9qz173514icoqudznap8n Networking / Public Ingress / HTTPS Impacted - Europe West Type: Incident Duration: 15 minutes Affected Components: Networking Apr 15, 23:55:30 GMT+0 - Investigating - We are currently investigating this incident. Northflank public ingress for Services and Addons has been partially impacted, resulting in failed requests and increased latency in Northflank's Europe West—London region, which affects Paid projects. Northflank SREs are responding promptly. Apr 16, 00:03:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Apr 16, 00:10:17 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 15 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 15</var>, <var data-var='time'>23:55:30</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. Northflank public ingress for Services and Addons has been partially impacted, resulting in failed requests and increased latency in Northflank's Europe West—London region, which affects Paid projects. Northflank SREs are responding promptly..</p> <p><small>Apr <var data-var='date'> 16</var>, <var data-var='time'>00:03:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Apr <var data-var='date'> 16</var>, <var data-var='time'>00:10:17</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 15 Apr 2024 23:55:30 +0000 https://status.northflank.com/incident/clv1m4m4w134085b6ohx7l2euwa https://status.northflank.com/incident/clv1m4m4w134085b6ohx7l2euwa Issue with creating DNS entries for services and addons Type: Incident Duration: 1 hour and 1 minute Affected Components: Services, Addons Nov 8, 16:00:00 GMT+0 - Monitoring - There is an ongoing incident with our DNS provider (NS1) which is causing issues during provisioning of DNS entries for newly created services and addons. https://ns1status.com/#!/incident/507854 Existing services and addons are not affected. Nov 8, 17:01:01 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 1 minute

Affected Components: ,

<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - There is an ongoing incident with our DNS provider (NS1) which is causing issues during provisioning of DNS entries for newly created services and addons. https://ns1status.com/#!/incident/507854 Existing services and addons are not affected..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>17:01:01</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 8 Nov 2023 16:00:00 +0000 https://status.northflank.com/incident/clopz09mh1122c6o9ppaas7m7 https://status.northflank.com/incident/clopz09mh1122c6o9ppaas7m7 Intermittent Networking Issues Type: Incident Duration: 40 minutes Affected Components: Networking Aug 4, 14:09:40 GMT+0 - Resolved - This incident has been resolved. Aug 4, 13:29:12 GMT+0 - Investigating - We are currently investigating intermittent networking connectivity issues. Type: Incident

Duration: 40 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>14:09:40</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>13:29:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating intermittent networking connectivity issues. .</p> ]]> Fri, 4 Aug 2023 13:29:12 +0000 https://status.northflank.com/incident/clkwmgyc75779bbn85rj388vq https://status.northflank.com/incident/clkwmgyc75779bbn85rj388vq Logs Backend Impact Type: Incident Duration: 2 days and 12 hours Affected Components: Logs and Metrics Jun 16, 12:41:00 GMT+0 - Identified - Northflank's logging backend is experiencing performance and connectivity issues. We are investigating, and engineers are implementing a fix. Users may see intermittent log query failures, and live tailing issues in the UI, CLI and API. Jun 16, 14:21:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 days and 12 hours

Affected Components:

<p><small>Jun <var data-var='date'> 16</var>, <var data-var='time'>12:41:00</var> GMT+0</small><br><strong>Identified</strong> - Northflank's logging backend is experiencing performance and connectivity issues. We are investigating, and engineers are implementing a fix. Users may see intermittent log query failures, and live tailing issues in the UI, CLI and API..</p> <p><small>Jun <var data-var='date'> 16</var>, <var data-var='time'>14:21:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 16 Jun 2023 12:41:00 +0000 https://status.northflank.com/incident/cliymbn74112991b9ocbkt6b7ql https://status.northflank.com/incident/cliymbn74112991b9ocbkt6b7ql *.code.run dangerous security warning Type: Incident Duration: 1 hour and 21 minutes Affected Components: Networking May 11, 11:49:50 GMT+0 - Identified - Google’s Safe Browsing has marked *.code.run as a dangerous domain. This impacts all public HTTPS endpoints that do not use a custom domain. We recommend appealing for your specific service domains at [https://safebrowsing.google.com/safebrowsing/report_error/](https://safebrowsing.google.com/safebrowsing/report_error/). We have sent the relevant documentation via Google Search Console and await a response from Google. We will explore options including: - temporarily using another domain for new workloads - exploring wildcard and vanity replacement of *.code.run on a per-user basis - adding *.code.run to an allow-list Cause: A user was caught deploying a phishing website and was banned within 10 minutes of notification. *.code.run is a member of the PSL, and, unfortunately, it is not considered when blocking thousands of domains for one offending workload. May 11, 13:11:15 GMT+0 - Resolved - It appears the warning has been lifted for *.code.run domains and are now working as expected. We continue to monitor the situation closely. We apologise for any inconvenience. Type: Incident

Duration: 1 hour and 21 minutes

Affected Components:

<p><small>May <var data-var='date'> 11</var>, <var data-var='time'>11:49:50</var> GMT+0</small><br><strong>Identified</strong> - Google’s Safe Browsing has marked *.code.run as a dangerous domain. This impacts all public HTTPS endpoints that do not use a custom domain. We recommend appealing for your specific service domains at [https://safebrowsing.google.com/safebrowsing/report_error/](https://safebrowsing.google.com/safebrowsing/report_error/). We have sent the relevant documentation via Google Search Console and await a response from Google. We will explore options including: - temporarily using another domain for new workloads - exploring wildcard and vanity replacement of *.code.run on a per-user basis - adding *.code.run to an allow-list Cause: A user was caught deploying a phishing website and was banned within 10 minutes of notification. *.code.run is a member of the PSL, and, unfortunately, it is not considered when blocking thousands of domains for one offending workload..</p> <p><small>May <var data-var='date'> 11</var>, <var data-var='time'>13:11:15</var> GMT+0</small><br><strong>Resolved</strong> - It appears the warning has been lifted for *.code.run domains and are now working as expected. We continue to monitor the situation closely. We apologise for any inconvenience..</p> ]]> Thu, 11 May 2023 11:49:50 +0000 https://status.northflank.com/incident/clhj2hrjr1525823symzpj0ntbei https://status.northflank.com/incident/clhj2hrjr1525823symzpj0ntbei Certificate Generation Delayed Type: Incident Duration: 1 hour and 6 minutes Affected Components: Certificates, Addons Jan 12, 11:56:47 GMT+0 - Identified - We have identified an issue with our DNS provider with delayed propagation of DNS entries. This is delaying custom domain certificate creation, new public code.run endpoints and addon creation. We are monitoring and will update when we have more updates. Jan 12, 13:02:59 GMT+0 - Resolved - DNS is performing as expected, you can create domains and databases in a timely manor. NS1 DNS incident has been resolved: https://ns1status.com/#!/incident/400072 Type: Incident

Duration: 1 hour and 6 minutes

Affected Components: ,

<p><small>Jan <var data-var='date'> 12</var>, <var data-var='time'>11:56:47</var> GMT+0</small><br><strong>Identified</strong> - We have identified an issue with our DNS provider with delayed propagation of DNS entries. This is delaying custom domain certificate creation, new public code.run endpoints and addon creation. We are monitoring and will update when we have more updates..</p> <p><small>Jan <var data-var='date'> 12</var>, <var data-var='time'>13:02:59</var> GMT+0</small><br><strong>Resolved</strong> - DNS is performing as expected, you can create domains and databases in a timely manor. NS1 DNS incident has been resolved: https://ns1status.com/#!/incident/400072.</p> ]]> Thu, 12 Jan 2023 11:56:47 +0000 https://status.northflank.com/incident/clct1cc7c11147ian3ryx2ydnu https://status.northflank.com/incident/clct1cc7c11147ian3ryx2ydnu Certificate Generation Delayed Type: Incident Duration: 9 hours and 1 minute Affected Components: Certificates, Addons Nov 29, 19:32:44 GMT+0 - Identified - We have identified an issue with our DNS provider with delayed propagation of DNS entries. This is delaying custom domain certificate creation, new public code.run endpoints and addon creation. We are monitoring and will update when we have more updates. Nov 29, 23:28:17 GMT+0 - Monitoring - NS1 continues to work on mitigations and a fix. https://ns1status.com/#!/incident/390630 Nov 30, 04:34:00 GMT+0 - Resolved - Our DNS provider has resolved the propagation issue. https://ns1status.com/#!/incident/390630 Systems are operational. Type: Incident

Duration: 9 hours and 1 minute

Affected Components: ,

<p><small>Nov <var data-var='date'> 29</var>, <var data-var='time'>19:32:44</var> GMT+0</small><br><strong>Identified</strong> - We have identified an issue with our DNS provider with delayed propagation of DNS entries. This is delaying custom domain certificate creation, new public code.run endpoints and addon creation. We are monitoring and will update when we have more updates..</p> <p><small>Nov <var data-var='date'> 29</var>, <var data-var='time'>23:28:17</var> GMT+0</small><br><strong>Monitoring</strong> - NS1 continues to work on mitigations and a fix. https://ns1status.com/#!/incident/390630.</p> <p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>04:34:00</var> GMT+0</small><br><strong>Resolved</strong> - Our DNS provider has resolved the propagation issue. https://ns1status.com/#!/incident/390630 Systems are operational..</p> ]]> Tue, 29 Nov 2022 19:32:44 +0000 https://status.northflank.com/incident/clb2m97el72586toerg4esxuz https://status.northflank.com/incident/clb2m97el72586toerg4esxuz Log Ingestion and Log sinks are delayed Type: Incident Duration: 1 hour and 7 minutes Affected Components: Logs and Metrics Nov 17, 12:23:15 GMT+0 - Investigating - We are currently investigating this incident. Nov 17, 12:55:31 GMT+0 - Monitoring - We implemented a fix and currently monitoring the result. Nov 17, 13:29:54 GMT+0 - Resolved - The issue is now resolved. All logging and log sink ingestion is successfully being processed and doing so performantly. Type: Incident

Duration: 1 hour and 7 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>12:23:15</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>12:55:31</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and currently monitoring the result..</p> <p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>13:29:54</var> GMT+0</small><br><strong>Resolved</strong> - The issue is now resolved. All logging and log sink ingestion is successfully being processed and doing so performantly..</p> ]]> Thu, 17 Nov 2022 12:23:15 +0000 https://status.northflank.com/incident/clal1mnrm66758hdoc770jqkhj https://status.northflank.com/incident/clal1mnrm66758hdoc770jqkhj