Duration: 3 hours and 21 minutes

Affected Components: Addons, Jobs, Services

Mar 25, 14:28:04 GMT+0
Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations..

Mar 25, 16:01:00 GMT+0
Monitoring - The region is currently stable. We are still actively mitigating the ongoing attack..

Mar 25, 17:48:48 GMT+0
Resolved - The DDoS attack has stopped we will continue to monitor the affected region..

Duration: 34 minutes

Affected Components: Northflank App

Mar 16, 18:16:28 GMT+0
Resolved - This incident has been resolved..

Mar 16, 18:15:58 GMT+0
Monitoring - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation..

Mar 16, 17:42:54 GMT+0
Identified - We are investing an incident where the UI is failing to load correctly..

Duration: 29 minutes

Affected Components: Networking, Addons, Services

Mar 9, 12:06:32 GMT+0
Investigating - We are currently investigating this incident..

Mar 9, 12:35:56 GMT+0
Resolved - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads..

Duration: 1 hour and 34 minutes

Affected Components: Builds

Feb 19, 15:00:00 GMT+0
Identified - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released..

Feb 19, 16:34:09 GMT+0
Resolved - We have released a fix for the issue..

Duration: 39 minutes

Affected Components: Networking, , Addons, Jobs, Services, Northflank Platform →

Dec 11, 13:54:32 GMT+0
Identified - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation..

Dec 11, 14:13:28 GMT+0
Monitoring - The mitigation is in place and the region is operating as expected..

Dec 11, 14:33:03 GMT+0
Resolved - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution..

Duration: 1 hour and 44 minutes

Affected Components: Addons, Jobs, Services

Sep 24, 23:53:50 GMT+0
Identified - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. <https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9>.

Sep 25, 01:11:08 GMT+0
Monitoring - We are observing that image pull requests for Docker Hub images are now succeeding..

Sep 25, 01:38:09 GMT+0
Resolved - Dockerhub is fully operational and image pulling is now working as expected..

Duration: 18 hours and 29 minutes

Affected Components: Addons, Services, Certificates

Sep 16, 15:09:51 GMT+0
Monitoring - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: <https://status.ai-apps-comms.ibm.com/ns1connect>.

Sep 17, 09:39:01 GMT+0
Resolved - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour..

Duration: 1 hour and 17 minutes

Affected Components: Builds

Aug 25, 21:48:19 GMT+0
Identified - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this..

Aug 25, 22:45:24 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 25, 23:04:54 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour

Affected Components: Logs and Metrics

Aug 5, 07:23:59 GMT+0
Investigating - We are currently investigating this incident..

Aug 5, 07:55:18 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 5, 08:23:56 GMT+0
Resolved - This incident has been resolved..

Duration: 11 hours and 31 minutes

Affected Components: Certificates

Jul 21, 19:13:49 GMT+0
Monitoring - [Let's Encrypt is currently experiencing an outage](https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/687e8d62b8a4e804fad85799) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons or domains..

Jul 22, 06:44:30 GMT+0
Resolved - Certificate generation is functioning without issue. We will continue to monitor the situation..

Duration: 2 hours and 45 minutes

Affected Components: Builds, Addons, Logs and Metrics

Jun 12, 20:40:23 GMT+0
Monitoring - We are observing improved operations in the US - Central region. Addon creation and backup issues are now resolved. Ongoing Issues: \* GCP BYOC cluster processing.

Jun 12, 18:00:00 GMT+0
Identified - We have noticed issues with Google Cloud APIs. This seems to be widespread across the Google Cloud platform. We are currently determining the impact of the outage, and we will provide updates when we know more. Currently, logs, metrics, builds, BYOC and addon backups may be affected..

Jun 12, 19:21:45 GMT+0
Monitoring - Google Cloud has acknowledged the incident. For detailed information, please visit: <https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW> Most services have been restored to normal operation. However, we are still experiencing the following issues: * Addon creation in the US - Central region * Addon backups in the US - Central region * GCP BYOC cluster processing We will continue monitoring the situation and provide updates as they become available..

Jun 12, 20:44:39 GMT+0
Resolved - GCP BYOC cluster processing is now operational..

Duration: 2 hours and 17 minutes

Affected Components: Logs and Metrics

Jun 11, 22:00:00 GMT+0
Identified - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results..

Jun 11, 23:16:34 GMT+0
Monitoring - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up..

Jun 12, 00:16:31 GMT+0
Resolved - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future..

Duration: 1 hour and 10 minutes

Affected Components: Networking, Addons, Services

May 21, 16:04:00 GMT+0
Monitoring - .

May 21, 17:13:33 GMT+0
Resolved - **We sincerely apologise for the outage you experienced.** **What happened:** Yesterday, we rolled out an update to support ARM architecture across our platform. As part of that, a migration was initiated today to extend support to x86 architecture within our database schemas. This triggered a higher-than-expected volume of database rescheduling events, which in turn put excessive pressure on the underlying GCP infrastructure. Some nodes failed under this load, compounding the issue and causing degraded service for several customers. **Timeline of events (UTC):** * **15:36** — Northflank engineers were alerted to failing health checks on customer endpoints. * **15:41** — Elevated 503 errors were detected from a specific cluster. * **15:42** — An internal incident was formally declared. * **15:57** — Replacement infrastructure began provisioning to recover failed nodes. * **16:32** — All customer addons returned to a healthy state, with the exception of four databases which required manual intervention. Thank you for your patience—we know how critical uptime is, and we are reviewing and improving our systems to ensure this doesn’t happen again. Will Stewart CEO @ Northflank.

Duration: 1 hour and 17 minutes

Affected Components: Addons

Nov 8, 19:20:22 GMT+0
Resolved - This incident has been resolved..

Nov 8, 18:02:57 GMT+0
Monitoring - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region.

Duration: 1 hour and 27 minutes

Affected Components: Services, Addons, Certificates

Oct 11, 18:41:37 GMT+0
Monitoring - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. <https://ns1connect.trust.pagerduty.com/posts/details/PK35B1W>.

Oct 11, 20:09:04 GMT+0
Resolved - This incident has been resolved..

Affected Components: Logs and Metrics

Jul 16, 10:46:00 GMT+0
Resolved - Metrics experienced ingestion issues between 10:46AM -> 11:30AM UTC We implemented a fix and are currently monitoring the result..

Affected Components: Logs and Metrics

Jul 16, 04:00:00 GMT+0
Resolved - Engineers identified an incident with Northflank's logging backend between 0400UTC and 0700UTC. The problem has been resolved. The impact is log loss between this timeframe. We are still investigating the cause of the issue and will keep you updated and whether logs can be recovered..

Duration: 15 minutes

Affected Components: Networking

Apr 15, 23:55:30 GMT+0
Investigating - We are currently investigating this incident. Northflank public ingress for Services and Addons has been partially impacted, resulting in failed requests and increased latency in Northflank's Europe West—London region, which affects Paid projects. Northflank SREs are responding promptly..

Apr 16, 00:03:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 16, 00:10:17 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 1 minute

Affected Components: Services, Addons

Nov 8, 16:00:00 GMT+0
Monitoring - There is an ongoing incident with our DNS provider (NS1) which is causing issues during provisioning of DNS entries for newly created services and addons. https://ns1status.com/#!/incident/507854 Existing services and addons are not affected..

Nov 8, 17:01:01 GMT+0
Resolved - This incident has been resolved..

Duration: 40 minutes

Affected Components: Networking

Aug 4, 14:09:40 GMT+0
Resolved - This incident has been resolved..

Aug 4, 13:29:12 GMT+0
Investigating - We are currently investigating intermittent networking connectivity issues. .

Duration: 2 days and 12 hours

Affected Components: Logs and Metrics

Jun 16, 12:41:00 GMT+0
Identified - Northflank's logging backend is experiencing performance and connectivity issues. We are investigating, and engineers are implementing a fix. Users may see intermittent log query failures, and live tailing issues in the UI, CLI and API..

Jun 16, 14:21:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 21 minutes

Affected Components: Networking

May 11, 11:49:50 GMT+0
Identified - Google’s Safe Browsing has marked *.code.run as a dangerous domain. This impacts all public HTTPS endpoints that do not use a custom domain. We recommend appealing for your specific service domains at [https://safebrowsing.google.com/safebrowsing/report_error/](https://safebrowsing.google.com/safebrowsing/report_error/). We have sent the relevant documentation via Google Search Console and await a response from Google. We will explore options including: - temporarily using another domain for new workloads - exploring wildcard and vanity replacement of *.code.run on a per-user basis - adding *.code.run to an allow-list Cause: A user was caught deploying a phishing website and was banned within 10 minutes of notification. *.code.run is a member of the PSL, and, unfortunately, it is not considered when blocking thousands of domains for one offending workload..

May 11, 13:11:15 GMT+0
Resolved - It appears the warning has been lifted for *.code.run domains and are now working as expected. We continue to monitor the situation closely. We apologise for any inconvenience..

Duration: 1 hour and 6 minutes

Affected Components: Certificates, Addons

Jan 12, 11:56:47 GMT+0
Identified - We have identified an issue with our DNS provider with delayed propagation of DNS entries. This is delaying custom domain certificate creation, new public code.run endpoints and addon creation. We are monitoring and will update when we have more updates..

Jan 12, 13:02:59 GMT+0
Resolved - DNS is performing as expected, you can create domains and databases in a timely manor. NS1 DNS incident has been resolved: https://ns1status.com/#!/incident/400072.

Duration: 9 hours and 1 minute

Affected Components: Certificates, Addons

Nov 29, 19:32:44 GMT+0
Identified - We have identified an issue with our DNS provider with delayed propagation of DNS entries. This is delaying custom domain certificate creation, new public code.run endpoints and addon creation. We are monitoring and will update when we have more updates..

Nov 29, 23:28:17 GMT+0
Monitoring - NS1 continues to work on mitigations and a fix. https://ns1status.com/#!/incident/390630.

Nov 30, 04:34:00 GMT+0
Resolved - Our DNS provider has resolved the propagation issue. https://ns1status.com/#!/incident/390630 Systems are operational..

Duration: 1 hour and 7 minutes

Affected Components: Logs and Metrics

Nov 17, 12:23:15 GMT+0
Investigating - We are currently investigating this incident..

Nov 17, 12:55:31 GMT+0
Monitoring - We implemented a fix and currently monitoring the result..

Nov 17, 13:29:54 GMT+0
Resolved - The issue is now resolved. All logging and log sink ingestion is successfully being processed and doing so performantly..