Northflank - Node infrastructure stability - London Region – Incident details

All systems operational

Node infrastructure stability - London Region

Resolved
Partial outage 30 %
Started about 2 months agoLasted 39 minutes

Affected

Northflank Platform

Partial outage from 1:54 PM to 2:33 PM

Services

Partial outage from 1:54 PM to 2:33 PM

Addons

Partial outage from 1:54 PM to 2:33 PM

Jobs

Partial outage from 1:54 PM to 2:33 PM

Networking

Partial outage from 1:54 PM to 2:33 PM

Updates
  • Resolved
    Resolved

    The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution.

  • Monitoring
    Monitoring

    The mitigation is in place and the region is operating as expected.

  • Identified
    Identified

    There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation.