At 18:39 UTC on 26 February 2025, we received a monitoring alert for a possible issue with Managed Identities for Azure resources. Subsequently, communications were sent to customers, notifying them of this possible issue.
Upon further investigation during our post-incident review, we have determined that a significant percentage of those notified were not impacted by this event. We apologize for any confusion or inconvenience this may have caused.
If you were not impacted, please disregard the previous notification. We are committed to ensuring the accuracy of our communications and will continue to improve our processes and tooling to prevent such false notifications in the future.
For those customers who were impacted, you will receive subsequent messaging with the final Post Incident Review (PIR).
What happened?
Between 18:39 and 20:55 UTC on 26 February 2025, we experienced an issue which resulted in an impact for customers being unable to perform control plane operations related to Azure Managed Identity. This included impact to the following services: Azure Container Apps, Azure SQL, Azure SQL Managed Instance, Azure Front Door, Azure Resource Manager, Azure Synapse Analytics, Azure Data Bricks, Azure Chaos Studio, Azure App Services, Azure Logic Apps, Azure Media Services, MSFT Power BI and Azure Service Bus.
What do we know so far?
We identified an issue with our Managed Identity infrastructure related to a key rotation. We performed manual steps to repair the key in each region, which resolved the issue.
How did we respond?
What happens next?
Impact Statement: Starting at 16:48 UTC on 26 February 2025, you have been identified as a customer using Managed Identities who may be unable to create, update, delete, scale-up Azure resources using Managed Identities, and/or request tokens in some cases. Chaos customers may also not have been able to create or run experiments.
Current Status: We have identified the issue and have begun to roll out a fix region-by-region. The regions where customers should see mitigation are Central US, North Europe, West US, UK West, West Europe, East US, East US 2, Korea Central, Canada Central, West US 2, Australia Central, Australia East, Japan East, Sweden Central, UK South, South Central US, Southeast Asia, West US 3, UAE Central, West Central US, Canada East, Brazil South, Central India, France Central, Germany West Central, North Central US, UAE North, Switzerland North, South India, Australia South East, Norway East, Italy North, Korea South, Switzerland West, Sweden South, South Africa North, Mexico Central, Norway West, South Africa West, Israel Central, Poland Central, Jio India West, West India, France South, Germany North, Australia Central, Brazil Southeast, Jio India Central.
Impact Statement: Starting at 16:48 UTC on 26 February 2025, you have been identified as a customer using Managed Identities who may be unable to create, update, delete, scale-up Azure resources using Managed Identities, and/or request tokens in some cases. Chaos customers may also not have been able to create or run experiments.
Current Status: We are currently investigating this issue and suspect it is related to a certificate. We will provide additional information as it becomes available. The next update will be provided in 60 minutes, or as events warrant.
Between 18:39 and 20:55 UTC on 26 February 2025, we experienced an issue which resulted in an impact for customers being unable to perform control plane operations related to Azure Managed Identity. This included impact to the following services: Azure Container Apps, Azure SQL, Azure SQL Managed Instance, Azure Front Door, Azure Resource Manager, Azure Synapse Analytics, Azure Data Bricks, Azure Chaos Studio, Azure App Services, Azure Logic Apps, Azure Media Services, MSFT Power BI and Azure Service Bus.
Information on steps taken to mitigate this incident will be provided shortly.
Impact Statement: Starting at 16:48 UTC on 26 February 2025, you have been identified as a customer using Managed Identities who may be unable to create, update, delete, scale-up Azure resources using Managed Identities, and/or request tokens in some cases. Chaos customers may also not have been able to create or run experiments.
Current Status: We are currently investigating this issue and suspect it is related to a certificate. We will provide additional information as it becomes available. The next update will be provided in 60 minutes, or as events warrant.