In summary, the underlying issue was with GCP Authentication APIs. During the incident, all Zenoss Cloud customers would have been unable to log in for approximately 1 to 3 hours depending on which region their Zenoss Cloud incident is hosted in. CZ functionality and CZ monitoring were unaffected, other than some minor Cloud layer data ingest delays for some CZs. These minor delays were fully resolved by morning of Friday June 13th.
Posted Jun 13, 2025 - 11:43 CDT
Resolved
All Zenoss Cloud instances are fully functional. There is a minor backlog on metric ingest in us-central1 production instance that is still clearing and we expect that to be cleared this evening. We'll post another update with summary of the incident tomorrow morning. Please open a support ticket if you are having any issues at this point.
Posted Jun 12, 2025 - 17:50 CDT
Update
Zenoss Cloud instances in Iowa are now functioning normally again as Google's recovery in us-central1 progresses. There may still be some delays to data ingest to the Cloud layer and Data Access but all CZs are operating normally.
Posted Jun 12, 2025 - 15:27 CDT
Update
GCP status update indicates that all regions have recovered except for us-central1. As such, Zenoss Cloud customers in EMEA, Australia and in the Vegas US region should now have Zenoss Cloud full functionality restored. Customers in the Iowa US region and all customers dev environments may still be having issues as google continues recovery in their us-central1 region.