Summary
On Tuesday, 22 March 2022, Google Cloud Networking experienced congestion on network infrastructure to and from the network edge locations [1] in Queretaro, Mexico, for a duration of 1 hour and 45 minutes, following multiple fiber cuts between the United States and Mexico. Affected customers may have experienced high latency, high retransmits and elevated errors.
We would like to apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.
[1] - https://cloud.google.com/vpc/docs/edge-locations
Root Cause
Google’s wide-area network is primarily comprised of infrastructure owned and operated by Google, although some physical links use fiber infrastructure which is owned and operated by third parties. Google has multiple resilient fiber paths and network points of presence serving users in Mexico.
Multiple simultaneous fiber cuts on diversely-routed paths, at the same time as a planned maintenance event on another diversely-routed path, resulted in a capacity shortfall in Google’s wide-area network between Queretaro, Mexico and the United States. This multiple failure event represents a rare case in which the resulting capacity shortfall was visible to Google's users.
Traffic was subsequently redirected to alternate links; however, there was insufficient capacity to serve all the redirected traffic, resulting in packet loss for user-facing network traffic to Google from users in Mexico.
Remediation and Prevention
Google’s automated repair mechanism detected the congestion on Tuesday, 22 March 2022, at 15:42 US/Pacific and redirected the traffic through alternate edge locations. However, the alternate link did not have sufficient capacity to serve all the redirected traffic, resulting in packet loss. The congestion was cleared when traffic was manually routed around the failed links on Tuesday, 22 March 2022, at 17:15 US/Pacific.
Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. However, this incident was the result of multiple concurrent failures on links which are designed to provide resilience; such failures, whilst very rare, are expected at a low frequency. We are taking the following steps to ensure we respond quickly to similar events in the future:
- Improving the reliability of tooling that manually reroutes traffic in extreme multiple-failure scenarios.
- Improving the diagnostic systems that help Google's Cloud Networking teams decide how to deal with extreme multiple-failure scenarios.
We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.
Detailed Description of Impact
On Tuesday, 22 March 2022 from 15:30 to 17:15 US/Pacific, customers reaching Google services from Mexico may have experienced network slowness and packet loss, and errors returned from our services. This may have resulted in customers not being able to access services running on Google Cloud and Workspace services including:
- Google Meet
- Gmail
- Google Drive
We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.
(All Times US/Pacific)
Incident Start: 22 March 2022 15:30
Incident End: 22 March 2022 17:15
Duration: 1 hour, 45 minutes
Affected Services and Features:
Google Cloud Networking - Cloud VPN, Cloud Interconnect, Cloud Load Balancers
Workspace Products - Gmail, Meet, Drive and Other Workspace products
Regions/Zones: Mexico
Description:
Google Cloud Networking experienced congestion on network infrastructure to and from the network edge locations in Queretaro, Mexico, for a duration of 1 hour and 45 minutes, following a fiber cut between the United States & Mexico.
Google’s automated repair mechanism detected the congestion and redirected the traffic through alternate edge locations. However, the alternate link did not have sufficient capacity to serve all the redirected traffic, resulting in packet loss for user-facing network traffic to Google. When traffic was manually routed to take other paths, the congestion was cleared.
Customer Impact:
Customers reaching Google services, including GCP and Workspace, from Mexico may have experienced high latency, high packet loss, high retransmits, and errors returned from our services, resulting in customers not being able to access those services.
The issue with Cloud Interconnect, Cloud Load Balancing, Cloud NAT, Cloud Networking, Cloud Router, Cloud VPN has been resolved for all affected users as of Tuesday, 2022-03-22 17:15 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
Summary: Customers connecting from Mexico and Central America may experience issues with accessing GCP services
Description: Mitigation work is currently underway by our engineering team.
We do not have an ETA for mitigation at this point.
We will provide more information by Tuesday, 2022-03-22 18:30 US/Pacific.
Diagnosis: Customers may experience delays in accessing GCP services from Mexico and Central America.
Workaround: None at this time.
Summary: Customers connecting from South-Central America may experience issues with accessing GCP services
Description: We are experiencing an issue with Cloud Interconnect, Cloud Load Balancing, Cloud NAT, Cloud Networking, Cloud Router, Cloud VPN beginning at Tuesday, 2022-03-22 16:15 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Tuesday, 2022-03-22 17:50 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Customers may experience delays in accessing GCP services from South-Central America.
Workaround: None at this time.
Summary: Customers connecting from South-Central America may experience issues with accessing GCP services
Description: We are experiencing an issue with Cloud Interconnect, Cloud Load Balancing, Cloud NAT, Cloud Networking, Cloud Router, Cloud VPN beginning at Tuesday, 2022-03-22 16:15 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Tuesday, 2022-03-22 20:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Customers may experience delays in accessing GCP services from South-Central America.
Workaround: None at this time.
