Incident Report

Summary

On Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes.

We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future.

Root Cause

The disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.

The primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in "hotspotting," where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes.

The failure sequence was as follows:

1. Technical Trigger: The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded.
2. Systemic Effect: This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a "hot shard" condition.
3. Contention and Latency: As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly.
4. Failure State: The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%.

Automated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent.

Remediation and Prevention

The issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:

• Throttling: Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover.
• Index Redistribution: Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards.
• Cache Policy Update: Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.
• Deadline Adjustment: Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.

Prevention

To prevent a recurrence, Google engineering will be implementing the following actions:

• Index Design Standards: Restructure of database index to prevent "hotspotting".
• Adaptive Cache Tuning: Implement more robust cache policies that prevent load amplification during periods of backend stress.
• Enhanced Hotspot Monitoring: Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic.
• Service Resiliency: Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads.

Detailed Description of Impact

The impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026.

Gemini App (Web and Mobile):

• High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving "Something went wrong" messages .
• Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers .
• Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the "Add from Drive" functionality to be visually disabled and unresponsive following interaction with the "+" prompt.

Integrations:

• Google Chrome Integration: The Gemini Side Panel and integrated conversational features encountered persistent timeouts.

Preliminary Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

Date/Time of the Issue (All time US/Pacific)

Incident Start: 10 June 2026 03:20

Incident End: 10 June 2026 10:25

Duration: 7 hours, 5 minutes

Summary

On Wednesday, 10 June 2026 03:20 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for 7 hours and 5 minutes.

Preliminary Root Cause

The disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata.

The primary cause was back-pressure on specific database replicas that experienced high loads. The issue was made worse because of an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with a similar fixed value. This resulted in a condition where traffic was concentrated on a very small number of database instances, overwhelming them. Additionally, the service utilized an in-memory cache with a short Time To Live (TTL) of only one minute, requiring frequent database refreshes.

The failure sequence was as follows:

1. Technical Trigger: The database service became overloaded, causing back-pressure on specific replicas.
2. Systemic Effect: An application level caching layer configuration is set to a short 1-minute expiration window for deployed tools. This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition. The failed DB queries resulted in failure to populate application level cache.
3. Contention and Latency: As request volume increased, the internal tool management service experienced >10x surge in database calls because the cache hit rate dropped significantly.
4. Failure State: The underlying storage layer reached its transaction limits, with database response success rates dropping as low as 10% on the affected partitions.

Remediation

The issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service:

• Throttling: Applied rate limiting to the most affected database instances to allow internal caches to recover.
• Index Redistribution: Gradually redistributed fixed index values in the database to random values in a specific range, effectively spreading the load across multiple instances.
• Cache Policy Update: Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load.
• Deadline Adjustment: Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency.

Description of Impact

The impact window occurred between 03:20 US/Pacific and 10:25 US/Pacific on 10 June 2026.

Gemini App (Web and Mobile):

• Users experienced a 50% error rate when attempting to send prompts, receiving "Something went wrong" messages. The issue affected both consumer accounts and enterprise Google Workspace customers .

Integrations:

• Google Chrome Integration: The Gemini Side Panel and integrated conversational features encountered persistent timeouts.

Title

Gemini App in Workspace users were experiencing “Something Went Wrong” errors.

Description

The issue with Gemini has been resolved for all the affected users as of Wednesday, 2026-06-10 10:30 US/Pacific.

From preliminary analysis, the issue was triggered by a performance issue in our backend database which impacted the retrieval of Gemini App tools catalog.

Our engineers mitigated the issue by optimizing the load distribution across our backend database fleet.

We thank you for your patience while we worked on resolving the issue.

Symptoms

Affected users may have seen "Something went wrong" errors (codes 1099, 1076) when prompting Gemini while using the following services.

Gemini App in Workspace includes the following platforms:

• Web, MacOS, iOS and Android
• Gemini in Chrome

Workaround

None as the issue is now resolved.

Title

Gemini App in Workspace customers are experiencing “Something Went Wrong” errors.

Description

We are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.

Our engineering team has applied mitigations to reduce impact and confirmed the majority of users are no longer observing impact as of Wednesday, 2026-06-10 10:30 PDT. We will continue to monitor the service for stability.

We will provide an update by Wednesday, 2026-06-10 20:30 PDT with current details.

Symptoms

The following services are experiencing “Something Went Wrong 1099 or 1076” errors:

Gemini app in Workspace includes the following platforms:

• Web, MacOS, iOS and Android
• Gemini in Chrome

Workaround

None at this time.

Title

Gemini App in Workspace customers are experiencing “Something Went Wrong” errors.

Description

We are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.

Our engineering team has applied mitigations to reduce impact and continues to investigate the root cause. We are seeing signs of recovery and will continue to monitor progress.

We will provide an update by Wednesday, 2026-06-10 14:30 PDT with current details.

Symptoms

The following services are experiencing “Something Went Wrong 1099 or 1076” errors:

Gemini app in Workspace includes the following platforms:

• Web, MacOS, iOS and Android
• Gemini in Chrome

Workaround

None at this time.

Title

Gemini App in Workspace customers are experiencing “Something Went Wrong” errors.

Description

We are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.

Our engineering team has identified a mitigation which is currently in progress. We do not have an ETA for mitigation at this time.

We will provide an update by Wednesday, 2026-06-10 12:30 PDT with current details.

Symptoms

The following services are experiencing “Something Went Wrong 1099 or 1076” errors:

Gemini app in Workspace includes the following platforms:

• Web, MacOS, iOS and Android
• Gemini in Chrome

Workaround

None at this time.

Title

Gemini App in Workspace customers are experiencing “Something Went Wrong” errors.

Description

We are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.

Our engineering team continues to investigate the issue and identify a mitigation strategy.

We will provide an update by Wednesday, 2026-06-10 09:30 PDT with current details.

Symptoms

The following services are experiencing “Something Went Wrong” errors.:

• Gemini app in Workspace includes the following platforms;
• Web, MacOS, iOS and Android
• Gemini in Chrome

Workaround

None at this time.

Title

Gemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.

Description

We are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2026-06-10 08:00 PDT with current details.

We apologize to all who are affected by the disruption.

Symptoms

Gemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.

Workaround

None at this time.

Title

Gemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.

Description

We are experiencing an issue with Gemini beginning on Wednesday, 2026-06-10 03:26 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2026-06-10 06:30 PDT with current details.

We apologize to all who are affected by the disruption.

Symptoms

Gemini App in Workspace and Gemini Side Panel (Gemini in Workspace) customers are experiencing “Something Went Wrong” errors.

Workaround

None at this time.