User indexing caught up with real time around 11:00 UTC. The incident is resolved.
Posted May 14, 2021 - 08:19 PDT
Update
User indexing continues to make great progress indexing the backlog from earlier today. We will provide our next update in 12 hours.
Posted May 13, 2021 - 21:59 PDT
Update
Users search functionality continues to be operational. The Users search datastore currently reflects that data we received roughly 5 hours ago, when the incident began.
User indexing is proceeding and may take several hours to catch up with real time. We will provide our next update in 4 hours.
Posted May 13, 2021 - 17:28 PDT
Monitoring
We have re-enabled user indexing. We will continue to monitor the health of the Users search functionality.
Posted May 13, 2021 - 16:55 PDT
Update
After a brief unavailability of Users search functionality, the snapshot restore has completed. User indexing is still paused.
Posted May 13, 2021 - 16:35 PDT
Update
The Users dashboard and attribute search may be temporarily unavailable as we proceed with restoring a data snapshot.
Posted May 13, 2021 - 15:49 PDT
Identified
We have identified data corruption on a shard of the datastore backing the Users search functionality. We are initiating a restore of a snapshot.
User indexing was automatically paused at the time the incident started. We will be able to process received data after the snapshot is restored. The Users search functionality is currently operational but may be momentarily unavailable during a part of the restore process.