We believe that the audit log restoration is successful and are declaring this incident resolved.
There will be no further scheduled communications regarding this incident.
Sep 6, 23:51 PDT
Audit log functionality operationally normal. Will continue monitoring until event recovery is complete.
Next update in 12 hours barring change in situation
Sep 6, 12:06 PDT
Audit log display is working properly. Cluster is being reindexed to ensure all events appear correctly. We will continue to monitor and the incident will be closed as soon as reindexing is completed.
Sep 6, 00:13 PDT
Audit logs display is working properly, cluster is being reindexed to ensure all events appear correctly.
Next update will be in 12 hours barring change in status.
Sep 5, 12:03 PDT
Engineers have implemented a fix to the audit log systems and are monitoring systems for health.
During this time, audit logs are collected, but access may be degraded as the ElasticSearch service recovers.
The next communication will be in 12 hours.
Sep 4, 23:27 PDT
As of 12:20pm PST, our engineers have discovered a degradation in service with one of the ElasticSearch clusters that powers the audit log functionality.
Our engineers have been monitoring this service.
As of 7:28pm PST, the affected ElasticSearch cluster began failing.
The expected behavior during this time is that requesting the audit log will fail, and there will appear to be no events, for either flag toggling or administrative tasks. This is a temporary display problem, audit logs are still being saved and will be recovered.
We are not aware of any other degraded systems at this time.
The next communication regarding this incident will be in 4 hours.
Sep 4, 19:53 PDT