We believe that the audit log restoration is successful and are declaring this incident resolved.
There will be no further scheduled communications regarding this incident.
Posted Sep 06, 2017 - 23:51 PDT
Update
Audit log functionality operationally normal. Will continue monitoring until event recovery is complete.
Next update in 12 hours barring change in situation
Posted Sep 06, 2017 - 12:06 PDT
Update
Audit log display is working properly. Cluster is being reindexed to ensure all events appear correctly. We will continue to monitor and the incident will be closed as soon as reindexing is completed.
Posted Sep 06, 2017 - 00:13 PDT
Update
Audit logs display is working properly, cluster is being reindexed to ensure all events appear correctly.
Next update will be in 12 hours barring change in status.
Posted Sep 05, 2017 - 12:03 PDT
Monitoring
Engineers have implemented a fix to the audit log systems and are monitoring systems for health.
During this time, audit logs are collected, but access may be degraded as the ElasticSearch service recovers.
The next communication will be in 12 hours.
Posted Sep 04, 2017 - 23:27 PDT
Investigating
As of 12:20pm PST, our engineers have discovered a degradation in service with one of the ElasticSearch clusters that powers the audit log functionality.
Our engineers have been monitoring this service.
As of 7:28pm PST, the affected ElasticSearch cluster began failing.
The expected behavior during this time is that requesting the audit log will fail, and there will appear to be no events, for either flag toggling or administrative tasks. This is a temporary display problem, audit logs are still being saved and will be recovered.
We are not aware of any other degraded systems at this time.
The next communication regarding this incident will be in 4 hours.