Data Backfill has been completed. System is stable.
Posted Dec 23, 2021 - 21:19 PST
Update
Backfill event ingestion is ongoing. No ETA at this time on completion of backfill. System is stable and processing incoming events.
Posted Dec 20, 2021 - 10:02 PST
Update
Backfill event ingestion has been started. Data inconsistencies for Flag Insights evaluations after 2021-12-08 2:00 UTC have been resolved.
Posted Dec 16, 2021 - 09:49 PST
Update
We are continuing to work on resolving data inconsistencies for Flag Insights evaluations after 2021-12-08 2:00 UTC.
Posted Dec 15, 2021 - 16:25 PST
Update
We have identified a data inconsistency for Flag Insights evaluations in the timeframe of 2021-12-07 15:00 to 2021-12-09 19:00 UTC. We are working to correct the data.
Posted Dec 14, 2021 - 14:08 PST
Update
Flag Insights (evaluations) data from the incident period on Tuesday has been fully processed. There are still some gaps in data on the Usage page for client and server connections, experiment events received, and Data Export events published; we will resume filling those data gaps on Monday.
Posted Dec 11, 2021 - 10:09 PST
Update
We are still reprocessing data for Flag Insights (flag evaluations) from the incident period.
Posted Dec 09, 2021 - 21:17 PST
Update
We have fully processed experimentation data ingested during the incident. We are currently working on processing data for Flag Insights (flag evaluations).
Posted Dec 08, 2021 - 22:01 PST
Monitoring
Our event processing components have resumed to normal as our cloud provider has stabilized. We are continuing to ingest and process new event data, but there may be gaps and inconsistencies in data ingested between 15:34 and 23:45 UTC. We will keep this incident in Monitoring until we have confirmed that event data from earlier today has been processed and recorded properly.
Posted Dec 07, 2021 - 19:12 PST
Update
Services are continuing to progress toward full recovery, but we are not fully there yet. We have also identified that Scheduled Flag Changes have been only intermittently operational since 15:34 UTC. The Integrations dashboard has returned to operational.
Posted Dec 07, 2021 - 16:09 PST
Update
We are seeing some forward progress in event processing -- Flag Insights, Account Metrics, and Experimentation data have resumed updating but are still behind realtime. Some email and Slack notifications related to flag changes are being delivered with a significant delay.
Newly created numeric experiments since December 1 have not been affected by this incident.
Posted Dec 07, 2021 - 10:48 PST
Update
We have identified additional components affected by the service incident: The Integrations management page is failing to load. Some email and Slack notifications are failing to be delivered.
Known services impacted include Flag Insights, Account Metrics, and Experimentation processing, Integrations management, and flag notifications.
Posted Dec 07, 2021 - 08:52 PST
Identified
Our cloud provider is experiencing a major incident since 15:34 UTC which is causing various event processing systems to degrade. We do not believe data has been lost. Core services around flag management and flag delivery are still functioning correctly.
Known services impacted include Flag Insights, Account Metrics, and Experimentation processing.
Posted Dec 07, 2021 - 08:16 PST
This incident affected: Automations (non-core functionality) (Emails and notifications, Integrations and webhooks) and Feature management (non-core functionality) (Data Export, Flag usage metrics).