This incident has been resolved. Numeric experimentation results are no longer delayed.
As mentioned earlier, all customers who may have been impacted by the loss of any data have been notified via email.
Posted Jan 02, 2022 - 22:12 PST
Update
Numeric experimentation event processing is continuing to consistently progress through the data backlog. At the current rate, we expect experimentation event processing to be caught up to real-time by 7:00 AM UTC on Monday, January 3.
All customers who may have been impacted by the loss of any data have been notified via email.
Posted Dec 30, 2021 - 04:33 PST
Update
Numeric experimentation event processing is consistently progressing through the data backlog. At the current rate, we expect experimentation event processing to be caught up to real-time by 7:00 AM UTC on Monday, January 3.
Additionally, we learned that some numeric experimentation results are lost and unrecoverable. Results received between December 23 23:00 UTC and December 25 14:00 UTC may be lost and unrecoverable. Numeric experimentation data received after December 25 14:00 UTC has processing delays but is not lost. All other forms of data sent to LaunchDarkly are unaffected. We will reach out to impacted customers individually regarding their impacted experiments.
Posted Dec 28, 2021 - 14:33 PST
Identified
Our event processing infrastructure is running behind which is causing stale data to be shown in numeric experimentation results. Numeric experimentation results received on or after December 23 may be delayed until this incident is resolved. No data has been lost and the system is currently catching up.