This incident has been resolved. We are in the process of conducting a root cause analysis and will be sharing more details shortly.
Posted Jul 07, 2022 - 09:04 PDT
Monitoring
All services are operational and error rates have subsided. We’re continuing to monitor our systems.
Posted Jul 06, 2022 - 09:14 PDT
Update
We are applying recovery steps, and the health of the database is improving. The API is still degraded, and code references and saved dashboards are disabled, but all other services are operational.
Posted Jul 06, 2022 - 08:06 PDT
Identified
We are experiencing issues in our primary DB cluster again, leading to increased API errors and some UI unavailability.
Posted Jul 06, 2022 - 07:26 PDT
Monitoring
We have finished the recovery steps and we are monitoring the results.
Posted Jul 05, 2022 - 20:23 PDT
Update
We are continuing in applying recovery steps and some services are recovering. We will continue to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 19:03 PDT
Update
We are continuing in applying recovery steps and some services are recovering. Audit log has recovered and we will continue to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 18:15 PDT
Update
We are continuing in applying recovery steps and some services are recovering. We will continue to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 18:02 PDT
Update
We are continuing in applying recovery steps and some services are recovering. Experimental results have recovered and are no longer delayed. We will continue to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 17:05 PDT
Update
We are continuing in applying recovery steps and some services are recovering. We will continue to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 15:57 PDT
Update
We are continuing in applying recovery steps and some services are recovering. We will continue to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 15:15 PDT
Update
We are still in the process of applying recovery steps and continuing to monitor the impact on all downstream services.
Posted Jul 05, 2022 - 14:02 PDT
Update
We are beginning to see our primary DB recover, but some of the recovery steps are still in progress. The web UI is available but we are still seeing an impact on downstream services
Posted Jul 05, 2022 - 13:05 PDT
Update
We are still working to restore the availability of our primary database cluster. We’re still seeing a significant impact across all our services. Experimentation results are delayed currently but we do not anticipate any data loss.
Posted Jul 05, 2022 - 12:05 PDT
Update
We’re still seeing a significant impact across all our services. Error rates for flag delivery (streaming and polling) are elevated, but oscillating between 1% and 10% failure rates. Most SDK clients connecting to the service should still be receiving correct flag variations.
Posted Jul 05, 2022 - 11:01 PDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 05, 2022 - 10:51 PDT
Identified
Our primary database cluster is unhealthy. This is causing widespread availability issues with our site, APIs and other downstream services.
We do not anticipate any data loss and are working to restore service as quickly as possible
Posted Jul 05, 2022 - 10:03 PDT
Update
We are still investigating the issue
Posted Jul 05, 2022 - 09:06 PDT
Update
We are continuing to investigate this issue.
Posted Jul 05, 2022 - 07:23 PDT
Investigating
We are currently investigating increased error rates across LaunchDarkly services.
Posted Jul 05, 2022 - 07:15 PDT
This incident affected: Flag Delivery Network (core functionality) (Server-side streaming API, Client-side streaming API, Polling API), Feature management (core functionality) (Feature workflows, Flag targeting), Global (non-core functionality) (Audit log), Automations (non-core functionality) (Integrations and webhooks), and Account management.