User search is failing intermittently

Incident Report for LaunchDarkly

Resolved

The user search cluster has fully recovered to a stable healthy state. We re-enabled persisting user updates more than 8 hours ago, and the search requests remain performant. If you suspect any issues with your account's user data in LaunchDarkly, please contact LaunchDarkly Support.
Posted Aug 25, 2017 - 10:54 PDT

Monitoring

We have restored user search functionality, which initially started failing due to hardware failures. The search cluster is still recovering and rebalancing, so search query performance will suffer during this time.

We have also disabled user saving/updates to allow the search cluster to recover in a swift and stable manner. User updates received by our service during this period will not be available.
Posted Aug 24, 2017 - 15:22 PDT

Update

User search is now consistently failing across the search cluster. We are working to restore user search functionality.
Posted Aug 24, 2017 - 12:59 PDT

Identified

User search is occasionally timing out due to a system failure in our search cluster. We are building a new system to replace the failed server.
Posted Aug 24, 2017 - 09:34 PDT