Client connection errors for some clients due to a recent TLS configuration change
Incident Report for LaunchDarkly
Resolved
This incident has been resolved. At around 10pm, we changed our certificate bundle to exclude “AddTrust External CA Root” which is scheduled to expire in a few hours and whose continued presence would interfere with connectivity to our Roku clients (see https://community.roku.com/t5/Roku-Developer-Program/Potential-service-disruption-Upcoming-SSL-certificate-expiration-could-impact-channel-operations/td-p/567871). We subsequently observed an increase in TLS negotiation failures. On investigation, we discovered that the new certificate bundle contained a valid chain that worked for all clients on most platforms but some clients on some platforms were unable to complete a valid trust chain without the presence of the removed, expiring certificate. At just after 11pm, we introduced a new intermediate certificate into our certificate bundle. This change allowed the remaining clients to establish a valid trust chain and restore their connection to our service.
Posted May 29, 2020 - 23:36 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 29, 2020 - 23:22 PDT
Investigating
Some clients are experiencing connection errors due to a change in our TLS configuration. We are investigating and will update when the issue is resolved.
Posted May 29, 2020 - 23:00 PDT
This incident affected: Streaming API and Mobile Streaming API.