This incident has been resolved. At around 10pm, we changed our certificate bundle to exclude “AddTrust External CA Root” which is scheduled to expire in a few hours and whose continued presence would interfere with connectivity to our Roku clients (see https://community.roku.com/t5/Roku-Developer-Program/Potential-service-disruption-Upcoming-SSL-certificate-expiration-could-impact-channel-operations/td-p/567871). We subsequently observed an increase in TLS negotiation failures. On investigation, we discovered that the new certificate bundle contained a valid chain that worked for all clients on most platforms but some clients on some platforms were unable to complete a valid trust chain without the presence of the removed, expiring certificate. At just after 11pm, we introduced a new intermediate certificate into our certificate bundle. This change allowed the remaining clients to establish a valid trust chain and restore their connection to our service.
Posted May 29, 2020 - 23:36 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 29, 2020 - 23:22 PDT
Investigating
Some clients are experiencing connection errors due to a change in our TLS configuration. We are investigating and will update when the issue is resolved.
Posted May 29, 2020 - 23:00 PDT
This incident affected: Flag Delivery Network (core functionality) (Server-side streaming API, Client-side streaming API).