Our fix is proceeding as planned. We continue to see elevated latencies to a small percentage scoring ATO requests, as well as asynchronous requests for non-ATO scores.
Posted May 07, 2019 - 17:44 PDT
Update
We're proceeding with our fix for the issue.
Posted May 07, 2019 - 16:58 PDT
Update
Fewer than 1% of ATO requests continue to see elevated latency. For non ATO products, synchronous requests latencies have returned to normal but we are still seeing elevated latency in a small percentage of asynchronous requests. So, a small percentage of non-ATO, asynchronous workflow runs are evaluating with a delay. We are deploying a fix to address these issues.
Posted May 07, 2019 - 16:18 PDT
Update
We’ve isolated the issue and have made further progress in reducing elevated latencies. We’re working to completely fix the issue.
Posted May 07, 2019 - 15:30 PDT
Identified
We’ve made progress in reducing the percentage of calls with elevated latencies. We believe we have identified the issue and are working to get things fully back to normal. We noticed elevated latencies around 12:47pm Pacific Time, so you may have seen an increase in timeouts between then and now. Again, the vast majority of requests are being served at low latency and no data has been lost.
Posted May 07, 2019 - 14:53 PDT
Update
Ingestion of all events is operational and no data is being lost. The vast majority of all scoring continues to be done at low latencies. We are still seeing elevated scoring latencies in our synchronous and asynchronous scoring services for a small percentage of requests and are attempting to address.
Posted May 07, 2019 - 14:14 PDT
Investigating
We are currently experiencing elevated synchronous score API latencies and investigating the underlying problem.
Posted May 07, 2019 - 13:36 PDT
This incident affected: API, Score, and Workflows.