Delays in asynchronous event processing

Incident Report for Sift

Resolved

All systems are healthy.
Posted Apr 03, 2019 - 07:45 PDT

Update

The backlog of events have been processed and there is no more delay on asynchronous processing.
Posted Apr 03, 2019 - 05:15 PDT

Update

The backlog of events have been processed and there is no more delay on asynchronous Scores and Workflows. There are delays to data in the Explore tab and we are currently working through the backlog of delayed messages.
Posted Apr 03, 2019 - 04:39 PDT

Monitoring

The backlog of events have been processed and there is no more delay on asynchronous processing.
Posted Apr 03, 2019 - 02:39 PDT

Update

We are continuing to work through a backlog of messages and expect to resolve delays in 8 hours. We'll update this estimate if this changes.
Posted Apr 02, 2019 - 17:24 PDT

Update

The issue with delays to asynchronous processing has been identified and a fix has been deployed. We are continuing to work through a backlog of delayed messages.
Posted Apr 02, 2019 - 16:53 PDT

Update

The issue with delays to asynchronous processing has been identified and a fix has been deployed. We are continuing to process delayed messages.
Posted Apr 02, 2019 - 16:21 PDT

Update

The issue with delays to asynchronous processing has been identified and a fix has been deployed. We will continue to process delayed messages and provide updates.
Posted Apr 02, 2019 - 15:42 PDT

Identified

The issue with delays to asynchronous processing has been identified and a fix is being deployed.
Posted Apr 02, 2019 - 14:57 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 14:22 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 13:46 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 13:17 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 12:46 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 12:15 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 11:43 PDT

Update

We are continuing to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 11:12 PDT

Update

We continue to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 10:40 PDT

Update

We continue to actively monitor the delay in asynchronous processing.
Posted Apr 02, 2019 - 10:06 PDT

Update

We continue to actively monitoring the delay in asynchronous processing.
Posted Apr 02, 2019 - 09:33 PDT

Update

We are adding additional capacity to address the processing delay.
Posted Apr 02, 2019 - 04:14 PDT

Monitoring

We are currently experiencing a processing delay for events received through the Event API. Updated scores, data in Explore, and workflows may be delayed. We have implemented a fix and are monitoring. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.
Posted Apr 02, 2019 - 00:58 PDT
This incident affected: API.