Delays in asynchronous event processing

Incident Report for Sift

Resolved

This incident has been resolved.
Posted Apr 04, 2019 - 21:29 PDT

Update

The backlog of events has been processed. All systems are healthy.
Posted Apr 04, 2019 - 21:27 PDT

Update

The backlog of events have been processed and there is no more delay on asynchronous Scores and Workflows. There are delays to data in the Explore tab and we are currently working through the backlog of delayed messages.
Posted Apr 04, 2019 - 18:38 PDT

Update

We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 2 hours after the event is received.
Posted Apr 04, 2019 - 18:11 PDT

Update

We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.
Posted Apr 04, 2019 - 17:40 PDT

Update

We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.
Posted Apr 04, 2019 - 17:09 PDT

Monitoring

We are continuing to work through a small backlog of delayed events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.
Posted Apr 04, 2019 - 16:37 PDT

Update

We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 7 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.
Posted Apr 04, 2019 - 16:06 PDT

Update

We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.
Posted Apr 04, 2019 - 15:05 PDT

Update

We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.
Posted Apr 04, 2019 - 14:34 PDT

Update

We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.
Posted Apr 04, 2019 - 13:59 PDT

Investigating

We are currently experiencing a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.
Posted Apr 04, 2019 - 13:25 PDT
This incident affected: Console, API, Score, and Workflows.