[EU Only] Processing Delays in some attempt generation

Incident Report for Cirrus Assessment

Postmortem

Impact

Some Prod-EU candidates with exams scheduled in advance could not start their exam unless their schedule was updated (Workaround).

Root Cause

After maintenance to apply security patches to the jobs database, schedule jobs were not being executed while all other jobs were executing fine (without any alerts as monitoring all green).

Resolution

Impacted customers informed of work-around through schedule update.

Roll-over (restart without down-time) of processing infrastructure, including temporary scale up

Preventative Measures

DONE: Wide-audience Root Cause Analysis to review incident and incident response
DONE: ~~In Progress~~ Further improve processing health check [CR-20740] / [CR-20864]
DONE: ~~In Progress~~ Further improve processing monitoring [SYSOPS-712]
DONE: ~~Planned~~ Table top exercise Incident Management [CISM-244]
DONE: ~~In Progress~~ Further improve Ops manual
DONE: ~~In Progress~~ Refresher stand-by Ops staff

Posted Jun 09, 2023 - 08:20 CEST

Resolved

This morning the Cirrus platform encountered processing delays affecting attempt generation, leading to some Prod-EU (not Premium) candidates not being able to start their exam on time.

To ensure all processing is working in a timely manner the processing of prod-eu was temporarily scaled up. For the other regions we rolled over without downtime just in case.

Our initial investigation shows this to be related to the maintenance of Sunday night although most processing, including part of the attempt generation, still worked as it should. The team will investigate further incl. potential preventative measures.

Apologies for the inconvenience and the fact that due to an issue in our internal processes our status page was not updated in a timely manner. We will schedule an internal incident management training session.

Posted Jun 05, 2023 - 10:00 CEST

This incident affected: EU Candidate Delivery (Dublin) (Candidate Delivery incl Proctoring API (EU)).