Postmortem
Impact
For an EU customer with Proctorio some candidates got an error during their exam. For most of these Cirrus automatically recovered but those candidates that closed their browser lost their unsaved answers.
Root Cause
An intermittent error in Cirrus front-end (Cirrus in browser), could cause the front-end to stop "processing"; sending answers to the backend (Cirrus server).
Resolution
Full rollback to the previous release where this error did not occur to safeguard other exams.
Improve robustness of front-end processing so that if any error again will occur there, or in similar places, Cirrus will continue working fine (and additionally send a message to the invigilator log).
Besides extensive manual and automated testing, also testing by artificially raising these errors to thoroughly validate improved error handling. This will be released on September 12th, 20230912F (superseding the September 5th release).
Preventative Measures
- [DONE] Extend automated testing using one real exam (at least temporarily)
- [DONE] Artificially inject errors to test all parts of our queue processing.
- [DONE 20230912F] Improved robustness of all front-end processing loops, including logging to invigilator log - CR-21552
- [DONE 20230912F] Improve wording of Q43 message - CR-21567
- [DONE 20230912F] Eliminate (fix) distracting "Uncaught Type Error" false alerts in the invigilation logs - CR-21560
- [DONE 20230912F] Eliminate (fix) distracting "Last answer Id is not valid" false alerts in the invigilation logs - CR-21565
- ([DONE 20230912F] Improved wording of offline invigilation log message (change “Candidate offline” into something like “Candidate offline or navigated away” to new tab) - CR-21562)
- [Pending] Monitor invigilator logging for queue processing messages for further investigation
- [Retro] Team Briefing on making fundamental code, like queue processing loops, robust.
- [Future] Improve temporary storage of answers in browser - CR-21566
Posted Aug 29, 2023 - 15:31 CEST
Completed
The scheduled maintenance has been completed.
Posted Aug 23, 2023 - 07:00 CEST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Aug 23, 2023 - 06:00 CEST
Scheduled
Tomorrow morning at 06:00 instead of a hotfix today's Feature Update will be reverted (rolled back)!
After this roll back the Cirrus platform will be working the same as it was before the 22 August update.
Though only one customer encountered a critical issue with only some of their candidates, in order to safeguard every ones exams we have taken this unusual step at this late hour to allow us more time for both resolution and quality assurance.
As well delivered exams are in our shared interest we trust you support us in this.
More details will published here in a "Post Mortem" next week.
Time of release per hosted region:
EU: 06:00 CEST
EU Premium: 06:00 CEST
CA: 06:00 CEST
SG: 21:00 SGT
AU: 00:00 AEST
Posted Aug 22, 2023 - 23:34 CEST
This scheduled maintenance affected: EU Assessment Management (Dublin) (Scheduling (EU), Marking (EU), Authoring - Library/Assessments (EU), Administration (EU), Auxiliary - Services (EU)), EU Candidate Delivery (Dublin) (Candidate Delivery incl Proctoring API (EU), Single Sign-On SAML 2.0 / LTI (EU), Invigilation (EU)), AU Candidate Delivery (Sydney) (Candidate Delivery incl Proctoring API (AU), Single Sign-On SAML 2.0 / LTI (AU), Invigilation (AU)), AU Assessment Management (Sydney) (Scheduling (AU), Marking (AU), Authoring - Library/Assessments (AU), Administration (AU), Auxiliary - Services (AU)), SG Candidate Delivery (Singapore) (Candidate Delivery incl Proctoring API (SG), Single Sign-On SAML 2.0 / LTI (SG), Invigilation (SG)), SG Assessment Management (Singapore) (Scheduling (SG), Marking (SG), Authoring - Library/Assessments (SG), Administration (SG), Auxiliary - Services (SG)), CA Candidate Delivery (Montreal) (Candidate Delivery incl Proctoring API (CA), Single Sign-On SAML 2.0 / LTI (CA), Invigilation (CA)), CA Assessment Management (Montreal) (Scheduling (CA), Marking (CA), Authoring - Library/Assessments (CA), Administration (CA), Auxiliary - Services (CA)), EU Premium Assessment Management (Dublin) (Scheduling (EU Premium), Marking (EU Premium), Authoring - Library/Assessments (EU Premium), Administration (EU Premium), Auxiliary - Services (EU Premium)), and EU Premium Candidate Delivery (Dublin) (Candidate Delivery incl Proctoring API (EU Premium), Single Sign-On SAML 2.0 / LTI (EU Premium), Invigilation (EU Premium)).