Confluence & JIRA SSO unavailable
Incident Report for The Fuel Rats
Postmortem

Description

On November 21st a Tech Rat team member reported to the rest of the team that they had logged into Jira as another user. This behavior is consistent with the previous incidence of cross-session authentication, where a tentative fix was deployed, along with extensive logging to ensure further incidents were traceable.

The logging information supplied sufficient information to identify the root cause of the problem, a singleton instance of a web request API on the frontend. The problem has now been patched.

Timeline

Nov 22 14:36 UTC – Tech Rat team member logged into another user, and reported it to his fellow teammates.

Nov 22 18:44 UTC – The issue was verified and steps to trace down the root cause began.

Nov 22 23:35 UTC – A fix to the website was pushed.

Nov 22 23:55 UTC – The fix was verified after testers replicated the conditions that would have triggered the bug.

Contributing Factor(s)

The issue was caused by a programming error on the front end website’s code that did not account for potential race conditions when handling web requests triggered from the server side. The OAuth2 flow for authenticating users against third party applications includes the front end sending a HTTP GET through a singleton Axios instance on the server, containing the user’s auth token as read from the user’s cookies. This is the point of vulnerability, as a second request arriving while the first is being processed overwrites the token in the Axios instance.

The deployed fix ensures that all requests being sent to Axios now sets the user’s token.

Impact

The incident was expected, as the fix to the previous incident was not verified. Thus, the occurrence was handled expediently in the tech team, and the additional logging provided the needed clues to what was happening. Once the root cause was identified, the fix itself was easily applied. No personal information was leaked during the incident.

Corrective Actions

Frontend website code was patched to ensure that all requests to the API are sent using the correct user context. Further information about the previous incident is available here.

Posted Dec 09, 2020 - 21:26 UTC

Resolved
This incident has been resolved.
Posted Nov 22, 2020 - 00:00 UTC
Monitoring
We have deployed a fix for the issue and are monitoring the situation.
Posted Nov 21, 2020 - 23:58 UTC
Investigating
JIRA and Confluence SSO has been turned off to investigate issues with logging in. Support can be given through IRC when needed.
Posted Nov 21, 2020 - 19:15 UTC
This incident affected: Atlassian (Confluence, JIRA).