JIRA & Confluence SSO unavailable
Incident Report for The Fuel Rats
Postmortem

October 18, 2020 – Session crosslinking in JIRA

 

Meeting

Postmortem meeting was held on October 19th, at 5 PM UTC. Present were:

Clapton

Unknown

Absolver

Adanaran

Becci

 

A run-through of the incident was performed, and actions to secure the data was explained. Further investigation tasks were delegated and commenced.

 

Description

After update to our internal software systems (API v3 specifically being related), on October 2nd a user reported that they were logged into JIRA as a different user than their own, after OAuth sign-in. Given the state of our systems at the time, the fact that the user had cleared the error condition themselves by logging out and back in again, an assumption was made that this was either a visual fault or a very specific conditional state stemming from our recent updates. JIRAs session cache was flushed, forcing all users to reauthenticate.

The true extent of the problem was revealed after a second maintenance window on October 17th, during which the server hosting JIRA was restarted and the same session cross-linking occurred again. At that time, a security breach was declared and the SSO plugin used to allow users to log in to JIRA using their Fuelrats.com account was turned off.

After consulting with the vendor for our SSO plugin and Atlassian, corrective measures were undertaken and the SSO system was brought back online.

After investigation of access logs, the extent of the security breach is, as far as we can see, limited to one email address being exposed to a different user than its owner. The user affected has been notified of this breach.

 

Timeline

Oct 2 16:21 UTC – User reports being logged in as someone else to a Techrats Team member.

Oct 2 16:25 UTC – Incident is discussed in Techrats backroom. Audit information is pulled from JIRA to investigate the logins. This incident happens simultaneously with intensive activity to implement updated network ban configuration due to a banned user evading through the use of VPNs.

Oct 2 19:00 UTC – Conclusions are made that there is no issue with how the API handles sign-ins, and that the session crosslink is likely either a visual error or a fluke due to recent updates to our systems.

Oct 17 15:34 UTC – After a JIRA restart, an ops team member logged in to JIRA and was actually logged in as another ops team member. The user themselves does not notice this until much later.

Oct 17 23:58 UTC – User notices that their JIRA login is incorrect. Tech team is alerted to this.

Oct 18 00:00 UTC – Tech team investigates the incorrect login and attempts to find the source for this. This is during the night European time, and many of the tech team members are asleep at this point. JIRA login tokens are cleared as a precaution, forcing all users to reauthenticate through SSO.

Oct 18 01:00 UTC – Barebones investigation into the error concludes with no conclusive indication of what is happening.

Oct 18 08:00 UTC – Tech team members based in Europe start investigating the breach. JIRA and Confluence SSO is disabled temporarily as a safeguard against further session leakage.

Oct 18 08:58 UTC – MiniOrange is contacted regarding the issue. Tech team continues own investigation into the problem, including pulling authentication and debugging logs from JIRA and Confluence.

Oct 18 11:22 UTC – MiniOrange agent contacts Tech team and connects to a screen sharing session to investigate against our live environment. The agent concludes that the SSO plugin is functioning normally and that the issue is caused by caching somewhere upstream from the JIRA application.

Oct 18 13:09 UTC – Updates to NGINX configuration are made to ensure no caching happens on the reverse proxy for JIRA. This is the suggested fix by the SSO plugin vendor

Oct 18 17:00 UTC – Tech rats hold postmortem meeting for the incident.

Oct 19-20 (All day) - Further investigation into impact of breach, affected accounts and possible alternate paths of failure for the problem. Implementation of safeguards and monitoring for signs of incorrect session linking.

Oct 21 12:00 UTC – Finalization of security review and postmortem documentation. User notifications sent out.

What is session crosslinking/leaking

Sessions are something many websites use to hold your authentication information, letting the website know which user you are when accessing the website. JIRA uses this same system of session cookies to authenticate users. When someone signs in through our SSO (Single Sign-On) provider, they are accessing JIRA by providing their Fuelrats.com credentials. The SSO plugin we use then creates a user for them in JIRA, sets the appropriate access, and authenticates them as that user in JIRA itself. This authentication information is stored in the browser in the form of a session cookie.

Session crosslinking or leaking is when a session is sent by the webserver to a different user than the one it is intended to. This means that the user’s browser is then authenticated as that user and can fully act as if they are that user.

 

Contributing Factor(s)

Investigation into the source of the error lead to a few conclusions about the conditions present when session leaking was observed.

1)      JIRA itself had recently been restarted

2)      The users that were given access to other accounts than their own logged in shortly after JIRA started up, or during its warm-up phase (The application is running, but is not ready to serve requests yet)

The vendor for our SSO plugin was contacted regarding this security error, and given the required information about what was happening, but no information about our own conclusions up to that point. Their investigation was done against our live environment, by means of screen sharing a session where the incident leader performed commands as directed by the agent.

The SSO plugin vendor concluded that no session crosslinking was occurring in the plugin itself, and that the issue was likely the reverse proxy caching responses while JIRA was unresponsive.

Atlassian was also brought in on the problem, but deferred investigation to MiniOrange, as the authentication plugin vendor.

 

Stabilization Steps

As per vendor’s recommendations, explicit instructions were added to NGINX configurations to prevent caching in the reverse proxy. Additionally, debug logging was turned on in the SSO plugin to ensure that we have more in-depth information about the session process if the vendor suggested fix does not prove effective.

We have implemented logging and alerting for suspicious sessions in the meantime, and are notifying users that they should be vigilant when logging in to JIRA to ensure they are logged in as the correct user, and if not, to alert the tech rats so they can investigate the error condition while it is still present.

At present, we have no reason to believe the session leakage should occur again, but neither do we have an assurance that it can’t happen again. We are thus observing the situation vigilantly.

 

Impact

After triaging the situation, a careful review of logs and security audit information was performed, to determine how many incidents of session crosslinking has occurred. The team found 4 incidents of session crosslinking, and audited what information was accessed by these crosslinked sessions. Two of the sessions were established and then logged back out almost immediately, with no information accessed. One session accessed a drill request not belonging to themselves, exposing the other user’s email address to the incorrectly authenticated user. Another session was a Janitor ending up authenticated as another Janitor, and thus had no higher privileges than they themselves usually have. The exposure of PI (Private Information) is thus limited to a single user’s email address. This user has been notified of the exposure.

 

Corrective Actions

Increased logging and monitoring of JIRA (and Confluence) session authentication has been established.

The tech team has been instructed to be more vigilant to ‘strange’ occurrences and not dismiss potential problems as one-time occurrences without concrete evidence of this being the case.

Posted Oct 21, 2020 - 12:02 UTC

Resolved
Confluence and JIRA SSO are now under normal operation. This incident has a postmortem available.
Posted Oct 21, 2020 - 12:02 UTC
Identified
JIRA and Confluence SSO is once again available, while we're looking carefully at some logs. Normal usage can resume for now.
Posted Oct 18, 2020 - 12:24 UTC
Investigating
We are experiencing issues with our single sign on solutions for JIRA and confluence and have therefor deactivated SSO.
This renders all helpdesks unusable for the time being. Technical help can be provided through IRC if needed.
Posted Oct 18, 2020 - 08:40 UTC
This incident affected: Atlassian (Confluence, JIRA).