TL/DR: An SCA IT web configuration error exposed confidential email messages.
- For three years, the SCA mistakenly published all email sent to Board of Directors’ feedback address, allowing anyone on the Internet to read messages that had been sent in confidence, including reports of harassment and sexual assault.
- If you emailed sca-comments@sca.org between March 6, 2020 and February 2, 2023, you should be aware that the message you sent is no longer secret and has likely been read by other people outside of the organization’s leadership.
- Six mailing lists used by committees for internal communication were also affected.
Discovery
Sometimes, when you’re looking for something in particular, you stumble onto something completely unexpected.
This is how I felt in late January when I was searching for information about a lawsuit involving the SCA (a topic for another day), and one of the items in my Google search results turned out to be an email from one of the plaintiffs addressed to the Board. At the time I just focused on the letter’s content, integrating it into my ten-thousand-word record of the legal dispute — but later it occurred to me to wonder: why was that correspondence on the web in the first place?
When I went back to look again a week later, I realized that it wasn’t just this email: thousands of other email messages that had been sent to the board using the sca-comments@sca.org address were available at the same website.
Was this a laudable gesture towards transparency? As I browsed through the accumulated messages, that possibility seemed less and less likely. There were heartfelt messages addressed to the Board on a wide range of topics — opining on the best way to implement new peerages… in favor of tightening Covid restrictions, or loosing them… supporting DEI efforts, or denouncing them as wokeism… complaining about bad behavior by various people involved with the Society… and some of these messages clearly hadn’t been written for public dissemination.
And then as I clicked around the archive at random, I encountered a detailed eyewitness account from an SCA event at which someone had been sexually assaulted — and the person who’d been attacked was someone I knew. Yikes.
Intervention
At that point I stopped reading and started composing an email to the Society’s IT Manager reporting the problem in detail, and identifying the path forwards:
- Take the archive offline,
- Contact major search engines to clear their caches, and
- Notify the people whose emails were exposed about the breach.
I also reached out to the Webministry, using the Slack servers operated by the East Kingdom and their inter-kingdom counterparts, describing the problem and emphasizing the importance of fixing it. I was surprised to learn that another East Kingdom webminister had encountered the same problem and reported it up the chain of command to the Society level more than a week before I did, although they had been more circumspect about it.
I figured that the time for circumspection had passed, and made a fuss about the need to act promptly, suggesting that failure to do so would look bad should anyone ever open a legal claim regarding the breach, and by late afternoon the archive link I had reported was no longer accessible.
… and then four days later, another East Kingdom webminister pointed out that the same archive was still available at a slightly different URL (http links had been fixed but https remained open), and within an hour that was corrected.
… and then I noticed that in addition to the sca-comments address, there were another six internal mailing lists which also weren’t intended to be public but had been exposed in the same way, and within a few hours those had been fixed as well.
Notification
After confirming that the caches at major search engines had been cleared, and providing some guidance about how to do the same thing for the Archive.org Wayback Machine, I returned to the subject of disclosure: was there a plan to notify the people whose emails had been inadvertently published of the breach?
I raised the issue several times over the next few weeks, making it clear that I was willing to hold off writing about this for a while to allow the Society to disclose it on their own terms, but didn’t want to do so forever. I was told to be patient, as the issue was under consideration by the Board, and finally two months later a public announcement was made on April 11.
… and then it turned out the announcement had significantly understated the duration of the breach as lasting just nine months, and had to be corrected to reflect that more than thirty-four months of messages had been exposed.
Affected Addresses
In addition to the Board’s comments address, the archives of six much lower-traffic internal mailing lists were also improperly exposed in the same way.
- Board Comments (sca-comments@sca.org / comments@lists.sca.org): March 6, 2020 – February 2, 2023, 2,782 messages.
- Finance Committee (finance@lists.sca.org): July 20, 2017 – July 30, 2018, 292 messages.
- Membership Committee (membercomm@lists.sca.org): July 17, 2017 – July 22, 2018, 135 messages.
- Communications Committee (communicationscomm@lists.sca.org): August 22, 2017 – December 31, 2017, 63 messages.
- Exchequer Handbook (exhandbook@sca.org): August 2, 2016 – January 4, 2017, 40 messages.
- Chirurgeonate (chirurgeonate@sca.org): January 12, 2004 – April 14, 2012, 23 messages.
- Chirurgeonate Handbook (ch-handbook-committee@sca.org): August 24, 2011 – August 26, 2011, 2 messages.
Technical Details
The addresses in question were all mailing lists handled by GNU Mailman.
Message archiving was enabled, with web archives generated by Pipermail.
The archives were set to only be accessible by authenticated list members. This setting causes Mailman the /listinfo page to link to an authenticating /private/ URL path to view the archives, and is supposed to also disable the unauthenticated /pipermail/ URL path.
However, an unknown web configuration error prevented the second half of this protection from taking effect, and the archives remained available at the unauthenticated URLs.
Because the /listinfo view was using the /private/ links, visitors to the lists.sca.org site could not access the archives by clicking links, but anyone who was told about the /pipermail/ links, or guessed that they might exist, or encountered them on another website, could access the archives unimpeded.
Those links were available via both http and https, with very similar configurations, both of which required remediation.
Timeline
2001 November: Apparent date of setup of lists.sca.org server, and initial use for sending out official announcements.
Early 2000’s: The comments@lists.sca.org address is set up as a feedback mechanism for commentary to the Board.
2004 January: First archived message to the Chirurgeonate mailing list.
2018 July: Internal committee mailing lists appear to be migrated to another server, and no additional traffic is added to their web archives after this date.
2020 March 6: Configuration changes made by the SCA’s Manager of IT, Aaron Palomides, cause web archiving to begin for the comments@ address.
2021 April 30: Date of first web capture of the sca-comments archive by the Archive.org Wayback Machine crawler.
2022 April 24: Thomas Blackmoore takes over as Manager of IT. [Announcement, February 18]
2022 June 18: Primary address used for Board feedback changed from comments@sca.org to sca-comments@sca.org. This is done with the intention of later disabling the old comments@ address, which is the target of a significant amount of spam. [Announcement, June 20]
2022 July 24: Iain MacArthur takes over as Society Webminister. [Announcement, July 27]
2023 January 19: Most recent web capture of the sca-comments archive by the Archive.org Wayback Machine crawler. Snapshot includes copies of 620 individual URLs.
2023 January 19: An East Kingdom webminister learns of the exposed sca-comments archive and reports it to the Kingdom Webminister and the Society Webminister. The Society Webminister reportedly forwards this notification to the Society IT Manager. No action is taken.
2023 January 20: I discover an email message from the sca-comments archive in a Google search result. Puzzled by the availability of these archives, I visit the information page for the list and fill out the subscription form.
2023 January 23: My sca-comments subscription request is rejected by a list moderator with the message “Sorry, subscriptions to this mailing list are not open to the public.”
2023 January 20–28: Second-hand reports after the fact suggest that there was at least one and perhaps two additional notifications of the breach sent to Society technical leadership during this period.
2023 January 28, 11:00 PM: I realize that the message I discovered is a symptom of a larger problem. I spend some time browsing through archived messages and discover highly sensitive revelations.
2023 January 29, 4:45 AM: I send an email about the problem to the Society IT Manager with copies to the Society Webminister and to the sca-comments@ address.
2023 January 29 4:45 PM: The SCA IT Manager makes a configuration change that causes the archive web address I reported to go offline, returning a not-found error. Pages remain visible in Google search results and cache.
2023 January 31: Search results no longer available via Google. No results found in Bing or other search engines. Archive.org still has 621 matches.
2023 February 1: I send email to Society IT Manager outlining the archive.org remediation steps. Response within minutes saying that it had been requested.
2023 February 2, 3:00 PM: A third East Kingdom webminister reports that while http URLs for the sca-comments list have been blocked, the archive is still available at the same path via https. Half an hour later, Society Webminister reports that this has been fixed, by blocking all https access to this server.
2023 February 2, 6:00 PM: I discover that six more mailing lists have the same problem, and report this. Within a few hours those have been fixed as well.
2023 February 2, 10:00 PM: I send another message to the Society’s IT Manager encouraging the SCA to disclose the breach, and am told that this is a decision to be made by the Board.
2023 February 3: The snapshots on Archive.org are removed. At this point, all known public caches of the mistakenly-exposed messages have been expunged.
2023 March 10: I send another message to the Society’s IT Manager encouraging the SCA to disclose the breach, and am told that this has been passed to the Board.
2023 March 21: I am told that the Board is considering disclosing the breach.
2023 April 11: Breach is announced to the public. Following feedback, a correction is made to the dates of the breach. [Announcement]
Thank you for following up to make sure this is addressed properly. Your work has been invaluable and is appreciated.
Thank you for your detailed discussion of the issue and its correction.
While I understand the SCA is run by volunteers, a breach this serious required a more urgent response than is apparent here. This is especially true, given how litigation-averse the Society is. Why would it take multiple people to point out multiple times the extent of data spillage?