Here’s a feel good story to start the new year.
Just before the holidays, we detected a cross-site scripting (XSS) vulnerability while running a web application scan for one of our customers. Nothing special about that; we detect thousands of these things every week. But as we discussed this particular finding, we noticed that the layout of the website looked… familiar. As it turned out, the discussion forum where we found the XSS was a SaaS-based product called Lithium.
From Lithium’s website: “The world’s most innovative companies such as AT&T, Barnes & Noble, Best Buy, Sephora, Univision, Home Depot, and HP use Lithium to engage their customers in breathtaking new ways (literally, breathtaking).” Run this Google search and you’ll find a bunch of Fortune 500 companies using their software.
So now, instead of one XSS, we have hundreds. It’s not just our customer who is impacted. Suddenly it’s kind of a big deal.
Here’s how everything played out. We’ll do this timeline CORE Security style (all times EST).
- 2011-12-15 5:29 PM. We fire off a quick email to the address listed on Lithium’s Security page.
- 2011-12-15 6:12 PM. We receive a response from Misha Logvinov, Lithium’s CIO.
- 2011-12-15 6:23 PM. We encrypt and send over the vulnerability details to Lithium.
- 2011-12-15 11:40 PM. Lithium reports they have a patch ready to go and will update in the morning.
- 2011-12-16 2:30 PM. We do a little poking around and it seems the vulnerability is patched for some domains but not others. We email for a status check.
- 2011-12-16 2:37 PM. Lithium confirms that the patch is in the process of being rolled out and will be completed by close of business.
- 2011-12-16 COB-ish. We’re not seeing any more vulnerable instances.
Anybody who has reported vulnerabilities before can appreciate how unusual it is for a vendor to respond this quickly. Everything was accomplished in under 24 hours! That is practically unheard of.
From a “big picture” perspective, this whole situation illustrates some important application security themes:
- It’s a canonical example of software supply chain risk. A single XSS vulnerability simultaneously affected hundreds, maybe thousands of customers. No matter how securely these companies developed software internally, they were still exposed to vulnerabilities in third-party software.
- It emphasizes the ecosystem effect of vendor security assessments. One Lithium customer did an analysis of third-party code they were operating. A defect was found, and the vendor fixed it. Now all Lithium customers benefit, without having to lift a finger! Imagine if all companies assessed their third-party code and insisted on fixes from their suppliers.
- It shows that SaaS can have huge security benefits. Imagine if Lithium had been deployed as an on-premise product at each customer sites, requiring each customer to download and install a fix themselves. Some companies would probably never get around to patching their servers. The flip side is if the SaaS company dragged their feet — or simply refused — to patch the software, leaving the customer without a viable mitigation.
- It demonstrates that application security response can be done right. Lithium engaged quickly, took the vulnerability report seriously, and wasted no time in fixing the problem. It’s not uncommon for some vendors to take months.
Increasingly, we’re seeing our customers rethink how they vet the software they purchase or license from third-party suppliers. I hope success stories like these become commonplace as we start holding software suppliers — both SaaS and on-premise — accountable for security, not just functionality.