npm operational incident, 6 Jan 2018
By npm Team
January 08, 2018
The npm registry had an operations incident Saturday that caused 97 packages to be temporarily unavailable for download for approximately 30 minutes, and an additional 9 packages to be unavailable for approximately three hours. Early this coming week, we will share a full analysis and technical explanation of the incident. We wanted to communicate with you sooner, however, to eliminate any doubts: no malicious actors were involved in yesterday’s incident, and the security of npm users’ accounts and the integrity of these 106 packages were never jeopardized.
The incident was caused by npm’s systems for detecting spam and malicious code on the npm registry.
We don’t discuss all of our security processes and technologies in specific detail for what should be obvious reasons, but here is a high-level overview. Automated systems perform static analysis in several ways to flag suspicious code and authors. npm personnel then review the flagged items to make a judgment call whether to block packages from distribution.
In yesterday’s case, we got it wrong, which prevented a publisher’s legitimate code from being distributed to developers whose projects depend on it.
We identified the error within five minutes and followed defined processes to reverse this block. Unfortunately, the process was complicated by well-meaning members of the npm community who believed that a malicious actor or security breach was to blame and independently attempted to publish their own replacements for these packages. Ensuring the integrity of the affected packages required additional steps and time.
We are fully evaluating the processes and technologies involved, and we’ve already made immediate changes to prevent what happened yesterday from happening again. My most important job is ensuring the reliable delivery of safe code to the millions of developers who depend on npm. I promise you a full accounting of how we fell short this weekend, and my continued focus on improving our systems and processes — you deserve no less.