KQED, one of the largest public media companies in America, was hit by a ransomware attack in July. The disruption was barely visible to the public, but behind the scenes, employees struggled with Rube Goldbergian workarounds while tech staff worked long months to rebuild a more secure system.

Sponsored by

Become a KQED sponsor

Originally published Oct. 31, 2017

Lesley McClurg is trudging through the KQED newsroom, oozing distress.

McClurg, who in the course of her career as a radio reporter has conducted interviews amid raging fires and torrential floods, has finally met her match: an office printer. She has just spent an entire hour trying to get her radio script to print, a task that took the cumulative analytical skills of three nearby co-workers plus a guy from IT.

“And now I need to go on the radio and sound like I’m calm and relaxed and NPR-friendly,” McClurg tells me. “But, really, I need to scream.”

And so went Day 33 of the Great KQED Ransomware Attack.

This summer, criminals somewhere in Russia, Ukraine or maybe down the block, hacked into KQED’s computer system, installed malicious code that encrypted the station’s files, software and servers, and demanded money for their safe return. The perpetrators, as is routine in ransomware attacks, demanded payment in the form of the cryptocurrency bitcoin—if the victims ever wanted to see their data alive again. The message, rife with grammatical and spelling errors (cold comfort to no one but the copy editors), included directions on how to recover the data in “3 Easy Step” (sic), with step one being the transfer of funds.

The chaos commenced on June 15, as KQED’s employees began suddenly reporting computer crashes en masse. The IT department at the San Francisco-based NPR and PBS station decided to play it safe—they shut down the entire network to stop the malware’s spread.

The results were difficult to face: Phones went dead; the internet vanished; reporters’ audio interviews disappeared. And it meant the only way the head of IT, John Reilly, could get a message out to each of the 503 full- and part-time employees to keep their hands off their computers was to post handwritten signs at every entrance and on every desk.

KQED, one of the largest public media companies in America, has been hobbling its way to recovery ever since. What was it like? Think of a really boring episode of “The Twilight Zone”—or better yet, “Black Mirror.” Or getting stuck in an absurdist satire about human dependence on technology. Everyone had their particular breaking point, triggered by one Rube Goldbergian workaround or another.

One afternoon, a KQED marketing employee attempted to send out a meeting agenda in a single email to some 60 people. Our temporary email system wouldn’t allow grouping of multiple addresses into one alias, so each destination had to be entered one at a time. Any tiny problem with the syntax—an extra period here, an omitted semicolon there—grounded the message until you could figure it out. Four hours, that job took.

Managing editor Ethan Lindsey (R) leads the morning story meeting on Sept. 25, 2017. Pictured (L-R) are: politics and government reporter Marisa Lagos, photo/video intern Serginho Roosblad, cartoonist Mark Fiore, producer Sam Harnett, editor Erika Kelly. (Alison Yin Photography)

It’s a Problem

Ransomware—it’s a growth industry. Last week, a new strain called “Bad Rabbit” hit computer systems in Russia, Eastern Europe and beyond. This year, a posse of federal agencies, including the Justice Department and the CIA, issued a report describing ransomware as the “fastest growing malware threat, targeting users of all types—from the home user to the corporate network.”

And in another study, anti-virus software maker Symantec estimated 463,000 ransomware attacks worldwide in 2016, a 36 percent increase from the previous year.

Entrepreneurial hackers also have innovated “ransomware as a service,” in which they peddle kits to criminals who are less technically adept.

“It’s not some teenager sitting in his garage,” USA Today tech reporter Elizabeth Weise told KQED. “These are effectively organized crime; they have offices, they have office workers who come in.”

Over the last two years, researchers from New York University and Google tracked about $25 million in victim payments. Not bad, but not the global drug trade either. Estimates of monetary gain for perpetrators of the WannaCry ransomware scourge, a worldwide attack last May on computers running the Microsoft Windows operating system, were remarkably low, even though the virus disabled more than 200,000 computers, including those of FedEx and England’s National Health Service.

That does not, however, take into account the cost in lost sales and business interruptions, which can be considerably higher. The U.K. company Reckitt Benckiser, maker of Clearasil, Lysol and Airborne, estimated a $117 million loss resulting from the Petya ransomware attack in June.

In KQED’s case, the attackers demanded 1.7 bitcoins per computer (roughly $2,500 at the exchange rate of the time) or—act now!—just $27,000 for the lot.

KQED briefly considered paying up, said Chief Technology Officer Dan Mansergh, but abandoned the idea after the FBI laid out the risks. They were: being seen as an easy mark; inclusion of code in the decryption key that would set the stage for future intrusions; and failure of the perps to keep their part of the bargain to set your data free.

KQED has yet to fully calculate the attack’s cost, but much of it will be in IT. In fiscal year 2018, the company is allocating $475,000 for new network protections. Those funds will pay for four new IT employees, plus more sophisticated anti-virus software.

As for the cost in employee productivity and stress? A lot.

‘Worst Stress of My Life’

The attack took place on a Thursday. The next day, news reporters and editors at KQED, the most-listened-to radio station in the Bay Area, could access the internet only through personal hot spots on their cellphones. Reporters read stories into their recording decks to get on the air and didn’t use sound bites; no one crumpled cellophane to emulate the sound of a fire, but it was still old school.

The station’s weekly current affairs television program, KQED Newsroom, had to abandon its original production plans and run a previously aired special show. And when the television studios were still out the following week, the staff took production out of the building, to a makeshift studio at UC Hastings College of the Law and another at the Electronic Frontier Foundation for a segment on ransomware.

Meanwhile, IT collected and wiped clean every computer that the malware, called Samas, had so much as breathed on. But that was nowhere near the end of it. To stop the infection from spreading, IT had taken the network down. No phones, one rickety internet router, no access to documents or audio, and the only way to print was to walk your laptop to a printer and plug in the cable. And after the malware was eradicated, the network did not come back up.

“We could not just put it back the way it was, because there were all kinds of security holes,” said Michael Kadel, KQED’s network systems engineer.

So the staff went to work creating temporary systems to keep things functioning until a more secure infrastructure could be built. They worked long hours, long being a euphemism for 12-, 16- and 20-hour shifts at a time.

“Worst stress of my life,” said IT director Reilly.

Kadel agreed. “A solid two months of just terrible, terrible stress.”

Mina Kim, afternoon drive-time news anchor and Forum host. ‘Initially, we all had this attitude, the show must go on. After that surge of adrenaline and can-do attitude subsides, you’re just exhausted. It’s workaround fatigue.’ (Arash Malekzadeh/KQED)

Kadel said he’d once suffered through cancer. “This was way worse.”

Those views are not necessarily unheard of in these types of situations, said Jonah Silas Sheridan, a Bay Area computer security consultant for nonprofits.

“It’s a big deal for security practitioners,” Sheridan said. “How can you secure someone’s systems when you are in a place of sleeplessness and arrhythmia yourself?”

Meanwhile, the radio news team and their engineers went full-on MacGyver to rig a system that kept the news on the air. In normal times, reporters and editors were fully networked, with universal real-time access to every story, revision and audio clip. Now, every story had to pass through three or four computers. Audio had to be uploaded to one site, then downloaded somewhere else. Cellphones and the instant messaging app Slack were conscripted into the workflow. The motto: Whatever works.

As weeks turned into months, the newsfolk reported on the congressional health care debate, the move to end DACA, the total solar eclipse and the eruption of far-right rallies in Charlottesville and other cities, all using a system that took three to four times the usual amount of time.

The audience most likely didn’t notice. But the staff found the relentless workarounds, jury-rigging and duct-taping exhausting.

“Initially, we all had this attitude, ‘The show must go on,’ ” said Mina Kim, the afternoon drive-time news anchor. “After that surge of adrenaline and can-do attitude subsides, you’re just exhausted. It’s workaround fatigue.”

I asked John Reilly, who’s done a lot of consulting in his career, if he’d ever seen an organization experience the level of disruption KQED had. “No, not through an attack,” he said.

How could it have happened? And how much at risk are other companies?

Convenience vs. Security

Security experts like to recite an aphorism to describe the basic dilemma of their profession.

The safest computer, they say, is one sitting in a metal box in a locked room, switched off and unconnected to anything. Such a computer, goeth the Parable of the Completely Safe Workstation, is impenetrable.

And utterly useless.

“You do have to make trade-offs of usability and convenience for security,” said Paul Guthrie, a vice president for PSC, the security firm hired by KQED after the attack. “And sometimes the pendulum swings too far.”

KQED Director of IT John Reilly and Network Systems Engineer Michael Kadel. Much of the burden for rebuilding the network in a safer way fell on them. (Arash Malekzadeh/KQED)

Guthrie is one of the tech and security people both inside and outside KQED who agree the company’s pendulum over the years had strayed quite a way over into convenience territory.

Kadel and Reilly, who had been the head of IT for a year and a half when the attack hit, were frank about the vulnerabilities the ransomware exposed.

“Honestly, I’m surprised this didn’t happen long ago,” Reilly said.

Kadel put it this way: The culture of KQED, a multifaceted media organization that produces a huge amount of radio, TV, educational materials and websites, tilted toward easier collaboration. If someone wanted to put the video from a news interview into the video editing system, for instance, and then share the audio with radio reporters, they could. And they could do that from the same computer they used for daily email.

But security people know this type of one-big-network-for-all is risky. Their job, after all, is to keep malicious hackers from turning the place into “The Day the Earth Stood Still.” And when systems connect to each other like they could at KQED, malicious actors can more easily infect the entire infrastructure.

Another problem: The station’s IT department allowed employees to download and install free software from the web. IT folks call this the “granting of local admin rights.” Many in the industry also call it “A Catastrophe Waiting to Happen.” Free software is often laden with viruses, worms and other nasty extras.

Kadel said one reason IT gave the staff free rein was due to a lack of IT employees to oversee the process.

“You can’t visit 600 people to do a Flash update,” he said. “You have to let people do their own Flash update.”

Lots of consultants say companies routinely shortchange IT. But Kadel may have a point. At the time of the attack, KQED employed nine IT employees to service roughly 503 workers. That means IT represented less than 2 percent of the organization’s workforce.

A 2016 survey of media and entertainment companies by the research firm Gartner found that IT workers made up an average of 6.5 percent of employees, more than three times the level at KQED. Across all industries, the number is 5 percent.

Finally, KQED used standard anti-virus programs to protect its system. But hackers know how to get around these by altering their code. In a test run by KQED’s security consultant, just 10 out of 60 off-the-shelf anti-virus programs recognized the variant of Samas that hit the company.

“If somebody wishes to put out a new variant, it takes a little bit for the anti-virus vendors to catch up,” said Guthrie.

KQED is now plugging all these holes, redesigning its network and all of its media systems. It’s also deploying much more sophisticated anti-virus software that looks for processes that are outside the normal behavior, shutting them down. Among the new IT employees coming on board: a full-time security pro.

What Kept Working

When everything went down, the safest place for data turned out to be the cloud. KQED lost no member or donor information, because it wasn’t stored in the network. And as the attack short-circuited system after system, KQED’s network of websites kept on keeping on. How so?

“We’ve moved most of KQED.org to be hosted in the cloud, so it’s sitting on [Amazon] servers,” said Tim Olson, KQED’s chief digital officer. “It’s more disaster recovery, more efficient for everybody.”

The cloud is “certainly a strategy that a lot of smaller organizations employ, because it’s a lot of work to keep your systems patched,” said IT consultant Sheridan. And big cloud services like those of Amazon, Microsoft and Google spend tons of money to pay for cybersecurity. “That being said, you’re now in a very deep trust relationship with someone else, Sheridan said.

But in terms of maintaining access to data after the attack, the cloud was the silver lining. When everything went down and all of KQED’s storage systems became inaccessible, there sat my audio interviews, my budget numbers, my everything, safe and snug on Google Drive.

Talking About It

Below the hard truths about security vs. convenience now adorning whiteboards at KQED are a few about survival, too.

“We’ve learned a lot,” said Reilly. “People have realized that we weren’t prepared in a lot of ways. But then again, in some ways I view this as a dress rehearsal. Because we’re in an earthquake zone.”

What’s being done to fight back against the spreading threat of ransomware across the world?

Companies and individuals dealing with a ransomware attack may find help at NoMoreRansom.org, an initiative of the Netherlands’ police, Europol and the computer security companies Kaspersky Lab and McAfee. The website includes decryption tools for dozens of known ransomware infections, as well as advice on how to prevent an attack in the first place. (Update Nov. 15: The U.S. Dept. of Homeland Security has banned federal agencies from using Kaspersky products due to alleged ties to the Russian government and intelligence services.)

One thing every expert recommends: Back up like crazy.

“The number one thing to do is back up your data,” said Alexander Garcia-Tobar, co-founder of ValiMail, a San Francisco company that authenticates email in order to prevent intrusions like ransomware.

“If you’re backing up on a daily basis or even on a weekly basis, you back up to before the known infection and you cross your fingers.”

Guthrie said high-end tools designed to keep things running in the event of a catastrophe can be a lifesaver. But they are expensive.

“There’s network backup and restore systems that are very good and very fast and very expensive,” said Guthrie. “Smaller companies tend to do what they can with the resources they’re given.”

KQED decided to go public with our tale as a kind of public service, so that other organizations can learn not only from our mistakes but also from the way we kept functioning with minimal disruption to the public, despite maximum disruption internally.

Sheridan said opening up can only help.

“You might be surprised at how many different kinds of breaches, how many kinds of incidents, have happened in other organizations, and continue to happen,” he said. “We would be well-advised as a community to figure out a way to talk about these things without shame. We need to learn from them collectively.”

The Crippling Ransomware Attack on a San Francisco NPR Member Station 12 January,2018Jon Brooks

  • Nick Rubalcaba

    I’m the sole IT guy for a small company in the East bay and we were hit with ransomware two weeks ago. Luckily most of our workstations were not vulnerable to the attack, but one computer managed to encrypt about half of the data on our file server. It took about two days of lost worktime to get everything restored from a cloud backup, but there isn’t much else to do to prevent future attacks without being prohibitively expensive or cumbersome.

  • Tushar Borole

    ‘In some ways I view this as a dress rehearsal. Because we’re in an earthquake zone.’

  • Rupert Clayton

    tl;dr: Effective patch management and a SIEM will serve you better than Acme anti-virus. You can get much more security without trading away all convenience.

    This was very interesting as an insight into the devastating effects of a ransomware attack within a particular organization. Thanks to KQED for being so open about this and to Jon Brooks for writing it up. I do think the article inadvertently fosters some misunderstandings about how these attacks happen and how they can best be prevented. Given that IT security is a complex field with a lot of marketing hype the confusion is understandable.

    Some background: the Samas ransomware that hit KQED on June 15, 2017 was first detected in March 2016. Its operators use a variety of network security tools to find routes into a corporate network, but the ultimate goal is to gain control of some machine from which it can then spread further. (One disabled machine is an inconvenience; dozens of disabled machines add up to a disaster.) Like most other malware, Samas gets control of that first machine by exploiting a known vulnerability – basically some software flaw, almost always one that has been known for months or years. One vulnerability that Samas is known to exploit is a bug in the RedHat JBoss server code. This vulnerability (CVE-2010-0738) has been known since April 2010, and an update to fix the bug (a “patch”) was released immediately. Once the first machine is breached, Samas then uses Microsoft Active Directory to locate other machines to infect and user accounts to compromise.

    So how could this kind of attack have been prevented? Reporting often tends to focus on anti-virus products because that’s what consumers are most familiar with. But that’s a minor factor in defending against this kind of attack. A very important part of any effective defense is to reliably patch the entire environment – the vast majority of malware attacks like this get a toehold through a machine with out-of-date software. There’s a reliable way to address this: through a patch-management product. There are several on the market. The one I work with would manage the 500-600 computers at a cost of about $3,000. Add in 100 or servers, like the one running an unpatched version of JBoss, and the cost is still probably under $7,000.

    Good patch management will prevent the vast majority of attacks, but it’s not a complete guarantee. Another important consideration is to quickly and accurately identify this kind of attack while in progress. The industry’s name for this type of analysis and identification tool is “security information and event management” (SIEM) software. SIEMs used to only be for the largest companies, but the advent of cloud-based products in the past five years has made them a necessary component of the security defense for small and mid-sized companies, too.

    Smaller companies will likely need to rely on co-management services to help them make sense of what the SIEM finds and act on that knowledge – basically this means a contract with an experienced IT security provider who will be monitoring your network around the clock and is delegated to act on your behalf. These services would cost a company more than just patch management, but a lot less than the lost productivity from two months of ransomware hell.

    Lastly, I think the article’s characterization of the trade-off between security and convenience misses the point a little. While it’s entirely true that the most secure computer is the one that’s turned off and unplugged, people and companies can achieve a lot more security with very little loss of convenience. Running up-to-date software isn’t less convenient for the user than running older versions with gaping vulnerabilities. Sure, it may require effort from your IT and security staff, and some investment by management, but it doesn’t need to interfere with your ability to collaborate freely. Segmenting networks and removing local administrator access rights may be a little inconvenient, but good planning can minimize that impact, too.

    Those CEOs who are spurred to take IT security seriously by reading articles like this will spare their employees the agony of slogging through their work for months without modern IT infrastructure. But the CEOs that are persuaded to “fix” these issues by switching to a different anti-virus vendor are gambling with their employees’ livelihoods.

  • rocketryguy

    Kapersky is basically no longer a credible security vendor, as they have been coopted by the GRU, which means the nomoreransomware.org link is now on shaky ground, thanks Kapersky!

    Security, Convienence, Cheap – Pick two. (Hint, there’s nothing more expensive than cheap infrastructure).

    • Ruby

      Thank you, I was just about about to say the same thing.

      I hate trusting Google, Amazon, et al with all of my organization’s data, but they have the resources to protect from this kind of attack. No realistic IT department does. I know journalists hate to change their ways, but any organization that relies on the Internet to accomplish their mission, should be letting the pros handle that instead of doing it themselves.

      • rocketryguy

        At least with Google and Amazon, you understand it’s an evil empire based on money, and they generally have decent business practices. One note, if you’re going to use AWS, they have really good redundancy tools… that almost everybody forgot to use. The cloud isn’t strictly speaking just somebody else’s computer, but to make real use of the cloudy elements, you do need to understand a bit how it works, or it really is just shifting to another computer that’s somewhat better protected.

Author

Jon Brooks

Jon Brooks is the host and editor of KQED’s health and technology blog, Future of You. He is the former editor of KQED’s daily news blog, News Fix. A veteran blogger, he previously worked for Yahoo! in various news writing and editing roles. He was also the editor of EconomyBeat.org, which documented user-generated content about the financial crisis and recession. Jon is also a playwright whose work has been produced in San Francisco, New York, Italy, and around the U.S. He has written about film for his own blog and studied film at Boston University. He has an MFA in Creative Writing from Brooklyn College.