What We Can Learn from the Capital One Hack

02
Aug 19

What We Can Learn from the Capital One Hack

On Monday, a former Amazon employee was arrested and charged with stealing more than 100 million consumer applications for credit from Capital One. Since then, many have speculated the breach was perhaps the result of a previously unknown “zero-day” flaw, or an “insider” attack in which the accused took advantage of access surreptitiously obtained from her former employer. But new information indicates the methods she deployed have been well understood for years.

What follows is based on interviews with almost a dozen security experts, including one who is privy to details about the ongoing breach investigation. Because this incident deals with somewhat jargon-laced and esoteric concepts, much of what is described below has been dramatically simplified. Anyone seeking a more technical explanation of the basic concepts referenced here should explore some of the many links included in this story.

According to a source with direct knowledge of the breach investigation, the problem stemmed in part from a misconfigured open-source Web Application Firewall (WAF) that Capital One was using as part of its operations hosted in the cloud with Amazon Web Services (AWS).

Known as “ModSecurity,” this WAF is deployed along with the open-source Apache Web server to provide protections against several classes of vulnerabilities that attackers most commonly use to compromise the security of Web-based applications.

The misconfiguration of the WAF allowed the intruder to trick the firewall into relaying requests to a key back-end resource on the AWS platform. This resource, known as the “metadata” service, is responsible for handing out temporary information to a cloud server, including current credentials sent from a security service to access any resource in the cloud to which that server has access.

In AWS, exactly what those credentials can be used for hinges on the permissions assigned to the resource that is requesting them. In Capital One’s case, the misconfigured WAF for whatever reason was assigned too many permissions, i.e. it was allowed to list all of the files in any buckets of data, and to read the contents of each of those files.

The type of vulnerability exploited by the intruder in the Capital One hack is a well-known method called a “Server Side Request Forgery” (SSRF) attack, in which a server (in this case, CapOne’s WAF) can be tricked into running commands that it should never have been permitted to run, including those that allow it to talk to the metadata service.

Evan Johnson, manager of the product security team at Cloudflare, recently penned an easily digestible column on the Capital One hack and the challenges of detecting and blocking SSRF attacks targeting cloud services. Johnson said it’s worth noting that SSRF attacks are not among the dozen or so attack methods for which detection rules are shipped by default in the WAF exploited as part of the Capital One intrusion.

“SSRF has become the most serious vulnerability facing organizations that use public clouds,” Johnson wrote. “The impact of SSRF is being worsened by the offering of public clouds, and the major players like AWS are not doing anything to fix it. The problem is common and well-known, but hard to prevent and does not have any mitigations built into the AWS platform.”

Johnson said AWS could address this shortcoming by including extra identifying information in any request sent to the metadata service, as Google has already done with its cloud hosting platform. He also acknowledged that doing so could break a lot of backwards compatibility within AWS.

“There’s a lot of specialized knowledge that comes with operating a service within AWS, and to someone without specialized knowledge of AWS, [SSRF attacks are] not something that would show up on any critical configuration guide,” Johnson said in an interview with KrebsOnSecurity.

“You have to learn how EC2 works, understand Amazon’s Identity and Access Management (IAM) system, and how to authenticate with other AWS services,” he continued. “A lot of people using AWS will interface with dozens of AWS services and write software that orchestrates and automates new services, but in the end people really lean into AWS a ton, and with that comes a lot of specialized knowledge that is hard to learn and hard to get right.”

In a statement provided to KrebsOnSecurity, Amazon said it is inaccurate to argue that the Capital One breach was caused by AWS IAM, the instance metadata service, or the AWS WAF in any way.

“The intrusion was caused by a misconfiguration of a web application firewall and not the underlying infrastructure or the location of the infrastructure,” the statement reads. “AWS is constantly delivering services and functionality to anticipate new threats at scale, offering more security capabilities and layers than customers can find anywhere else including within their own datacenters, and when broadly used, properly configured and monitored, offer unmatched security—and the track record for customers over 13+ years in securely using AWS provides unambiguous proof that these layers work.”

Amazon pointed to several (mostly a la carte) services it offers AWS customers to help mitigate many of the threats that were key factors in this breach, including:

Access Advisor, which helps identify and scope down AWS roles that may have more permissions than they need;
GuardDuty, designed to raise alarms when someone is scanning for potentially vulnerable systems or moving unusually large amounts of data to or from unexpected places;
The AWS WAF, which Amazon says can detect common exploitation techniques, including SSRF attacks;
Amazon Macie, designed to automatically discover, classify and protect sensitive data stored in AWS.

William Bengston, formerly a senior security engineer at Netflix, wrote a series of blog posts last year on how Netflix built its own systems for detecting and preventing credential compromises in AWS. Interestingly, Bengston was hired roughly two months ago to be director of cloud security for Capital One. My guess is Capital One now wishes they had somehow managed to lure him away sooner.

Rich Mogull is founder and chief technology officer with DisruptOPS, a firm that helps companies secure their cloud infrastructure. Mogull said one major challenge for companies moving their operations from sprawling, expensive physical data centers to the cloud is that very often the employees responsible for handling that transition are application and software developers who may not be as steeped as they should in security.

“There is a basic skills and knowledge gap that everyone in the industry is fighting to deal with right now,” Mogull said. “For these big companies making that move, they have to learn all this new stuff while maintaining their old stuff. I can get you more secure in the cloud more easily than on-premise at a physical data center, but there’s going to be a transition period as you’re acquiring that new knowledge.”

Since news of the Capital One breach broke on Monday, KrebsOnSecurity has received numerous emails and phone calls from security executives who are desperate for more information about how they can avoid falling prey to the missteps that led to this colossal breach (indeed, those requests were part of the impetus behind this story).

Some of those people included executives at big competing banks that haven’t yet taken the plunge into the cloud quite as deeply as Capital One has. But it’s probably not much of a stretch to say they’re all lining up in front of the diving board.

It’s been interesting to watch over the past couple of years how various cloud providers have responded to major outages on their platforms — very often soon after publishing detailed post-mortems on the underlying causes of the outage and what they are doing to prevent such occurrences in the future. In the same vein, it would be wonderful if this kind of public accounting extended to other big companies in the wake of a massive breach.

I’m not holding out much hope that we will get such detail officially from Capital One, which declined to comment on the record and referred me to their statement on the breach and to the Justice Department’s complaint against the hacker. That’s probably to be expected, seeing as the company is already facing a class action lawsuit over the breach and is likely to be targeted by more lawsuits going forward.

But as long as the public and private response to data breaches remains orchestrated primarily by attorneys (which is certainly the case now at most major corporations), everyone else will continue to lack the benefit of being able to learn from and avoid those same mistakes.

Tags: , , , , , , , , , , , ,

You can skip to the end and leave a comment. Pinging is currently not allowed.

More on Backdooring (or Not) WhatsApp

Yesterday, I blogged about a Facebook plan to backdoor WhatsApp by adding client-side scanning and filtering. It seems that I was wrong, and there are no such plans.

The only source for that post was a Forbes essay by Kalev Leetaru, which links to a previous Forbes essay by him, which links to a video presentation from a Facebook developers conference.

Leetaru extrapolated a lot out of very little. I watched the video (the relevant section is at the 23:00 mark), and it doesn’t talk about client-side scanning of messages. It doesn’t talk about messaging apps at all. It discusses using AI techniques to find bad content on Facebook, and the difficulties that arise from dynamic content:

So far, we have been keeping this fight [against bad actors and harmful content] on familiar grounds. And that is, we have been training our AI models on the server and making inferences on the server when all the data are flooding into our data centers.

While this works for most scenarios, it is not the ideal setup for some unique integrity challenges. URL masking is one such problem which is very hard to do. We have the traditional way of server-side inference. What is URL masking? Let us imagine that a user sees a link on the app and decides to click on it. When they click on it, Facebook actually logs the URL to crawl it at a later date. But…the publisher can dynamically change the content of the webpage to make it look more legitimate [to Facebook]. But then our users click on the same link, they see something completely different — oftentimes it is disturbing; oftentimes it violates our policy standards. Of course, this creates a bad experience for our community that we would like to avoid. This and similar integrity problems are best solved with AI on the device.

That might be true, but it also would hand whatever secret-AI sauce Facebook has to every one of its users to reverse engineer — which means it’s probably not going to happen. And it is a dumb idea, for reasons Steve Bellovin has pointed out.

Facebook’s first published response was a comment on the Hacker News website from a user named “wcathcart,” which Cardozo assures me is Will Cathcart, the vice president of WhatsApp. (I have no reason to doubt his identity, but surely there is a more official news channel that Facebook could have chosen to use if they wanted to.) Cathcart wrote:

We haven’t added a backdoor to WhatsApp. The Forbes contributor referred to a technical talk about client side AI in general to conclude that we might do client side scanning of content on WhatsApp for anti-abuse purposes.

To be crystal clear, we have not done this, have zero plans to do so, and if we ever did it would be quite obvious and detectable that we had done it. We understand the serious concerns this type of approach would raise which is why we are opposed to it.

Facebook’s second published response was a comment on my original blog post, which has been confirmed to me by the WhatsApp people as authentic. It’s more of the same.

So, this was a false alarm. And, to be fair, Alec Muffet called foul on the first Forbes piece:

So, here’s my pre-emptive finger wag: Civil Society’s pack mentality can make us our own worst enemies. If we go around repeating one man’s Germanic conspiracy theory, we may doom ourselves to precisely what we fear. Instead, we should ­ we must ­ take steps to constructively demand what we actually want: End to End Encryption which is worthy of the name.

Blame accepted. But in general, this is the sort of thing we need to watch for. End-to-end encryption only secures data in transit. The data has to be in the clear on the device where it is created, and it has to be in the clear on the device where it is consumed. Those are the obvious places for an eavesdropper to get a copy.

This has been a long process. Facebook desperately wanted to convince me to correct the record, while at the same time not wanting to write something on their own letterhead (just a couple of comments, so far). I spoke at length with Privacy Policy Manager Nate Cardozo, whom Facebook hired last December from EFF. (Back then, I remember thinking of him — and the two other new privacy hires — as basically human warrant canaries. If they ever leave Facebook under non-obvious circumstances, we know that things are bad.) He basically leveraged his historical reputation to assure me that WhatsApp, and Facebook in general, would never do something like this. I am trusting him, while also reminding everyone that Facebook has broken so many privacy promises that they really can’t be trusted.

Final note: If they want to be trusted, Adam Shostack and I gave them a road map.

Hacker News thread.

Disabling Security Cameras with Lasers

Clive RobinsonAugust 2, 2019 6:05 PM

@ All,

We have discused this in the past at various times on this blog.

One way to “jam” a camera is with two semiconductor lasers one red one green. You don’t collimate to burn the sensor, but to confuse the sensor system.

With cheap cameras especially the “low light black and white” they don’t have IR filters over the sensor so an IR laser can “white out” the image with little difficulty.

The cameras that are most susceptible to attack are the “night vision” type that actually use IR spot lights past experience has taught me not only are they quite fragile they are also quite expensive.

Oddly whilst you can buy systems to protect CCTV cameras from such attacks it appears nobody realy wanted to buy them outside of the military…

Something tells me that after this, police forces will be holding out their hand to those who have a hold on the public purse strings…