On Monday, the Internet received another reminder about its sad state of security. It was discovered that Dell decided to compromise their users’ Internet security in a way that’s difficult to top.

As elaborated further in this post, Dell, in tandem with Google, made it possible for anyone on earth, you or me, to break every single type of HTTPS connection that Dell users were making (including HPKP connections)—shiny lock icons be damned. Their reason?

to provide the system service tag to Dell online support allowing us to quickly identify the computer model, making it easier and faster to service our customers.

http://en.community.dell.com/dell-blogs/direct2dell/b/direct2dell/archive/2015/11/23/response-to-concerns-regarding-edellroot-certificate

In other words, Dell made it possible for anyone to create a 100k+ strong botnet, and for anyone to steal Dell user’s identities and financial information, so that they could tell what computer model their customers were using.

You’d be forgiven if you were tempted to invoke Hanlon’s razor in this situation: “Never attribute to malice that which is adequately explained by stupidity.”

However, the evidence suggests there was not just stupidity at play, and I’ll explain why:

#1: This was not the first time this happened.

Lenovo did the exact same thing with Superfish (but for worse reasons, to spam their own customers with ads), and when Lenovo was discovered abusing their users the news spread throughout the Internet like wildfire. That event also happened recently, only a few months ago, so Dell cannot say, “we forgot” or “we were too dumb to realize we weren’t supposed to be doing this.”

#2: Dell’s explanation does not make sense.

If I were a decision-maker at Dell faced with the decision of “how do I quickly get my support team the model number of a computer?”, I would not even consider the idea of doing what Dell did. It is simply unnecessary to install a root cert into Google Chrome’s local trust store to solve that problem. It’s almost like choosing to install a GPS tracker in your car in order to give your house a new kitchen.

Incidentally, Dell has another root-cert installed on your machine that few are paying attention to and Dell themselves are also currently ignoring. See the bottom of this article:

However, Dell did not address the DSDTestProvider self-signed certificate that we discovered yesterday. We have contacted Dell regarding this second certificate and will update this story when we receive an answer.

#3: Governments want you and your web browser to be insecure.

See, security is for government agencies, not for ordinary people. Our own government has been quite consistent on this point. No Security For You! You might be a terrorist.

Why? If browsers had strong security, their global spying operation would be undermined. Intelligence Agencies (IA) believe they need to be able to spy on everyone Because Terrorism, and seem to underestimate the effectiveness of simply not creating terrorists.

Thus, governments infiltrate Silicon Valley companies and successfully compromise standards bodies, all in the name of leaving you without security in order to protect you from insecurity.

 

Unfortunately for the IAs, there’s a small problem with their plan: you cannot compromise the security of the Internet without compromising yourself. I wonder how many more times the IAs will need to hear this before they understand? Perhaps it will be after the Director of the Central Intelligence Agency’s email gets hacked by a stoner, compromising over 3,500 federal employees?

No? OK, maybe it will be about a month later, when the same group, “Crackas With Attitude”, hack into the email of the FBI Deputy Director tasked with tracking them down?

Let that run through your head one more time: the “Intelligence” Agencies have been so successful in sabotaging attempts to fix the Internet’s security that drugged teens are now compromising National Security. These are the folks still calling for more insecurity? In Russian we have a word for this: позор.

I don’t know what comes after intoxicated hacktivists toying with the FBI and CIA, and I’m not eager to find out. There really are lives at stake. Hopefully this glimmer of enlightenment is a sign of things to come.

 

3 Tweets To Understand Government Sabotage Of Internet Security

Over on Twitter, I was asked to give a brief general primer of how sabotage of the Internet’s security works. Here it is in three easy pieces:

  1. Government agencies infiltrate Internet companies to ensure that you are insecure so they can spy on you.
  2. Their agents sabotage software, standards, whatever's necessary, and they do this by using bullshit as their primary weapon.
  3. Since most people are ignorant regarding the technicals, and many people are just dumb/gullible, this works remarkably well.
This collection is linked to here, a more detailed exposition can be found on CACert's wiki, and an example of an NSA-sabotaged standard is here. See also Wired's exposé, NSA’s Decade-Long Plan to Undermine Encryption Includes Backdoors, Stolen Keys, Manipulating Standards.

 

What Does This Have To Do With Google?

While reading tweets about Dell's customer abuse, I noticed this tweet from @SwiftOnSecurity:

“Remember”, she said, “Local Root CAs OVERRIDE certificate pinning.”

Anyone who understands what certificate pinning is should fall out of their chairs upon reading that.

Certificate pinning is the “platinum standard” of Internet security. You cannot make an Internet connection more secure than that (except perhaps through quantum cryptography). However now, literally everyone could override every single browser safeguard—including certificate pinning—for these machines.

Whose fault is this?

This time, the blame falls on Google’s shoulders. Multiple Google employees confirmed this behavior.

Some employees even considered it a laughing matter:

 

Compromising HPKP Certificate Pinning In One Paragraph

Google is the company behind the IETF certificate pinning RFC that web browsers use, called HPKP (HTTP Public Key Pinning).

In Section 2.6 of that document, there is a seemingly innocuous paragraph that reads:

It is acceptable to allow Pin Validation to be disabled for some Hosts according to local policy. For example, a UA may disable Pin Validation for Pinned Hosts whose validated certificate chain terminates at a user-defined trust anchor, rather than a trust anchor built-in to the UA (or underlying platform).

A straightforward reading of this paragraph suggests that Dell should not have been able to override HPKP pins because Dell falls under the category of "underlying platform" and also because the certificate they installed into Google Chrome's "local trust store" was not "user-defined".

However, as we saw above, multiple Googlers confirmed that HPKP pins can be overwritten, and according to one Googler, the paragraph above is the reason why.

So what’s going on?

There are several possible interpretations, one or more of which could be true:

  • Google is violating its own RFC by making it possible for Dell to destroy the usefulness of HPKP
  • There is a bug in the RFC in the sense that its language is too vague
  • Dell's behavior was not anticipated by the RFC authors and so they did not think to secure HPKP from this type of attack (or stupidity; the two can be indistinguishable)
  • Someone at Google doesn't want HPKP to work as it should
Well, what does Google have to say about all this?

Dissecting Google's Response

So far, Google's response appears to have arrived in the form of a blog post written by one of the RFC's principle authors.

In it, the author gives 3 reasons for why Chrome allows HPKP pins to be overwritten using a “local trust store”.

Let’s go through them and see whether they stand up to scrutiny.

Are The Reasons For Using A Local Trust Store To Overwrite HPKP Pins Legitimate?

Reason #1: "debugging TLS connections":
There are legitimate reasons to proxy TLS connections — not least of which is debugging. Personally I am extremely skeptical of the value of AV/IDS/IPS/DLP proxies, but some people use them on the computers they own. To them, that’s legitimate use, too.
Is it necessary to have a local trust store that can override HPKP pins in order to debug a TLS connection? No, it is not.

If people want to “debug TLS connections”, they could still do that in a special “debug mode” that the user never sees. It doesn’t require breaking HPKP for everyone.

Reason #2: “it’s futile to try”:

A user program cannot defend against software running at the same or a higher level of privilege, and it is pointless to try. The effort will be either entirely wasted, or will outweigh any marginal, temporary gains.
One of the most fundamental principles of information security is reducing the attack surface, meaning that when something is unnecessary and can lead to a vulnerability, it must be removed.

Is there anything Google could do to make it more difficult for Dell to compromise their customers? Yes, there is. The most obvious thing would be to not allow roots within the local trust store to override HPKP pins.

Had Google removed the red carpet to disabling all of its safeguards, where would that have left Dell? Sure, they could still compromise Chrome, but there would be no “Google Approved” method of doing so. Dell would either:

  1. Have to give up, or:
  2. Literally install malware on their computers. This would likely result in criminal charges, not to mention serious financial and reputational damage.
Are there any downsides to the user if Google were to do this? There do not appear to be any. Even "Hostile Pinning" can be addressed with a simple user interface for resetting pins. Other perfectly reasonable suggestions have been made, but they appear to be falling on deaf ears.

Reason #3: “computers should do what their owners want”:

Computers should do what their owners want, or at least give the owner priority over the desires of a remote server operator.
Indeed, if computers “should do what their owners want”, then shouldn't they establish secure Internet connections? That means Google Chrome has to respect the keys that are sent.

So this is in fact a reason in favor of not compromising HPKP.

 

Please note, I am not accusing any specific person of being a government spook intent on weakening security, but I can’t help notice the parallels in this situation with what is known about how Internet standards get compromised. Since these things seem to happen more than they get talked about, I think it’s only fair to raise the concern, and clearly I am not the first to do so. It remains to be seen whether this vulnerability will be closed, or whether new “reasons” will be given for keeping it around.

 

Conclusion: Close The HPKP Backdoor

We discussed the extent to which Dell compromised its users' security and the absurd justification they gave for doing so. We've reviewed some of the history, motivations, and techniques that Intelligence Agencies sometimes use to subvert Internet security. We went over the reasons why their activities make everyone, including themselves, less secure. We reviewed the reasons that were given for compromising HPKP pins in Google Chrome and we found none to be valid.

In light of these facts, we recommend extra scrutiny be given to all justifications for weakening the security of protocols and software. Security is about designing digital safety belts that will work when when you need them to. It is not about creating widgets that make it simple for random strangers to disable your safety belt at will.

This root store bypass is a backdoor that completely subverts the purpose of certificate pinning. After Superfish 1, 2, and 3, we should question the wisdom of a policy that has repeatedly left hundreds of thousands of people vulnerable for very questionable reasons.

I’ve opened a bug report (unfortunately restricted access since it’s security related), and I’m hoping the Chrome team will address this problem so that it doesn’t happen again.

EDIT: November 25, 2015: Well that was fast. Google immediately says they “WontFix”. They basically repeat the same nonsense. EDIT June 14, 2016: However, there is now a new related issue that someone else opened that’s getting some attention.

EDIT: November 27, 2015: I’ve now opened a bug report on bugzilla for the same issue with Firefox. EDIT: and there is also this existing issue from 2014 that someone else reported.

 

ERROR: Unable to embed this tweet (possibly deleted).

You can find the author on twitter here.

Thanks to Andrea Devers, Simon Grondin, Martin Köppelmann, Ryan Shea, and Vlad Zamfir for reviewing drafts of this post.

Like this post? Please support us so we can make more!

Comments