July 8, 2012

How much capacity do you really have?

I recently bought a Nissan Leaf, and I'm going to share the joys and travails of driving one.

We were going to head out blueberry picking today. Our destination was 34 miles away, and the Leaf claimed it had 80 miles of charge available. "Perfect!" I thought - I could exercise it at its full range, and trickle charge enough overnight to get to work tomorrow, where I can fully charge it.

The first five miles of our trip was uphill on an interstate. By the end of that, the Leaf claimed we had 47 miles of charge left. We turned around, went home, and switched to the Sienna for our blueberry picking adventure.

What happened here? Two things: route selection, and mileage variability. The route selection on the Leaf isn't what I'm used to: on my prior vehicles (Toyota/Lexus), when selecting a route, it would display several options. The Nissan interface didn't, although I'm sure it is there somewhere (something to go look for!). So I had selected the "long but fast route," which added 7 miles, but saved 3 minutes at normal driving speed.

Which leads to mileage variability: 80 mile range is really some number of kilowatt-hours; and different driving has different miles-per-kilowatt-hour efficiency. "Optimal" driving is at 38 mph; the "long but fast route" involved speeds that are at least 50% higher, with a concomitant reduction in efficiency. While 80 miles didn't assume optimal driving, it probably didn't expect such high speed driving.

Feature desire: If you put in a route, and the expected fuel efficiency for normal driving on that route won't get you home on your existing charge, give a warning. This probably requires some better GIS integration, but shouldn't be out of the realm of possibility.

September 16, 2011

The Unreliability of the Domain Name Service

Consider the case of DNS (the Domain Name Service). This innocuous-seeming protocol, which merely translates hostnames (like into IP addresses (like is such an important foundation of the Internet that, without it, humans would have had a serious challenge building and engaging meaningfully with such a vast distributed system. The web as we know it would not exist without DNS. The beauty of DNS is in its simplicity: you ask a question, and you get a response.

But if you can't get a response, then nothing else works -- your web browser can't load up news for you, your email client can't fetch email, and your music player can't download the latest songs. It's important to ensure the DNS system is highly available; counterintuitively, we built it assuming that its components are likely to fail. And from that we can learn how to build other survivable infrastructures.

First, DNS primarily uses UDP, rather than TCP, for transport. UDP (often nicknamed the Unreliable Data Protocol) is more like sending smoke signals than having a conversation; the sender has no idea if their message reached the recipient. A client sends a UDP query to a name server; the name server sends back a UDP answer (had this been implemented in TCP, the conversation would instead have taken several round trips).

Because DNS has no reliability assertion from UDP (and, frankly, the reliability assertion that TCP would have provided isn't worth much, but at least provides failure notification), the implementations had to assume -- correctly -- that failure would happen, and happen regularly. So failure was planned for. If the client does not get a response within a set time window, it will try again - but then the client may query a different server IP address. Because the DNS query/response is accomplished within a single packet, there is no need for server stickiness.

An array of DNS servers can be placed behind a single IP address, with simple stateless load-balancing required - no complex stateful load balancers required (higher end DNS systems can even use IP-anycasting, to have one IP address respond from multiple geographic regions, with no shared state between the sites). Clients can and do learn which servers are highly response, and preferentially use those.

DNS also has built into itself other means of reliability, like the TTL (time to live). This is a setting associated with every DNS response which indicates how long the response is valid. A client therefore does not need to make queries for some time; if a name server fails, a client may not notice for hours.

On top of this failure-prone infrastructure -- an unreliable transport mechanism, servers that might fail at any time, and an Internet that has an unfortunate tendency to lose packets -- a highly survivable system begins to emerge, with total DNS outages a rare occurrence.

August 19, 2011

Password Weakness

Randall Munroe opines in xkcd on password strength, noting that we've trained people to "use passwords that are hard for humans to remember, but easy for computers to guess." He's both right and wrong.

First off, the security industry owes Randall a debt of gratitude for this comic; people who don't normally interact with security technologies (or only grudgingly) are discussing and debating the merits of various password algorithms, and whether "correct horse battery staple" is, in fact, more memorable and more secure than "Tr0ub4dor&3". That's an important conversation to have.

Is it more secure?

Randall plays a trick on the audience, by picking a single strawman password implementation, and showing how weak it is compared to a preferred model. He also limits himself to a specific attack vector (against an online oracle), which makes the difference between the two seem larger than it really is.

Consider the following (obvious) variants of the "weak" password algorithm presented in the comic: "Tr0ub4dor&3!", "Troubbador&3", "2roubador&3". None of these match the algorithm presented - so they don't fit into the 28 bits of entropy of concern. That doesn't make them perfect, I merely note that Randall arbitrarily drew his line around "likely passwords" where he wanted to. That's not necessarily unreasonable: for instance, if a password scheme requires 8 characters, including at least one upper case, one lower case, one number, and one symbol, assuming people will pick "Upper case, five lower case with a number thrown in, symbol that is a shifted-number" probably isn't a bad idea, and lets you ignore 99.9975% of possible 8-character passwords. But it is unreasonable if you're arguing that your specific model might be better.

Let's say that we give users the simple model proposed: pick four random words. People fail at picking random things; see Troy Hunt's analysis of the passwords revealed in the Sony Pictures breach. So if you let the user pick the word list, you'll end up with common phrases like "patriots football super bowl" or "monkey password access sucks", and adversaries will start there. Or, we can give users their passphrases, and probably discover later that there was a bug in the random number generator used to select words, and half of our users have the passphrase "caffeine programmer staccato novel".

Randall is correct that, when it comes to user-memorized secrets, longer is better. So is less predictability. Most password rules are designed to move from easy predictability (common words) to harder predictability (common words plus some interspersed silly keystrokes).

The real risk

Going back to Troy Hunt's analysis, the real risk isn't that someone will use an online oracle to brute force your password or passphrase. The real risk is that some password holder will be breached, and like 67% of users, you'll have used the same password on another site. Password strength doesn't help at all with that problem.

But which one?

The answer is neither. If you're using either password scheme demonstrated by Randall, change it (e.g., add some random symbols between your words), as it's now more likely to be an adversarial target. The real question is how do we get away from passwords? SSL certificates - for all their issues - are one option. One time passwords - generated either by a dedicated token or application, or out-of-band, via SMS - also are an interesting choice.

But if the only threat you're worried about are online oracle attacks, you can defend against those by looking for them, and making them harder for adversaries to conduct. But that's a mostly losing battle in the long run.

March 24, 2011

How certificates go bad

The security echo chamber has gotten quite loud over the last few days over the Comodo sub-CA bogus certificate issuance. This is a good opportunity to look at what happened, why this isn't as startling as some might think, and general problems in the SSL CA model.

A primer on certificates and authorities

Certificates are built on top of asymmetric cryptographic systems - systems where you have a keypair that is split into a private half (held closely by the owner) and a public half (distributed widely). Information encrypted with one half is only decryptable with the other half. If you encrypt with the public key, we call it encryption (the information is now secret and can only be read by the private key owner); if you encrypt with the private key, we call it signing (the information can be verified by anyone, but only you could have generated it). There are additional optimization nuances around hashes and message keys, but we'll gloss over those for now.

Anyone can generate asymmetric keypairs; what makes them interesting is when you can tie them to specific owners. The SSL model is based on certificates. A certificate is just someone's public key, some information about that public key, and a signature of the key and information. The signature is what's interesting -- it's generated by another keyholder, whose private key & certificate we call a certificate authority (CA).

"You've got me. Who's got you?"

How do we trust a CA that has signed a certificate? It itself might be signed by another CA, but at some point, we have to have a root of trust. Those are the CAs that our web browsers and operating systems trust to sign other certificates. You should take a gander around the list (Mozilla ships about 50 organizations as root CAs, Internet Explorer far more). Those roots can directly sign any SSL certificate, or can sign an intermediate CA, which then signs certificates.

The most expensive part of issuing a certificate is verifying that the purchaser is authorized to hold one. Many CAs, including Comodo, have resellers who can instruct the CA to issue certificates; the reseller becomes what is known as the "Registration Authority (RA)." (Full disclosure: Akamai is a reseller of several CAs, including Comodo, although certificates we sign only work with the private keys that we hold on our platform.)

There are two major, fundamental flaws in this architecture.

First, the number of trusted CAs is immense. And each of those CAs can authoritatively sign certificates for any domain. This means that CA Disig (of the Slovak Republic) can issue authoritative certs for, the Government of Canada's website. (Full disclosure: my wife's mother is from Slovakia, and my father's side of the family is Canadian.) Fundamentally, the list of root CAs in everyone's browser contains authorities based anywhere in the world, including governments known to be hostile to theirs. A related issue is that most enterprises have their own "private CA" which signs intranet certificates; that CA becomes valid for any domain when the user adds it to their trust chain.

Second, RAs are a very weak point. Not only are they in a race to the bottom (if you can buy an SSL cert for under $50, imagine how little verification of your identity the RA can afford to do), but any one of them, if compromised, can issue certificates good for any domain in the world. And that's what happened in the case of the bogus Comodo certificates.

Kudos to Comodo for good incident response, and explaining clearly what happened. I suspect that's the rarity, not the issuance of bogus certificates.

February 16, 2011

Malware hunting

Today at the RSA Conference, Akamai Principal Security Architect Brian Sniffen is giving a talk titled "Scanning the Ten Petabyte Cloud: Finding the malware that isn't there." In Brian's talk, he discusses the challenges of hunting for malware hooks in stored HTML pages of unspecified provenance, and some tips and tricks for looking for this malicious content.

In conjunction with his talk, Akamai is releasing the core source code for our vscan software. The source code is BSD3-licensed.

We are hopeful that our experiences can be helpful to others looking for malware in their HTML.

January 27, 2011


I was reading Rafal Los over at the HP Following the White Rabbit blog discussing whether anonymous web browsing is even possible:

Can making anonymous surfing still sustain the "free web" concept? - Much of the content you surf today is free, meaning, you don't pay to go to the site and access it. Many of these sites offer feature-rich experiences, and lots of content, information and require lots of work and upkeep. It's no secret that these sites rely on advertising revenue at least partly (which relies on tracking you) to survive ...if this model goes away what happens to these types of sites? Does the idea of free Internet content go away? What would that model evolve to?

This is a great point, although insufficiently generic, and limited to the gratis view of the web -- the allegedly free content. While you can consider the web browsing experience to be strictly transactional, it can more readily be contemplated as an instantiation of a world of relationships. For instance, you read a lot of news; but reading a single news article is just a transaction in the relationships between you and news providers.

A purchase from an online retailer? Let's consider a few relations:

The buyer's relationship with:
  • the merchant
  • their credit card / alternative payment system
  • the receiver
The merchant's relationship with:
  • the payment gateway
  • the manufacturer
  • the shipping company
The payment card industry's interrelationships: payment gateways, acquiring banks, card brands, and card issuers all have entangled relationships.

The web is a world filled with fraud, and fraud lives in the gaps between these relationships (Often, relationships are only used one way: buyer gives credit card to merchant, who gives it to their gateway, who passes it into the banking system. If the buyer simply notified their bank of every transaction, fraud would be hard; the absence of that notification is a gap in the transaction). The more a merchant understands about their customer, the lower their cost can be.

Of course, this model is harder to perceive in the gratis environment, but is nonetheless present. First, let's remember:

If you're not paying for something, you're not the customer; you're the product being sold.

Often, the product is simply your eyeballs; but your eyeballs might have more value the more the merchant knows about you. (Consider the low value of the eyeballs of your average fantasy football manager. If the merchant knows from past history that those eyeballs are also attached to a person in market for a new car, they can sell more valuable ad space.) And here, the more value the merchant can capture, the better services they can provide to you.

A real and fair concern is whether the systemic risk added by the merchant in aggregating information about end users is worth the added reward received by the merchant and the user. Consider the risk of a new startup in the gratis world of location-based services. This startup may create a large database of the locations of its users over time (consider the surveillance possibilities!), which, if breached, might expose the privacy and safety of those individuals. Yet because that cost is not borne by the startup, they may perceive it as a reasonable risk to take for even a small return.

Gratis services - and even for-pay services - are subsidized by the exploitable value of the data collected. Whether or not the business is fully monetizing that data, it's still a fair question to ask whether the businesses can thrive without that revenue source.

December 16, 2010

Architecting for DDoS Defense

DDoS is back in the news again, given the recent post-Cyber Monday DDoS attacks and the Anonymous DDoS attacks targeted at various parties. This seems like a good time to remember the concepts you need in the front of your mind when you're designing to resist DDoS.

DDoS mitigation isn't a point solution; it's much more about understanding how your architecture might fail, and how efficient DDoS attacks can be. Sometimes, simply throwing capacity at the problem is good enough (in many cases, our customers just start with this approach, using our WAA, DSA, and EDNS solutions to provide that instant scalability), but how do you plan for when simple capacity might not be sufficient?

It starts with assuming failure: at some point, your delicate origin infrastructure is going to be overwhelmed. Given that, how can you begin pushing out as much functionality as possible into the edge? Do you have a set of pages that ought to be static, but are currently rendered dynamically? Make them cacheable, or set up a backup cacheable version. Put that version of your site into scaleable cloud storage, so that it isn't relying on your infrastructure.

For even dynamic content, you'd be amazed at the power of a short-term caching. A 2 second cache is all but unnoticeable to your users, but can offload significant attack traffic to your edge. Even a zero-second cache can be interesting; this lets your front end cache results, and serve them (stale) if they can't get a response from your application.

After you think about disaster resilience, you should start planning for the future. How can you authenticate users early and prioritize the requests of your known good users? How much dynamic content assembly can you do without touching a database? Can you store-and-forward user generated content when you're under heavy load?

The important point is to forget much of what we've been taught about business continuity. The holy grail of "Recovery Time Objective" (how long you can be down) shouldn't be the target, since you don't want to go down. Instead, you need to design for you Minimum Uninterrupted Service Target - the basic capabilities and services you must always provide to your users and the public. It's harder to design for, but makes weathering DDoS attacks much more pleasant.

August 30, 2010

A Cloud Balancing Act

Over at F5's Dev Central, Lori MacVittie talks about load balancing and the cloud:

When you decide to deploy an application to the cloud you have a very similar decision: do you extend your existing dwelling, essentially integrating the environment with your own or do you maintain a separate building that is still "on the premises" but not connected in any way except that it's accessible via your shared driveway.

Using a garage expansion as a metaphor for scaling laterally to the cloud is a great one, and captures a lot of the nuances to be dealt with.

I'd like to add a third option to Lori's first two, based on our experience with hundreds of enterprises -- the valet strategy. Rather than simply load balancing between multiple systems, a cloud can sit in front of multiple sites, performing application delivery functions, as well as load balancing betwixt the backend sites.

Many Akamai customers do this today. They may have several data centers of their own, and want to integrate cloud-based services seamlessly into their existing sites (like taking advantage of user prioritization to load balance between an application server and a cloud-based waiting room; or using storage in the cloud to provide failover services). Akamai's cloud accepts the end user request, and either takes care of the user locally, or gathers the necessary resources from among multiple backend systems to service the user. And that's a way to load balance transparently to the user.

July 19, 2010

Edge Tokenization

Visa released its Credit Card Tokenization Best Practices last week, giving implementors a minimum guide on how to implement tokenization. It's a good read, although if you're planning on building your own tokenizer, I'd strongly recommend reading Adrian Lane's take on the subject, including practices above and beyond Visa's for building good tokenization systems.

But I don't recommend building your own tokenizer, unless you're a payment gateway (but if you're going to, please read Adrian's guidance, and design carefully). The big goal of tokenization is to get merchants' commerce systems out of scope for PCI. And if you want to try to remove your systems from PCI scope, you should never see the credit card number.

That's why I'm really excited about Akamai's Edge Tokenization service. As discussed at, we've been beta testing a service that captures credit card data in a customer website, hands it to our partner gateways, and substitutes the returned token to our customer's systems.
Image of Akamai Edge Tokenization service.  Consumer credit card is entered into a form on a merchant website.  Akamai server captures the credit card, and sends it to a payment gateway for processing.   The payment gateway returns a token to Akamai, and Akamai delivers the token in the POST body to the merchant.   The merchant never sees the credit card.
We don't do the tokenization ourselves, so that we never have the ability to reverse the tokens. But the capture and replacement all happens inside our Level 1 merchant environment, so our customers get to simply reduce the number of their systems that see credit cards (potentially removing them from scope).

Our EdgeTokenization service is going to be publicly available early this fall, at which point we'll help the industry reduce the number of places that credit cards are even seen.

May 28, 2010

NSEC3: Is the glass half full or half empty?

NSEC3, or the "Hashed Authenticated Denial of Existence", is a DNSSEC specification to authenticate the NXDOMAIN response in DNS. To understand how we came to create it, and the secrecy issues around it, we have to understand why it was designed. As the industry moves to a rollout of DNSSEC, understanding the security goals of our various Designed Users helps us understand how we might improve on the security in the protocol through our own implementations.

About the Domain Name Service (DNS)

DNS is the protocol which converts mostly readable hostnames, like, into IP addresses (like At its heart, a client (your desktop) is asking a server to provide that conversion. There are a lot of possible positive answers, which hopefully result in your computer finding its destination. But there are also some negative answers. The interesting answer here is the NXDOMAIN response, which tells your client that the hostnames does not exist.

Secrecy in DNS

DNS requests and replies, by design, have no confidentiality: anyone can see any request and response. Further, there is no client authentication: if an answer is available to one client, it is available to all clients. The contents of a zone file (the list of host names in a domain) are rarely publicized, but a DNS server acts as a public oracle for the zone file; anyone can make continuous requests for hostnames until they reverse engineer the contents of the zone file. With one caveat: the attacker will never know that they are done, as there might exist hostname that they have not yet tried.

But that hasn't kept people from putting information that has some form of borderline secrecy into a zone file. Naming conventions in zone files might permit someone to easily map an intranet just looking at the hostnames. Host names might contain names of individuals. So there is a desire to at least keep the zone files from being trivially readable.

DNSSEC and authenticated denials

DNSSEC adds in one bit of security: the response from the server to the client is signed. Since a zone file is (usually) finite, this signing can take place offline: you sign the contents of the zone file whenever you modify them, and then hand out static results. Negative answers are harder: you can't presign them all, and signing is expensive enough that letting an adversary make you do arbitrary signings can lead to DoS attacks. And you have to authenticate denials, or an adversary could poison lookups with long-lived denials.

Along came NSEC. NSEC permitted a denial response to cover an entire range (e.g., there are no hosts between and Unfortunately, this made it trivial to gather the contents of a zone: after you get one range, simply ask for the next alphabetical host ( and learn what the next actual host is ( From a pre-computation standpoint, NSEC was great - there are the same number of NSEC signed responses in a zone as all other signatures - but from a secrecy standpoint, NSEC destroyed what little obscurity existed in DNS.


NSEC3 is the update to NSEC. Instead of providing a range in which there are no hostnames, a DNS server publishes a hashing function, and a signed range in which there are no valid hashes.. This prevents an adversary from easily collecting the contents of the zone (as with NSEC), but does allow them to gather the size of the zone file (by making queries to find all of the unused hash ranges), and then conduct offline guessing at the contents of the zone files (as Dan Bernstein has been doing for a while). Enabling offline guessing makes a significant difference: with traditional DNS, an adversary must send an arbitrarily large number of queries (guesses) to a name server (making them possibly detectable); with NSEC, they must send as many queries as there are records; and with NSEC3, they must also send the same number of requests as there are records (with some computation to make the right guesses), and then can conduct all of their guessing offline.

While NSEC3 is an improvement from NSEC, it still represents a small step down in zone file secrecy. This step is necessary from a defensive perspective, but it makes one wonder if this is the best solution: why do we still have the concept of semi-secret public DNS names? If we have a zone file we want to keep secret, we should authenticate requests before answering. But until then, at least we can make it harder for an adversary to determine the contents of a public zone.

"Best" practices in zone secrecy

If you have a zone whose contents you want to keep obscure anyway, you should consider:
  • Limiting access to the zone, likely by IP address.
  • Use randomly generated record names, to make offline attacks such as Dan Bernstein's more difficult.
  • Fill your zone with spurious answers, to send adversaries on wild goose chases.
  • Instrument your IDS system to detect people trying to walk your zone file, and give them a different answer set than you give to legitimate users.

Jason Bau and John Mitchell, both of Stanford, have an even deeper dive into DNSSEC and NSEC3.

May 26, 2010

Credit Card Tokenization

Every merchant that takes credit cards undergoes an annual ritual known as their PCI audit. Why? Because merchants take credit cards from end users, and the Payment Card Industry Data Security Standard (PCI-DSS) requires the annual audit of all environments that store cardholder data. Unfortunately, PCI audits are complicated, expensive, and distracting. But new technology may make them less necessary.

Consider what the credit card really is: It's a promise of payment; an agreement between an end user and their bank (and, by proxy, the entire card network) that enables the user to promise funds to a merchant. That's also the weakness of the system: the 16 digits of the card (plus a few other data points, like expiration date, CVV, and name) are equivalent to a public currency.

And that currency needs to be stored in a lot of places, for a long period of time. For user convenience and increased revenues, merchants want to support repeat buying without card reentry, so they need to keep the card (and to support chargebacks). For fraud detection, merchants want to check for card overuse. This leads to a threat to the currency, and much of the purpose behind the (PCI-DSS): cards at rest are a valuable, and omnipresent target.


Tokenization is a technology that replaces the credit card (a promise between a user and their bank) with a token (a promise between a merchant and their bank). With tokenization, when presented a credit card, a merchant immediately turns it over to their payment gateway, and receives a token in exchange. The token is only useful to that merchant when talking to their gateway; the public currency (credit card usable anywhere) has been converted to private scrip (token tied to merchant and gateway).

The gateway has to maintain a mapping between tokens and credit cards, so that when a merchant redeems a token, the gateway can convert the token to a credit card charge. But a merchant can use the token in their internal systems for anything they would have used a credit card for in the past: loyalty programs, fraud detection, chargebacks, or convenience.

Tokenization isn't a panacea for a merchant. The merchant still has a point in their infrastructure that takes credit cards, and that point is subject to PCI-DSS. If the merchant has a brick and mortar operation, PCI-DSS requirements will still apply there (although some POS terminals do support tokenization). But it can take a merchant a long way to defending their customers' credit cards, and represents several steps forward to a model where merchants don't need to see a credit card at all.

February 4, 2010

Why don't websites default to SSL/TLS?

When a client connects on TCP port 80 to a webserver, one of the first thing it sends is a line that tells the server what website it wants. This line looks like this:
This tells the webserver which configuration to use, and how to present content to the end-user. This effectively abstracts TCP and IP issues, and lets websites and webservers interact at the discretion of the owner. The designed user of HTTP is the administrator, who may need to host dozens of websites on a smaller number of systems.

Secure Socket Layer (SSL) and its successor, Transport Layer Security (TLS), on the other hand, were designed for exactly the opposite user. The designed user of HTTPS is the paranoid security academic, who doesn't even want to tell a server the hostname it is looking for (the fact that you were willing to tell a DNS server is immaterial). In essence, SSL requires that any server IP address can have only one website on it. When a client connects to a webserver on port 443, the first thing it expects is for the server to provide a signed certificate, that matches the hostname that the client has not yet sent. So if you connect to via SSL, you'll note you get back a different certificate: one for This is expected -- nothing on this hostname requires encryption. Similarly, for Akamai customers that are delivering whole-site content via Akamai on hostnames CNAMEd to Akamai's domain, attempts to access these sites via SSL will result in a certificate for being returned. (Aside: customers use our SSL Object Caching service to deliver objects on the hostname; customers who want SSL on their own hostnames use our WAA and DSA-Secure services).

The designs of HTTP and HTTPS are diametrically opposed, and the SSL piece of the design creates horrendous scaling problems. The server you're reading this from serves over 150,000 different websites. Those sites are actually loadbalanced across around 1600 different clusters of servers. For each website to have an SSL certificate on this network, we'd need to consume around 250 million IP addresses - or 5.75% of the IPv4 space. That's a big chunk of the 9% left as of today. Note that there isn't a strong demand to put most sites on SSL; this is just elucidating why, even if there were demand, the sheer number of websites today makes this infeasible.

Fortunately, there are paths to a solution.

Wildcard certificates
For servers that only serve hostnames in one domain, a wildcard certificate can help. If, for instance, in addition to, I had,, and, instead of needing four certificates (across those 1800 locations!), I could use a certificate for *, which would match all four domains, as well as future growth of hostnames.

You're still limited; a wildcard can only match one field, so a certificate for * wouldn't work on a site named Also, many security practitioners, operating from principles of least privilege, frown on wildcard certificates, as they can be used even for unanticipated sites in a domain.

Subject Alternate Name (SAN) certificates
A slightly more recent variety of certificate are the SAN certificates. In addition to the hostname listed in the certificate, an additional field lets you specify a list of valid hostnames for that certificate (If you look closely, the certificate on this host has a SAN field set, which includes both and * This permits a server to have multiple, disparate hostnames on one interface.

On the downside, you still only have one certificate, which is going to get larger and larger the more hostnames you have (which hurts performance). It also ties all of those hostnames into one list, which may present brand and security issues to some enterprises.

Server Name Indication (SNI)
The long term solution is a feature for TLS called Server Name Indication (SNI). This extension calls for the client to, as part of the initial handshake, indicate to the server the name of the site it is looking for. This will permit a server to select the appropriate one from its set of SSL certificates, and present that.

Unfortunately, SNI only provides benefit when everyone supports it. Currently, a handful of systems don't support SNI, most notably Windows XP and IIS. And those two major components are significant: XP accounts for 55-60% of the browser market, and IIS looks to be around 24%. So it'll be a while until SNI is ready for primetime.

January 13, 2010

The Evolution of DDoS

Last week, I spent some time with Bill Brenner (CSO Magazine) discussing DDoS attacks, and some insight into attacks over the last ten years. He put our discussion into a podcast.

December 21, 2009

Would DNSSEC have helped Twitter?

Twitter had its name servers "stolen". Would DNSSEC have helped protect them? To be brief, no. But looking at why not will help understand the limitations of DNSSEC. It helps to start with the basics of DNS, and DNSSEC.

DNS is broken down into zones. A zone is a hierarchical collections of names, roughly grouped by the periods in the hostname -- e.g., exists under three zones: . (the root zone), .com, and Each zone has a set of servers that are authoritative when answering queries in that zone, and can delegate responsibility for a subsidiary zone to a different set of servers.

With DNSSEC, each zone has a set of keys used to sign answers in the zone; the zone signing key, which signs the records, and the key signing key, which signs the zone signing key. You learn the key signing key for a zone when receiving a delegation; the delegating server adds a special record informing you of the key for the zone they're passing you to.


Looking at the incident, it appears that a compromised Twitter employee email account was used to reset an administrative account login with their registrar, and then that account was used to change the delegation for Had Twitter been using DNSSEC, Twitter would have needed to provide the public half of their key signing key to their registrar. Odds are, that would be in the same interface that was used to redirect An adversary would have been able to alter the DNSSEC delegation just as easily as they altered the zone delegation.

DNSSEC has a lot of strengths, but it isn't a magic bullet for all of the weaknesses in the DNS infrastructure.

December 18, 2009

Modeling Imperfect Adversaries

An important piece of risk assessment is understanding your adversaries. Often, this can degenerate into an assumption of perfect adversaries. Yet when we think about risk, understanding that our adversaries have different capabilities is critical to formulating reasonable security frameworks. Nowhere is this more true than in the streaming media space.

Brian Sniffen (a colleague at Akamai) recently presented a paper at FAST exploring ways of considering different adversaries, especially in the context of different business models. He presents some interesting concepts worth exploring if you're in the space:

  • Defense in breadth: The concept of using different security techniques to protect distinct attack surfaces, specifically looking at defeating how a specific adversary type is prone to conduct attacks.
  • Tag-limited adversaries: An extension to the Dolev-Yao adversary (a perfect adversary who sits inline on a communication stream), the tag-limited adversary may only have knowledge, capabilities, or desire to conduct attacks within a limited vocabulary.

His paper is also a good primer on thinking about streaming threat models.

December 15, 2009

Virtual Patching

Virtual patching, for those new to the term, is the practice of adding a rule to a Web Application Firewall (WAF) to filter out traffic that could exploit a known vulnerability in a protected application. This has triggered debate in the security community -- is this a good thing? Why would a developer fix the vulnerability if it is mitigated?

bandaid.jpgFirst off, Virtual Patching is, in fact, a good thing. The development turnaround time for a WAF rule is almost certainly shorter than the development cycle for the backend application, so this shortens your mitigation window. That shouldn't really be a topic of debate.

The interesting debate, then, is how we manage the underlying vulnerability. One school of thought argues that the WAF is only a stopgap, and developers should fix the vulnerability because that's the right thing to do. Another school's argument is a bit more complex. Don't fix the vulnerability. Fix the entire class of vulnerabilities.

If you have a vulnerability in a webapp, odds are that it's a symptom of a greater category of vulnerability. That CSRF bug on one page likely reflects a design flaw in not making GET requests nullipotent, or not using some form of session request management. Given the urgency of an open vulnerability, the developers will likely focus on fixing the tactical issue. Virtually patch the tactical vulnerability, and focus on the flaw in the foundation. It'll take longer to fix, but it'll be worth it in the long run.

December 10, 2009

DDoS thoughts

We are used to measuring the efficiency of DDoS attacks in ratios of bits-per-second. An attacker wants to consume many bits-per-second for each of his own bits-per-second that he uses. He would rather send one packet to generate a storm than have to send the storm himself. We can extend this efficiency measurement to other attacks. Let's use the name "flits per second" (for fully-loaded bits) for this more general measurement of cost and pain: sending or receiving one bit-per-second costs one flit-per-second. Doing enough computation to have an opportunity cost of one bit-per-second has a cost of one flit-per-second. Enough disk access to forestall one bit-per-second costs one flit-per-second. Now we can talk about the flit capacity of the attacker and defender, and about the ratio between flits consumed on each side during an attack.

From a defensive efficiency perspective, we have two axes to play with: first, reducing the flit-to-bit ratio of an attack, by designing optimal ways of handling traffic; and second, increasing the relative client cost to produce an attack.

To reduce the flitcost, consider the history of SYN floods. SYN floods work by sending only the first packet in the three way TCP handshake; the victim computer keeps track of the half-open connection after sending its response, and waits for the attacker's followup. That doesn't come; for the cost of a single SYN packet, the attacker gets to consume a sparse resource (half-open connections) for some period of time. The total amount of traffic needed historically was pretty minimal, until SYN cookies came along. Now, instead of using the sparse resource, targets use a little bit of CPU to generate a cryptographic message, embed it in their response, and proceed apace. What was a very effective attack has become rather ineffective; against most systems, a SYN flood has a lower flit-to-bit ratio than more advanced application layer attacks.

The other axis is more interesting, and shows why SYN floods are still prevalent even today: they're cheap to produce. They don't consume a lot of cycles on the attacking systems, and don't require interesting logic or protocol awareness. The fewer resources an attacker can consume, the more likely their attack will go unnoticed by the owners of the compromised hosts used in the attack (Case in point: look at how fast Slammer remediation happened. Why? ISPs were knocked offline by having infected systems inside). Many attack programs effectively reduce to "while (1) {attack}". If the attack is making an HTTP request, filtering the request will often generate a higher rate of requests, without changing the attacker's costs. If this higher rate has the same effect on you that the lower rate did, you didn't buy anything in your remediation. You might have been better off responding more slowly, than not at all.

In the general case, this leads us to two solution sets. Traffic filtering is the set of technologies designed to make handling attacks more efficient; either by handling the attack further out in an infrastructure, classifying it as malicious for a cheaper cost than processing it, or making processing cheaper.

Capacity increases, on the other hand, are normally expensive, and they're a risky gamble. If you increase far in excess of attacks you ever see, you've wasted money. On the other hand, if you increase by not quite enough, you're still going to be impacted by an event (and now, of course, you'll be criticized for wasting money that didn't help). Obligatory vendor pitch: this is where a shared cloud infrastructure, like a CDN, comes into play. Infrastructures that measure normal usage in terabits per second have a significantly different tolerance for attack capacity planning than most normal users.

November 23, 2009

H1N1 and telework

The nervousness around H1N1 has pretty much permeated every aspect of our lives. Remember a year or two ago, the hysteria around hand sanitizers and alcohol poisoning? Gone; in its place, we have dispensers in buildings everywhere. That's the power of the fear of H1N1.

Another place is in schooling. Not too long ago, if your kid got sick, school policy was "keep them home if they have a fever or are vomiting." Sanely, this migrated to "keep them home for 24 hours after a fever." Now, however, it is "48 hours fever-free with no medications." Some schools/daycares have added "and no symptoms either," which is moderately impractical for the kids who get a three-week long lingering cough.

This affects us in the workplace. If an employee has a small child and they don't have a stay-at-home caregiver, expect that they're going to miss more time than in prior years; and that the employee actually will be stressed about this (heck, anyone trapped at home with a no-longer-sick child on a school-day is going to end up pretty stressed). Also, you may want to suggest that employees with sick children stay at home even if they aren't the primary caregiver, just to minimize workplace infections.

Key to this is a sane telework plan. Like most things, this comes down to People, Process, and Technology.

People: Do the employee and manager have a good rapport, such that working remotely does not lead to communications failures? Can the employee work without direct management? Can the employee balance the needs of daytime home-life with work?

Process: Do you have understood ways for the employee's status to be communicated? Do other employees know how to reach them? How many hours do you expect when an employee is "working from home"?

Technology: What telework capabilities do you have? (VOIP phones in the home? VTC setups?) What about remote collaboration? (A wiki, IM, ticketing system or just email?) Do your employees have enough bandwidth at home to telework? Do you have enough in your office to support them?

It's going to happen to you -- you just need a little prep. And most of that prep? You can typeset it to hand to your auditors; it's a big piece of your DRP.

July 13, 2009

Security and hairdressing

I've become an amateur hairdresser in the past couple of years, thanks to my three year old (I suspect that, had I been unwilling to do so, her hair would be quite short right now). Along the way, I've realized that I know as much about hairdressing as I do about many of the disciplines InfoSec touches.

For those of you who've never braided hair, let's try a little manual experiment. Go get three strings. Tie them together at one end to a fixed object: maybe a railing. Now braid them: holding them extended, switch the middle and right one; then the (new) middle and left one. Repeat, each time making sure the middle one goes under the outer one. Do this a couple of times, until you're comfortable with it.

You now know as much about hairstyling as you probably know about some of the more esoteric security disciplines: executive extraction, availability analysis, crypto design, or fault isolation in Byzantine networks. You haven't had to deal with snarls, or working with multiple different hair types, or tried a French braid. Similarly, you may never have designed a communications protocol, or walked a perimeter, or managed IDS sensors on the other end of an intercontinental straw.

Yet every day, as infosec professionals, we are compelled to guide other professionals in how to do their jobs. My advice: be a bit humble. As smart and clever as we think we are, the professionals we deal with are smarter in their own disciplines.

June 26, 2009

The Problem with Password Unmasking

I disagree with this:

It's time to show most passwords in clear text as users type them. Providing feedback and visualizing the system's status have always been among the most basic usability principles. Showing undifferentiated bullets while users enter complex codes definitely fails to comply.

Most websites (and many other applications) mask passwords as users type them, and thereby theoretically prevent miscreants from looking over users' shoulders. Of course, a truly skilled criminal can simply look at the keyboard and note which keys are being pressed. So, password masking doesn't even protect fully against snoopers.

More importantly, there's usually nobody looking over your shoulder when you log in to a website. It's just you, sitting all alone in your office, suffering reduced usability to protect against a non-issue.

Even though Bruce Schneier agrees with it:

Shoulder surfing isn't very common, and cleartext passwords greatly reduces errors. It has long annoyed me when I can't see what I type: in Windows logins, in PGP, and so on.

Ignoring the issue of security controls enforced differently in browsers for type password versus type text, the arguments in favor of unmasking fail to address several issues:

  • What class of attackers are completely foiled by password masking? What is the exposure presented by these people? (Think: someone that happens to glance at your screen as they've interrupted you midlogin)
  • What is the additional level of likelihood of detection of someone trying to watch your fingers touchtype vs. someone simply reading off a screen?
  • What is the additional level of risk presented by a persistent display of the password vs. an ephemeral display of each keystroke as it is typed?
  • What is the additional attack surface presented by having the password sent back from a network application, rather than a one-way transmission?
  • How much of the uncommonality of shoulder surfing is due to user awareness of password protection needs, as communicated to them at every login?

All that aside, the correct answer is to reduce our dependency on passwords, both through SSO technologies, and through use of certificate and ephemeral authentication schemes.

March 31, 2007

Embedded bluetooth systems

I have a bluetooth handsfree device, which occasionally gets into an annoying error mode, where my phone thinks the device is connected, and the device disagrees. Modifications to the phone's state - turning bluetooth on or off, powercycling - have no effect. The only apparent cure is to turn the bluetooth device off, and then back on again.

Unfortunately, it's the embedded handsfree system in my car. So I have to turn off the car, and then turn it back on again. My mother has the same problem - this seems to be a defect in the Toyota/Lexus integration. So if you're behind me when I pull out of the garage, and I turn the car off and on at the first red light, you'll know why.

September 20, 2006

Sanitization vs. crypto

Bruce Schneier opines on NIST's proposed non-use of encryption as sanitization:

Encryption is not a generally accepted means of sanitization. The increasing power of computers decreases the time needed to crack cipher text and therefore the inability to recover the encrypted data can not be assured.

I have to admit that this doesn't make any sense to me. If the encryption is done properly, and if the key is properly chosen, then erasing the key -- and all copies -- is equivalent to erasing the files. And if you're using full-disk encryption, then erasing the key is equivalent to sanitizing the drive. For that not to be true means that the encryption program isn't secure.

While NIST has removed the paragraph, functionally, they are correct. Bruce has too many qualifications, and missed some more - if there were no implementation bugs in your crypto, if your key was properly generated, if you didn't escrow the key for DR someplace where it didn't get removed, if there are no known-plaintext or chosen-plaintext attacks against your encryption algorithm within the life of your hard drive... just a few too many ifs.

By all means, encrypt the hard drive - it's a great practice, especially in the event your hard drive is lost. But absolutely sanitize it if you also have the opportunity.

June 30, 2006


Zipcar just showed up in the new parking garage at work. Interesting to note that they've now added the Scions (xA and XB), Element, and Matrix to their line-up.

I assume that means they're seeing demand for more cargo space, which seems to me to be the big gap for people who rely on public transportation.

May 30, 2006

Invisibility cloak

Invisibility gets closer.

It's a cool concept. But once the price comes down, this is one of those potentially disruptive technologies (it reminds me a lot of Shield, by Poul Anderson). I think there are some scary uses, and some cool uses:

In the scary category:

  • concealed guns
  • concealed bombs
  • traffic hazards. People drop bricks off overpasses - what about a cloaked piece of furniture?

Fortunately, the cloaking on each of these would be far and away the most expensive item, so I doubt we'll see them anytime soon.

In the cool category:

  • Urban renewal - what if, instead of creating a hole in space that one looked through, you offshifted light, so that light appeared to go over the object? Imagine a downtown parking garage, with landscaping on the top. From the side, it appeared to be - just a park. Because just making the garage invisibile is ugly - you end up with strange sightlines and perspectives. But making it actually disappear? We could have saved a lot of money on the Big Dig.

  • Architectural features - imagine a building where every other floor is invisible. Or where the pillars aren't there.

  • Privacy umbrella - have your own portable changing station at the beach! Of course, I could see some uses for this that might best fit in the scary category, come to think of it....

May 25, 2006

Infosec - Failing, or Succeeding?

Noam Eppel at Vivica asserts that Information Security is a total failure:

Today we have forth and fifth generation firewalls, behavior-based anti-malware software, host and network intrusion detection systems, intrusion prevention system, one-time password tokens, automatic vulnerability scanners, personal firewalls, etc., all working to keep us secure. Is this keeping us secure? According to USA Today, 2005 was the worst year ever for security breaches of computer systems. The US Treasury Department's Office of Technical Assistance estimates cybercrime proceeds in 2004 were $105 billion, greater than those of illegal drug sales. According to the recently released 2005 FBI/CSI Computer Crime and Security Survey, nearly nine out of 10 U.S. businesses suffered from a computer virus, spyware or other online attack in 2004 or 2005 despite widespread use of security software. According to the FBI, every day 27,000 have their identities stolen.

Noam's article is a good read if you think the Internet is safe. But a lot of folks disagree with his conclusion, and I side with them. Noam's article is just a litany of all the doom and gloom statistics out there - like the ones you see on Slide 2 of every security vendor's pitch.

May 22, 2006

False positives

Driving in to work this morning, I discovered a wonderful failure mode of an alerting system. My car has a weight sensor in the passenger seat; if it detects a possible passenger in the seat, without a safety belt in use, it alerts you.

Now, our other car has had this, and it's just a little red light on the dash. But this car starts an audible dinging alarm, which then goes to a very fast audible alarm. I'm not sure if it will turn itself off, as I moved my backpack quickly from the passenger seat to the floor. But what was my first thought?

Man, I've got to disable that alarm.

Continue reading "False positives" »

May 21, 2006

Auto-turning headlights

We just bought a new car, and it has headlights that turn to the left or the right when the steering wheel has turned in that direction. It's a pretty neat feature, although I discovered an interesting "attack" you can do with it, even unintentionally. The distance the steering wheel needs to turn to trigger the auto-adjust happens to be the curvature of the onramp from Storrow Drive to 93 North. So if you happen to be behind someone, and the steering wheel has any jitter in it (not that, you know, you'd jitter it intentionally), your headlamps will wash back and forth across the inside of their car.

As if being followed by a tall car wasn't already painful enough.