June 16

On Reports (a perspective)…

By Adam Dodge

Lately, there has been a flurry of activity in the land of security breach reports with organizations such as Debix, Verizon, the Identity Theft Resource Center and the Department of Justice all releasing reports looking at security breaches, breach notification laws and the state of information security in general. As someone who has been in the world of tracking and monitoring breaches for two years now through Educational Security Incidents, I am excited over the increased attention and information that is coming forth and the lessons that can be learned from these breaches. However, it is important to remember that are inherent limitations on the applicability of breach statistics and therefore we all must be cautious about reading too deeply and arriving at conclusions that the information in these reports do not support.

Before we go any further, yes I do develop a similar report each year and yes my report is subject to the same limitations as all of these other reports. My point here is not that all other reports are wrong while the ESI YiR is the shining beacon of truth. The point is that the information delivered in these reports is simply that, information. It is up to the reader to interpret this information in a meaningful way. The problem, then, stems from misinterpretation and this

What do I mean by “misinterpretation”? Well a common problem with the statistics provided in these reports (remember, I’m including my own report as well) is that the numbers are based the sample set and the ability to apply these numbers depends a great deal upon the size of the sample and how randomly the sample was chosen from the total population. Alright, that might not be a good enough answer so allow me to explain further.

The Verizon report has made a big splash in the security world and for good reason. Verizon did an amazing job with this report. If you haven’t read it, go do so now. Seriously, stop reading this and go read the report. It is that good.

However, the report is based around 500 forensic investigations performed by Verzion’s Business RISK team between 2004 and 2007. These 500+ breaches that Verizon has analyzed for this report were not randomly chosen from all breaches that occurred. Instead, the information was mined from the investigations stemming from breaches that were serious enough for a company to reach out and contract with Verizon for assistance. This is a potential point of bias for this survey.

Most companies are not going spend money on investigations for small breaches or those that are easily explainable. Therefore, it is very likely that breaches of data such as information left in public, information accidently placed on a public web site, etc. are underrepresented in the sample Verizon used. It is also likely that smaller companies and non-profit organizations are underrepresented as well since these entities lack the funding that larger, for-profit organizations have at their disposal.

What does this sample bias mean for the validity of the Verizon report? Nothing. Nothing at all. There is no problem with the sample bias of the Verizon report. The simple fact is that all of security breach reports (again, including the ESI YiR) suffer from the same problem. Unfortunately, there is no go way around this problem yet. Everyone that I talk to involved with tracking breaches has the same complaint: There is no centralized reporting of breaches in the United States and those states that do require breach reporting to a central authority have different reporting requirements, litmus tests and public access to breach information.

So I am suggesting that everyone stop reading these reports? Absolutely not. It is not just self-preservation that makes me say this, however much I enjoy my work with ESI. These reports are an excellent way for information security practitioners to track the movement of threats and discover what types of security threats similar organizations are facing. The point of all of these is that each and every one of us (including the media) need to make sure that we are interpreting the data of these reports properly before we remove our firewall because the 2007 ESI YiR said that employee mistakes outnumber hackers as the cause of a breach 2:1 or before we discontinue our security awareness and training programs because the Verizon reports says that 73% of all breaches came from external sources.

How can these reports be so different and yet both be correct? Simple, look to the samples used to compile them.



You may also like

Are you using frameworks properly?

Leadership and communication are actually layers, not levels

  1. Adam:
    The Verizon study spotlights an important topic for debate. Legally speaking, what is “reasonable security?” FTC punished TJX for not having it, but I argue FTC was wrong. Verizon says 9 of 10 data breaches could have been avoided if “reasonable security” were present. That implies 9 in 10 breach victims were in violation of law. The study’s outlook is that the solution to identity theft is locking down corporate data. But a security consultant/solution provider like this Verizon unit naturally sets a high bar for what is reasonable. And when Verizon evaluates if reasonable security could have prevented a break-in, it does so with benefit of hindsight. Yet the study goes on to say that in modern systems knowing where all your data reside is “an extremely complex challenge.” In other words, the sheer problem of locating data (so you can apply security) is very expensive, and mistakes by data-holders who act in good faith are easy. The reasonable measures expected by FTC and Verizon are extravagantly hard to implement in practice. Hence, the portion of incidents preventable by FTC/Verizon’s reasonable procedures is much lower than 90%. We need to focus more attention on other solutions to identity theft. What do you think? –Ben http://hack-igations.blogspot.com/2008/03/ftc-treats-tjx-unfairly.html

Comments are closed.
{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}

Subscribe to our newsletter now!