Making sense of Risk Management Security Metrics In the IT Security Field, Part IV
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
The development or selection of currently developed metrics and their implementation can be extremely helpful or end up in a mess. Just like everything else in security, if we don’t create an architecture to work from that makes sense, then our efforts may be made with the best intentions and the worst results.
Defining useful metrics is not for the weak at heart. It can be time consuming and tedious. It requires a structured approach to security objectives and a discipline to follow through. Metrics is not the sexiest part of security, but it is one of the most important if we really want to understand where we are, where we need to go, and how we will get there.
Many times people create metrics and formulas, and then when it gets time for implementation, they realize they have slightly—or totally—missed the mark in the data that really needs to be collected. We need to figure out what we want to measure and why before churning out formulas and questionnaires.
The main goal of using metrics is to help us better understand the environment, so that we can make more effective decisions. Metrics should be used to provide relevant performance trends and indicate improvement actions that should take place. They should allow us to prioritize the items that need to be addressed and provide substantial evidence as to why those items need allocated funds and resources. Metrics can be very useful to support business cases that management uses to justify the investments necessary to secure the environment.
While the collected data at some point may be illustrated in charts and presentations in a more general or abstract way (red green/yellow or high/medium/low), it is important that real numbers support these illustration. We have to move from a gut feeling of how vulnerable a system might be to quantifiable data (integer values, percentages, averages).
In this series I will focus on metrics that indicate the level of compliancy with an organization’s policies and standards, and the associated impact values that correlate with non-compliance. I am not going to lay out metrics that helps ensure specific regulatory requirements (PCI, HIPAA, GLBA, etc.), but help indicate the level of security that is truly being practiced. We all know that there is commonly a difference between being compliant with some regulation and being secure—they are not always the same. While these metrics can help you ensure that you are compliant with some regulations, they have a focus on true security implementation. The metrics will be system-level and program-level values. We ultimately need to understand the overall health of the security program, but also have to have the capability to drill down into specific aspects of the program.
Two critical components must be in place before real metrics can be created: relevant policies and standards. These will help us determine our performance goals and help us use the same points of reference. Many companies have some policies and standards that were created from boiler templates or someone pulling policies off the Internet and changing out the company name. Many technology-oriented people have a visceral reaction to documentation, and others find policies and standards boring and maybe even useless. We need to create a symbiotic relationship between policies, standards, and metrics. Metrics should indicate our compliance with our policies and standards, but also the metrics should be used to improve our policies and standards. Feedback loops should be constructed so that continuous improvement can be facilitated.
Now I don’t walk into every one of my customers’ environments and start demanding that everything should get measured. The maturity of the security program will help indicate what type of metric data can be gathered and how much weight can be assigned to the specific data sets. Metrics must have evidence data, and the processes that create this type of data must be repeatable. A mature security program will have detailed and well-documented policies and standards, and its processes will be standardized and repeatable. In this situation, organizations will be able to gather not only a larger quantity of data, but also data of better quality. An immature security program does not usually have well-defined ways of gathering data, and their processes are more ad hoc in nature. Many times I find that immature security programs must be stabilized and predictable before measurements of activities can be introduced.
An organization with a mature security program commonly has ways of gathering data through automation versus manual data collection. Manual data collection requires surveys and questionnaires to be created and interviews to be conducted. As the program matures, there are more data inputs that can come from automated tools, which allow for not only more data to be gathered but much of the subjectivity to be subtracted.
When I first start working with a customer, I find out if we can even identify the controls that are or are not in place. Sadly, I find that many organizations do not even know what they have in servers, workstations, or products, so we have to back way up and get asset management setup and matured. Once an organization knows what it has (devices, software, and applications), it can move forward to understand and document the controls that are in place. Once the controls are understood, performance metrics can be set for them. Once performance metrics are set, data gathering techniques can be defined and implemented. The metrics will mature with the security program. The initial metrics need to track what is and is not in place, and then they will mature so that the effectiveness and efficiency of the controls can be understood. You cannot jump into trying to figure out how well your controls are working if you don’t know what controls you have in place.
Controls are obviously not just technical. We also have administrative controls (policies, employee management, training, etc.) and physical controls (security guards, fences, locked doors, etc.). Metrics will need to be defined for these also and they run into the same maturity issues as technical controls. For example, you cannot track the effectiveness of employee training unless it is in place.
So when starting to define metrics for your environment, make sure you are realistic and are gathering the most relevant types of data at the right time. A mature environment might be able to provide correlated data feeds from a security information and event management (SIEM) product. This type of environment might also have continuous monitoring taking place, which will allow you to collect nearly real time incident and response metrics. In this situation configuration, vulnerability and asset management are usually understood and automated. As we will see, the metrics that can be defined in these environment types are much more sophisticated and detailed, compared to environments that do not have automated and centralized processes.
So how do you know the maturity of your security program? You can use a Capability Maturity Model (CMM) such as the one below to understand where the program is today and the incremental steps necessary to move forward. If your program is a Level 1 or 2 of a CMM, your metrics should be basic in nature. As your program matures, the data that support the metrics will improve, which means that your metrics can mature.
)
What is the danger of creating the wrong metrics for your environment? Gathering data is time consuming, so you can waste resources. The metrics might only show part of the story instead of providing a holistic view of the environment. But here is what I find most common and disturbing: incorrect data that provides a false sense of security. All organizations have lovely pie charts and colorful graphs that indicate various risks and their severity levels, but what data is supporting them? Peek under the covers and many times the data that was gathered to generate the graphs are sparse and far too often just does not make any sense.
Stay tuned for Part V, where we will dive into the types of metrics we need to develop and why.