- Types
- Overview of Metrics Program
- Purpose, Approach, and Objectives
- Requirements
- Benefits of Using Metrics
- Metrics Types
- Data Management Concerns
- Stakeholder Interest Identification
- Goals and Objectives Definition
- Security Policies, Guidance, and Procedures Review
- System Security Program Implementation Review
- Metrics Development and Selection
- Establishing Performance Targets
- Feedback within Metrics Development Process
- Metrics Program Implementation
- Summary
- Appendix
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Metrics Types
The maturity of an organization’s security program determines the type of metrics that can be gathered successfully as shown the figure below.
)
Figure 3 Security Program Maturity Levels
A security program’s maturity is defined by the existence and institutionalization of its processes and procedures. As a security program matures, its policies become more detailed and better documented, the processes that it uses become more standardized and institutionalized, and it produces data that can be used for performance measurement in greater quantity and enhanced quality.
The security program progresses from: having policies (Level 1); to having detailed procedures (Level 2); to implementing these procedures (Level 3); to testing compliance with, and effectiveness of, the procedures (Level 4); and, finally, fully integrating policies and procedures into daily operations (Level 5).
A mature program normally deploys multiple tracking mechanisms to document and quantify various aspects of program performance. As more data becomes available, the difficulty of measurement decreases, and the ability to automate data collection increases. Manual data collection involves developing questionnaires and conducting interviews and surveys with the organization’s personnel. Automated data collection depends on the availability of data from automated sources, as opposed to data available from personnel.
Typically, more useful data will become available as a security program matures from semi-automated and automated data sources, such as self-assessment tools, security event management, incident reporting, and response databases. Metrics data collection is fully automated when all data is gathered by using automated data sources without human involvement or intervention.
The types of metrics—implementation, efficiency and effectiveness, and impact—that can realistically be obtained and that is useful for performance improvement depend upon the maturity of the implementation of the security controls. Although different types of metrics can be used simultaneously, the primary focus of metrics shifts as the implementation of security controls matures. When security controls have been defined in procedures and are being implemented, the primary focus will be on the level of implementation. When a system progresses through Level 1 and Level 2, the results of these metrics will be less than 100 percent, and the system will have not yet reached Level 3. When the results reach and remain at 100 percent, the system has fully implemented security controls and has reached Level 3.
As security controls are documented and implemented, the ability to reliably collect the outcome of their implementation improves. As an organization’s security program evolves and performance data becomes more readily available, metrics will focus on program efficiency—the timeliness of security service delivery and effectiveness—and the operational results of security control implementation. Once security is integrated into an organization’s processes, the processes become self-regenerating, measurement data collection becomes fully automated, and data correlation analysis can determine the mission or business impact of security-related actions and events.
The metrics at Level 4 and Level 5 concentrate on measuring effectiveness and efficiency of implemented security controls, as well as the impact of these controls on an organization’s mission. These metrics focus on the evidence and results of testing and integration. Instead of measuring the percentage of approved security policies and procedures, these metrics concentrate on validating whether security controls, described in the security policies and procedures, are effective in protecting an organization’s sensitive data. For example, computing the percentage of crackable passwords within a predefined time threshold will validate the effectiveness of an organization’s password policy, by measuring the length of time required to break policy-compliant passwords. The impact metrics would quantify incidents by type (e.g., root compromise, password compromise, malicious code, denial of service).
It is not feasible to develop Level 4 and 5 metrics for security program if it is really at a Level 1 or 2 in its maturity progress. As the security program matures, these higher level metrics that deal with effectiveness and efficiency can be developed and used.