- Types
- Overview of Metrics Program
- Purpose, Approach, and Objectives
- Requirements
- Benefits of Using Metrics
- Metrics Types
- Data Management Concerns
- Stakeholder Interest Identification
- Goals and Objectives Definition
- Security Policies, Guidance, and Procedures Review
- System Security Program Implementation Review
- Metrics Development and Selection
- Establishing Performance Targets
- Feedback within Metrics Development Process
- Metrics Program Implementation
- Summary
- Appendix
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Metrics Program Implementation
Implementation of security metrics involves using security metrics for monitoring security control performance. It also involves using the results of the monitoring to start performance improvement actions. The implementation process is iterative, and consists of six phases. These phases, when fully executed, will ensure continuous use of security metrics for performance monitoring and improvement of security controls. Figure 7 depicts the security metrics program implementation process.
){kind=small}
Figure 7 Implementation Process
Phase 1: Prepare for Data Collection
Phase 1 of the process involves activities that are crucial for establishing a comprehensive security metrics program. These include the security metrics identification, definition, development, and selection activities and development of a metrics program implementation plan.
After the metrics have been identified, specific implementation steps should be defined on how to collect, analyze, and report the metrics. These steps should be documented in the Metrics Program Implementation Plan.
The following items may be included in the plan:
- Metrics roles and responsibilities, including responsibilities for data collection, analysis, and reporting
- Audience for the plan
- Process of metrics collection, analysis, and reporting, tailored to the specific organizational structure, processes, policies, and procedures
- Creation or selection of data collection and tracking tools
- Modifications of data collection and tracking tools
- Metrics summary reporting formats
Phase 2: Collect Data and Analyze Results
Phase 2 of the process involves activities essential for ensuring the collected metrics help gain an understanding of security program security and identify appropriate improvements.
This phase includes the following activities:
- Collect metrics data, according to the processes defined in the Metrics Program Implementation Plan.
- Consolidate collected data and store it in a format conducive to data analysis and reporting, for example, in a database or a spreadsheet.
- Conduct gap analysis: compare collected measurements with targets, if defined, and identify gaps between actual and desired performance.
- Identify causes of poor performance.
- Identify areas requiring improvement.
The causes of poor performance can often be identified using the data from multiple metrics. To determine the cause of low compliance, information must be collected on the reasons for the low percentages (e.g., lack of guidance, insufficient expertise, or conflicting priorities). This information can be collected as separate metrics or as implementation evidence. Once this information is collected and compiled, corrective actions can be taken.
The following are examples of causation factors that contribute to poor security control implementation and effectiveness:
- Resources: Insufficient human, monetary, or other resources
- Training: Lack of appropriate training for the personnel installing, administering, maintaining, or using the systems
- Configuration Management Practices: New or upgraded systems that are not configured with required security settings and patches
- Awareness and Commitment: Lack of management awareness or commitment to security
- Policies, Standards and Procedures: Lack of policies, standards and procedures that are required to ensure existence, use, and audit of required security functions
- Architectures: Poor system and security architectures that make systems vulnerable
- Inefficient processes: Inefficient planning processes that influence the metrics
Phase 3: Identify Corrective Actions
Phase 3 of the process involves the development of the roadmap of how to close the implementation gap identified in Phase 2.
This phase includes the following activities:
- Determine range of corrective actions: Based on the results and causation factors, identify corrective actions that could be applied to each performance issue. Corrective actions may include: changing system configurations; training security staff, system administrator staff, or regular users; purchasing security tools; changing system architecture; establishing new processes and procedures; and updating security policies.
- Prioritize corrective actions based on overall risk mitigation goals: There may be several corrective actions, applicable to a single performance issue. However, some may be inappropriate if they are too costly or inconsistent with the magnitude of the problem.
- Select most appropriate corrective actions: Up to three corrective actions from the top of the list of prioritized corrective actions should be selected for conducting a full cost-benefit analysis.
Phase 4: Develop Business Case
Phases 4 and 5 both address the budgeting cycle for obtaining resources required for implementing remediation identified in Phase 3. The results of the prior three phases will be included in the business case as supporting evidence.
The following activities should be performed as a part of business case analysis:
- Document mission and objectives identified during Phase 2 of the metrics development process.
- Determine the cost of maintaining status quo, to use as the baseline for comparing investment alternatives.
- Document gaps between target performance and current measurements, identified during Phase 2.
- Estimate lifecycle cost for each corrective action or investment alternative, identified in Phase 3.
- Perform sensitivity analysis to discern which variables have the greatest effect on cost.
- Characterize benefits that are quantifiable and non-quantifiable returns which are delivered through improved performance, based on the prioritization of corrective actions performed in Phase 3.
- Perform risk analysis to take into account the likelihood of obstacles and programmatic risks of a particular alternative.
- Prepare budget submission by summarizing key aspects of the business case to accurately depict its merits.
In most cases the previous list of activities to create a business case is too time-consuming and overwhelming. It will be up to an organization to determine the requirements and process for this type of business case development. The previous list should be used as a guideline.
Phase 5: Obtain Resources
Phase 5 involves the following activities:
- Responding to budget evaluation inquiries
- Receiving allocated budget
- Prioritizing available resources, assuming that not all requested resources will be allocated
- Assigning resources to perform corrective actions
Phase 6: Apply Corrective Actions
Phase 6 of the process involves implementing corrective actions in technical, management, and operational areas of security controls. After corrective actions are applied, the cycle completes itself and restarts with a subsequent data collection and analysis.
Iterative data collection, analysis, and reporting will track progress of corrective actions, measure improvement, and identify areas for further improvement.
The iterative nature of the cycle ensures that progress is monitored, and that the corrective actions are affecting system security control implementation in the intended way. Frequent performance measurements will ensure that, if corrective actions are not implemented as planned, or if their effect is not as desired, quick course corrections can be made internal to the organization.