6.10 Metrics

By maintaining historical information from Incident control and Problem/error control, it is possible to derive measures that are indicative of the quality of service and the performance of the process.

6.10.1 Problem/error control reporting

Management information on this subject includes:

• the number of RFCs raised and the impact of those RFCs on the availability and reliability of the services covered
• the amount of time worked on investigations and diagnoses per organisational unit or supplier, split by Problem types
• the number and impact of Incidents occurring before the root Problem is closed or a Known Error is confirmed
• the ratio of immediate (reactive) support effort to planned support effort in Problem Management
• the plans for resolution of open Problems with regard to resources:
  • people
  • other used resources
  • costs (against budget)
• a short description of actions to be undertaken.

Information about weak components in the IT Infrastructure and breaches of agreed service levels with the business and by suppliers are of concern to Availability Management. The frequency and duration of Problems is a measurement of performance against agreed service levels. Information required will include:

• the number of Problems and errors split by:
  • status
  • service
  • impact
  • category
  • User group
• the total elapsed time on closed Problems
• the elapsed time to date on outstanding Problems
• the mean and maximum elapsed time to close Problems or confirm a Known Error, from the time of raising the Problem record, by impact code and by support group (including vendors)
• any temporary resolution actions
• the expected resolution time for outstanding Problems
• the total elapsed time for closed Problems.

6.10.2 Periodic audits

Process control requires periodic audits of all operations and procedures. These audits are intended to confirm that the Problem Management and support teams are adhering to defined procedures. The audits should analyse major problem reviews, and check:

• that reports are produced and analysed according to the agreed schedule
• a representative sample of Incidents, to verify that related Problems have been correctly identified and recorded
• a representative sample of Problems, to verify that Problems are diagnosed correctly and diagnosed within the prescribed period
• a representative sample of Known Errors, to verify that Known Errors are cleared by authorised Changes to CIs and within the prescribed period
• that thresholds for escalation have been adhered to
• a representative sample of records, for correctness and completeness
• that documentation is being maintained correctly - updates being distributed by Problem Management staff and executed by recipients
• that management reports are produced regularly and are meaningful
• for evidence of trend analyses and the identification of preventive actions
• staff training records.

6.10.3 Tips on metrics

The following are important points to bear in mind with regard to metrics:

• Define, during the specification stage, target effectiveness metrics for the process. Plan to use them as a control when Problem Management operations commence in order to assess effectiveness and evaluate trends objectively. Set actual targets in advance, if possible, or during the first few weeks of operation. If possible, obtain statistics from current support activities as a basis for the targets. These will be useful later in identifying the benefits accrued from the introduction of Problem Management.
• Do not set targets that cannot be measured.
• When purchasing support tools, take relevant metrics into account during the product evaluation and system-design stages and try to obtain tools that can provide the necessary statistics.
• Develop procedures to produce reports on key effectiveness criteria. Problem Management should review actual achievements against all targets on a regular basis, once a month. Record the results of each review and keep for audit purposes. If targets have to been relaxed, make sure the cause is overambitious targeting and not poor operations. Any deficiencies in the operation of Problem Management should be traced back and put right at source.
• Plan to establish formal effectiveness and efficiency reviews of the process, paying particular attention to the requirements of Customers. Ensure support requirements are fully satisfied by the process when it is in operation.
• Plan to set progressively more difficult targets, in line with the benefits planned from the introduction of Problem Management processes.
• Monitor the effect of a successful Problem Management system on the workloads of staff. Experience suggests that an effective process, particularly when working alongside effective Change Management and Release quality control, quickly reduces the incidence of Incidents and Problems. This reduced number of Incidents and a consequential reduction in the number of support requests can result in a reduction in staff numbers. New applications and new Users can increase Problem and Incident workloads. Problem Management should, therefore, be informed of any anticipated Changes that may affect these workloads, so that plans can be made, not only in relation to staffing levels and staff accommodation, but also to other resources such as database and other IT capacity.
• Sometimes an audit carried out in another Service Management process may uncover something that could indicate a Problem. Ensure that the relevant details are passed to Problem Management for any necessary corrective action.