Performance monitoring and management involves measuring operational activities, analyzing the resulting metrics, and comparing them to internally established standards and industry benchmarks to assess the effectiveness and efficiency of existing operations. Measurable performance factors include resource usage, operations problems, capacity, response time, and personnel activity. Management should also review metrics that assess business unit and external customer satisfaction. Diminished system or personnel performance not only affects customer satisfaction, but can also result in noncompliance with contractual SLAs that could result in monetary penalties. Refer to the IT Handbook's "Outsourcing Technology Services Booklet" for more detailed information.
If economically practicable, management should automate monitoring and reporting processes. Large mainframe systems have numerous automated tools available at the application and operating system level for generating technology and process-related metrics. Mid-range systems also typically possess native capability for capturing and reporting technology. There are also after-market reporting tools and vendor-supplied performance analysis tools available for mid-range systems. Client-server systems are not always equipped with analysis and reporting tools. Often management should decide between purchasing expensive after-market reporting tools to automate the data gathering and reporting or generating the reports manually.
Much of IT operations can and should be subject to measurement based on the size and complexity of the institution. The information gained from analysis supports not only daily management of operations and early diagnostics on impending problems, but provides the baseline and trend data used in capacity planning.
Examples of technology related metrics include:
- Central processing unit (CPU) utilization by application or time of day;
- Network availability;
- On-line performance measurements including availability, response time, distribution of access types by service, or average connect time;
- VRU performance (e.g. calls answered, average talk time, average wait time, call distribution by time of day, distribution of information access requests); and
- Abnormal program endings.
Examples of operations performance metrics include the following:
- Check processing - statement processing:
- Percent of statements mailed by internal guideline;
- Percent of exception statements mailed by internal guideline; and
- Percent and volume of mismatched debit and credit items.
- Item processing - proof of deposit:
- Average fields encoded per hour;
- Ratio of errors per number of items encoded; and
- Percentage of overall rejects for items captured.
- Total debit and credit transactions for the month;
- Percent of unposted items resolved the same day received; and
- Percent of unposted items versus total posted debits and credits.
- Imaging operations:
- Volume and percentage by document type (e.g. new account documents, loan documents, item processing) scanned and processed within internal guidelines;
- Volume of transactions; and
- Number of errors reported and percent to total maintenance volume.
- Electronic funds transfer and electronic banking:
- Number of wire processing errors caused by department and percent of total volume;
- Number of wires not processed due to failure to execute; and
- Number of incidents reported and compensation paid due to department processing errors.
- Technology services - IT help desk:
- Volume of calls received;
- Percent of calls dropped compared to internal guidelines;
- Average incoming call wait time compared to internal guidelines; and
- Average duration of incoming calls.
- Human resource:
- Actual versus budgeted staff size;
- Department or group staffing compared with averages for the two previous years and budgeted headcount; and
- Percent of staff with required training or certification.
Risk Monitoring and Reporting