Appendix H: Testing Program - Governance and Attributes
Board of Directors and Senior Management
The board and senior management should establish a testing program appropriate for the size, complexity, and risk profile of the organization and its business lines. They should ensure that the testing program demonstrates the institution's ability to meet its requirements for continuity of operations. The board and senior management should establish clear lines of authority and responsibility for all parties involved with developing, implementing, and monitoring the continuity testing program. They should also review and approve the continuity testing program at least annually, ensure that appropriate follow-up on test results is performed, and review test results.
Institutions may employ various approaches for ensuring coordinated and consistent testing across the organization and support for various quality assurance activities, including consistent standards for testing and reporting. For example, many institutions have created a business continuity oversight function, under the direction of a senior manager, with accountability and authority for business continuity planning and testing across the organization. The business continuity function is supported by a team of liaisons assigned from within the business lines and support functions. Some institutions rely on a steering committee, comprised of representatives from business and support functions, to ensure a coordinated and consistent approach to business continuity planning and testing. Regardless of the approach taken, it is the responsibility of the board and senior management to ensure that sufficient resources and qualified staff are allocated to the business continuity testing effort.
Business Line management
Business line management should have ownership and accountability for testing continuity of business operations, including applications and processes. While business line management has overall responsibility for testing their business processes and related interdependencies, they should coordinate with the enterprise-wide business continuity plan (BCP) testing function and support areas, such as IT and facilities management. Ultimately, business line management should ensure that its BCP is continually updated based on test results and changes in business processes.
The IT function should have ownership and accountability for testing recovery of the institution's systems, IT infrastructure, telecommunications, and the infrastructure of alternative computing facilities. Moreover, the IT function has custodial responsibility for business line data and applications. IT should coordinate with business line management and staff to establish test environments suitable for business line testing and should continue to coordinate throughout the testing process. Additionally, the IT function should, through effective management of the test schedule, provide sufficient opportunities for the various business functions to test the operational consistency of primary and alternate computing facilities. The IT group is responsible for maintaining the technology test environment, including controls such as change and configuration management and information security.
The board and senior management should ensure that the business continuity testing program includes the institution's crisis management capabilities. The testing program should include exercises to demonstrate that the crisis management program effectively meets the institution's objectives for responding to a crisis situation, including identifying and declaring emergencies, providing a central point for the management of an event, and coordinating internal and external communications and human resource issues.
The facilities management function should have ownership and accountability for testing the recovery of the institution's physical plant and equipment, environmental controls, and physical security. Environmental controls for data centers and the facilities that house critical business functions should be included in the institution's continuity testing program. When data centers or business functions are housed in vendor facilities, contracts should specify the requirements of the vendor for testing continuity of those facilities.
The internal audit department, or another qualified independent party, plays an important role in providing an independent review of the adequacy of the overall business continuity testing program. The depth and frequency of audit activities and reporting should be scaled to the criticality of the operation. While the scope of audit activities and deliverables may vary, in all cases they must encompass an independent and objective evaluation of the effectiveness of the testing program.
As part of the review of the testing process, internal audit should determine the reasonableness of the underlying assumptions that were made in developing the test program. The reasonableness of underlying assumptions, as well as the adequacy of test plans, scenarios, schedules, and reports, should be evaluated relative to (1) the size and complexity of the institution, (2) the criticality of the business line, and (3) the risk and impact of a possible business disruption. Audit should observe test exercises to assess the control environment of alternative locations, verify the results, ensure that proper reporting and escalation mechanisms are established and utilized, and ensure that test plans are updated to reflect prior test results.
Enterprise-wide testing strategies should be developed to properly validate the BCP. Management will achieve greater confidence in their testing strategies when consideration is given to the following elements and complexity issues:
The test strategy should encompass at least three elements: staffing, technology (data, systems, applications, and telecommunications), and the facilities that house the staff and technology environments.
- Staffing-Testing strategies should include demonstrations of the staff's ability to support business processes, including the processing and settlement of transactions, communication with key internal and external stakeholders, and reconciliation of transactions and books of record. Strategies may need to address the ability of staff to support increased workloads resulting from the transfer of processing to alternate sites for extended periods of time. For institutions that have implemented split processing business models, any aspects of the client relationship model that present challenges or complexities to the transfer of workloads across sites, and related dependencies, should be identified and incorporated into testing strategies. In addition, testing strategies should demonstrate the effectiveness of the institution's management succession plans.
- Technology-Testing strategies for technology should include the data, systems, applications, network, and telecommunications necessary for supporting business activities. In the event system recovery is dependent upon the retrieval of data files, programs, and other items maintained at the back-up facility; off-site testing procedures should only include the use of these back-up items to properly replicate the loss of any master data files and programs maintained at the main facility. Back-up data files should also be tested frequently to assess the integrity of the information, to determine if the data is being saved in the correct format, and to ensure that applicable files can be retrieved in a timely manner. Alternatively, institutions may employ other processes for data replication, such as synchronous and asynchronous data replication. Regardless of the data replication process used, the process for demonstrating consistency of data across different processing environments should be included in the testing strategy. In addition, strategies should test processes to recreate any data lost during a switch to alternate processing facilities, and periodic reviews of telecommunications services should be conducted to determine circuit diversity.
- Facilities-Testing strategies for business functions should encompass environmental controls, workspace recovery, and physical security to ensure continuity of facilities and environmental systems at primary and alternate processing sites. Testing strategies should include the adequacy of back-up power generators and heating, ventilation, and air conditioning systems to meet business recovery objectives at operating centers. Workspace recovery test strategies should include assessments of the availability and adequacy of workspace, desktop computers, network connectivity, e-mail access, telephone service, and physical security controls. For institutions relying on the physical relocation of hardware, software, or data storage devices to recover the technology infrastructure and applications at alternate locations, the facilities testing strategy should address the secure transportation of these items.
Organizations should develop testing strategies that demonstrate their ability to support connectivity, functionality, volume, and capacity using alternate facilities. The testing strategies should encompass internal and external dependencies, including activities outsourced to domestic and offshore business and technology service providers. For critical business functions, test strategies and plans may need to extend beyond network connectivity and include transaction processing to assess capacity and data integrity.
Crises management Test Plans
Test scenarios, plans, and objectives should include the institution's crisis management function to demonstrate the institution's ability to respond effectively to contingency events. The crisis management program should be tested, with particular emphasis on the institution's capability to gather information about the threat or event, initiate the BCP, and communicate relevant information to the appropriate staff, customers, vendors, service providers, regulators and other public authorities. Crisis management test plans should address the ability of crisis management team members, and their alternates, to carry out their designated responsibilities under various event scenarios.
Test scripts provide sequential procedures related to testing specific business or technology functions. Test scripts can be readily used by employees to test business processes within pre-established timeframes, and test scripts should include references to production documentation and procedures. Each test script should clearly document the test objective and procedures, including:
- Detailed information regarding the application, business processes, system, or facility to be tested;
- Sequential test steps to be performed by employees or external parties;
- Prompts for test participants to record quantifiable test metrics;
- Procedures to be followed for manual work-around processes, if applicable;
- A detailed schedule for completion of the test;
- Prompts for participants to record issues encountered with the continuity plan during the test; and
- Prompts for participants to record suggestions for improving continuity plans and associated test methods.
Test scripts may include steps for rotating staff involved in specific tests to simulate the inaccessibility of key employees during a disaster.
TESTING EXPECTATIONS FOR CORE CLEARING AND SETTLEMENT ORGANIZATIONS AND FIRMS THAT PLAY SIGNIFICANT ROLES IN CRITICAL MARKETS
The guidance provided in this section describes additional expectations regarding business continuity testing for those organizations that perform core clearing and settlement activities in critical financial markets (core firms) and those organizations that process a significant share of transactions in critical financial markets (significant firms). These organizations have been advised by their regulators that they have met the definition of a core or significant firm as set forth in the "Interagency Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System" (Sound Practices Paper).
Core and significant firms that are subject to the Sound Practices Paper should develop verification strategies and execute testing activities to validate the implementation of the interagency guidelines. The following discussion is not meant to limit the testing strategies or activities of core and significant firms and should be read in conjunction with more comprehensive guidance, available in the public and private sectors, to evaluate the scope and test the effectiveness of business continuity plans.
In general, core and significant firms should have a comprehensive, risk-based approach for testing and evaluating the effectiveness of all of its internal business continuity arrangements. It would be appropriate to include documented strategies and plans to determine whether the core or significant firm has established the facilities and arrangements necessary to assure substantial achievement of the recovery objectives and other expectations set forth in the Sound Practices Paper. In this regard, the Sound Practices Paper advises core and significant firms to routinely use or test their individual internal recovery and resumption arrangements for connectivity, functionality, and volume capacity. It is also suggested that significant firms, which have back-up sites within the current perimeter of synchronous back-up technology or that rely primarily on employees from the same workforce as the primary site, confirm that their plans would be effective if a wide-scale disruption affects both sites.
Moreover, in light of the dependencies between core firms and significant firms and the potential impact that a prolonged disruption of clearance and settlement activities would have on the operation of the financial system, verification strategies should include an external component. This external component should help the agencies and core and significant firms assess whether there is a consistent level of resilience across critical financial markets and whether their recovery arrangements are compatible.
Because of their critical role in the operation of financial markets, the external verification strategies of core firms should include ample opportunities for significant firms to test their recovery of critical clearing and settlement activities from their alternate processing sites. Significant firms are expected to test with the relevant core firms from their alternate sites and meet any testing requirements the core firms establish specifically for significant firms and for participants more generally. Significant firms should take advantage of these opportunities to test their ability to meet the recovery time objectives (RTOs) set forth in the Sound Practices Paper from their geographically dispersed alternate sites. Core firms and significant firms also are encouraged to participate in pertinent market-wide and cross-market tests (such as the "street tests" sponsored by the Securities Industry Association, Bond Market Association, and Futures Industry Association) that test connectivity from alternate sites and include transaction, settlement, and payment processes, to the extent practical. Verification strategies should incorporate lessons learned from prior tests and exercises to improve their effectiveness in validating back-up strategies.
Internal testing activities should confirm that each core and significant firm has identified all clearing and settlement activities, as well as the systems that support or are integrally related to the performance of those activities, for each critical market in which they are core or significant. These activities should also be designed to demonstrate the core and significant firm's ability to complete pending material payments and transactions, access funding, manage material open risk positions, and make related entries to books and records in the event of a wide-scale disruption from alternate geographically dispersed data centers and operations facilities. Moreover, testing activities should confirm that such critical clearing and settlement activities could be recovered within RTOs set forth in the Sound Practices Paper.
As noted earlier, test programs should address external interdependencies, such as connectivity to markets, payment systems, clearing agencies, messaging services, and other critical service providers. Moreover, test programs should validate the effectiveness of internal and external communication protocols with stakeholders. Test scenarios should include a wide-scale disruption in which primary data centers and operations facilities are rendered inoperable for some period without notice, making it necessary to recover critical clearing and settlement activities from an alternate site. Core firms should confirm that resumption of critical clearing and settlement activities can be sustained at alternate sites. Core or significant firms that use the same alternate sites or whose alternate sites rely on the same employees as their primary sites should assume that employees at primary sites are unavailable to clear or settle pending transactions for several days, or are that some employees are unavailable for longer period of time.
Examination and supervisory activities will include evaluations of verification strategies and test plans in order to assess whether core and significant firms, which are subject to the Sound Practices Paper, have achieved the resilience necessary to protect the financial system from a wide-scale disruption. Verification strategies should be incorporated into implementation plans and should have an external as well as internal component. If a core or significant firm finds it necessary to make incremental changes in its recovery strategies, it should modify its verification strategies and test plans to incorporate those changes. Core and significant firms should perform robust testing to assess the effectiveness of their recovery strategies. Verification strategies, test plans and test results should be documented and subject to a qualified, independent review, such as an internal or external audit. The agencies will evaluate a core and significant firm's verification strategies and test plans, the execution of such strategies and plans, and the test results.
Appendix G: Business Continuity Plan Components
Appendix I: Laws, Regulations, and Guidance