Appendix E: Interdependencies
Financial institutions can be very complex, with numerous interdependencies between internal and external systems and processes. Analyzing interdependencies represents a critical step in the business continuity process and is an integral part of a business impact analysis. The analysis of interdependencies allows financial institution management to evaluate the critical resources and services that are shared, identify the potential consequences in the event an interdependent system or process is disrupted, and develop business continuity plans that include mitigating controls and recovery strategies. While each financial institution has a unique business environment and may be dependent on different internal and external systems and processes, this section discusses three common interdependencies, including telecommunications infrastructure; third-party providers, key suppliers, and business partners; and internal systems and business processes. These interdependencies should be considered as part of the business continuity planning process.
Voice and data communications are essential for conducting business and connecting critical elements of an institution such as business areas, customers, and service providers/vendors. Advancements in network technologies allow greater geographic separation between people and system resources or primary and alternate processing locations. Network technologies have played a key role in enabling distributed processing environments, which reflect an increased reliance on telecommunications networks for both voice and data communications. Given their critical nature and importance, it is necessary for institutions to design high levels of redundancy into their voice and data communication infrastructures. In addition, as critical as it is to have effective business continuity arrangements for a data center, it is equally important to have effective back-up arrangements for voice and data telecommunications links. Since voice and data infrastructures are typically a shared resource across the different business areas of an institution, the dependency and importance of these resources are further heightened.
Single Points of Failure
The telecommunications infrastructure contains single points of failure that represent vulnerabilities and risks for financial institutions. Elements of risk reside within the public telecommunications network infrastructure and are outside the control of a single institution. As a result, financial institutions should establish robust processes to ensure that telecommunications are diverse and can be quickly recovered. Institutions should develop risk management practices to identify and eliminate single points of failure across their network infrastructures. Risk management strategies should be incorporated into the design, acquisition, implementation, and maintenance processes related to communication networks and should address single points of failure or points of commonality relating to:
- Primary and back-up network infrastructures;
- Telecommunications carriers;
- Telecommunications routing through central offices;
- Payment, clearing, and settlement processes, such as electronic funds transfer (EFT) and automated teller machine (ATM) services;
- Core processing providers;
- Points of entry into facilities; and
- Private branch exchanges within an institution.
Telecommunications Diversity Guidelines
A financial institution's BCP should address diversity guidelines for its telecommunications capabilities. This is particularly important for the financial services sector that provides critical payment, clearing, and settlement processes; however, diversity guidelines should be considered by all financial institutions and should be commensurate with the institution's size, complexity, and overall risk profile.
Diversity guidelines may include arrangements with multiple telecommunications providers. However, diverse routing may be difficult to achieve since primary telecommunications carriers may have an agreement with the same sub-carriers to provide local access service, and these sub-carriers may also have a contract with the same local access service providers. Financial institutions do not have any control over the number of circuit segments that will be needed, and they typically do not have a business relationship with any of the sub-carriers. Consequently, it is important for financial institutions to understand the relationship between their primary telecommunications carrier and these various sub-carriers and how this complex network connects to their primary and back-up facilities. To determine whether telecommunications providers use the same sub-carrier or local access service provider, management should consider performing an end-to-end trace of all critical or sensitive circuits to search for single points of failure such as a common switch, router, PBX, or central telephone office..
Management should also consider the following telecommunications diversity components to enhance the BCP:
- Alternative media, such as secure wireless systems;
- Internet protocol networking equipment that provides easily configurable re-routing and traffic load balancing capabilities;
- Local service to more than one telecommunications carrier's central office, or diverse physical paths to independent central offices;
- Multiple, geographically diverse cables and separate points of entry;
- Dedicated Synchronous Optical NETwork (SONET) technology using fiber-optic rings over two diverse routes for connections to telecommunications carrier central offices;
- Frame relay circuits that do not require network interconnections, which often causes delays due to concentration points between frame relay providers;
- Separate power sources for equipment with generator and/or uninterrupted power supply back-up;
- Separate connections to back-up locations;
- Regular use of multiple, active facilities in which traffic is continually split between the connections; and
- Separate suppliers for hardware and software infrastructure needs.
Monitoring Telecommunications Providers
In coordination with vendors, management should ensure that risk management strategies include the following, at a minimum:
- Establish service level agreements that address contingency measures and change management for services provided;
- Ensure that primary and back-up telecommunications paths do not share a single point of failure; and
- Establish processes to periodically inventory and validate telecommunications circuits and routing paths through comprehensive testing.
Business Continuity Arrangements
In addition to robust risk management practices, financial institutions should have viable business continuity arrangements for voice and data services. At a minimum, telecommunications plans should address skilled human resources, internal and external connectivity, communications media, network equipment, and telecommunications management systems. The BCP should establish priorities and identify critical network components. Original plan components such as reliability, flexibility, and compatibility must also be considered in formulating the back-up plan. For example, a modem used for back-up may not provide the level of service required, or a line may satisfactorily transmit voice, but be insufficient in quality and speed for data transmission. The costs of various back-up alternatives should be weighed against the level of risk protection provided by the alternatives. This assessment also should address costs associated with testing, since all components of a plan should be tested periodically, including the communications media.
The BCP should address the security and practicality of alternative telecommunications solutions. Switching from fiber optic to wire pairs, dedicated to switched lines, or digital to analog services may make the line more susceptible to a wiretap or to line noise, which could affect data security. Practicality issues should also be addressed, such as selecting alternatives that will accommodate the anticipated volumes at the necessary speeds to meet the established priorities. For example, several dial-up lines may not be a practical replacement for a T-1 line. Also, the back-up plan should recognize availability and lead times required to employ certain components, such as installing additional lines or modems and multiplexers/concentrators at a recovery site.
The relative importance of the applications processed and the extent to which an institution depends on its telecommunications system will determine the degree of back-up required. Management should make a careful appraisal of its back-up telecommunications requirements, decide on an effective plan, detail the procedures, and periodically test its effectiveness.
Telecommunications Service Priority System (TSPS)
Financial institutions that play a key role in the maintenance of financial systems should be aware of certain government programs and offices that work to coordinate and expedite the restoration or procurement of telecommunications services during an emergency. The Office of Priority Telecommunications (OPT) under the National Communications System (NCS) administers the TSPS, which ensures priority treatment of the nation's most important telecommunications services supporting national security and emergency preparedness missions.This means that TSPS designated circuits will be the first to be repaired in an emergency. All non-federal users requesting TSPS provisioning or restoration are required to have a federal agency sponsor. Institutions are encouraged to contact their primary federal regulator for information on the TSPS program and whether they qualify for a TSPS designation. If they do qualify, the financial institution's restoration and recovery plan should include the TSPS program as a key component.
Government Emergency Telecommunications Service (GETS) and Wireless Priority Service Program (WPS)
Some financial institutions may qualify for sponsorship in the GETS card program and the WPS program, which is the wireless complement to GETS. GETS and WPS are both administered by NCS and provide emergency access and priority processing for voice communications services in emergencies. Financial institutions that perform national security or emergency preparedness functions that are essential to the maintenance of the nation's economic posture during any national or regional emergency will qualify for program sponsorship.WPS users are encouraged to use GETS to enhance telecommunications services, and both of these programs may prove helpful when heavy usage of the public switched network or the wireless network creates congestion and decreases the probability of completing a call.
Additionally, in the event state and federal emergency response authorities commandeer cell phone circuits to manage disaster relief efforts, these programs may provide voice communications for financial institutions that have made prior arrangements for these services. Private sector financial institutions may request GETS Cards by submitting an application to their primary federal regulator. Institutions should limit GETS Cards requests to key personnel with crisis management responsibilities or other senior management personnel responsible for carrying out communications during times of emergency.
Third-Party Providers, Key Suppliers, and Business Partners
Reliance on third-party providers, key suppliers, or business partners may expose financial institutions to points of failure that may prevent resumption of operations in a timely manner. The risks in outsourcing information, transaction processing (core, ATM, and EFT), and cash and settlement activities include threats to the security, availability and integrity of systems and resources, to the confidentiality of information, and to regulatory compliance. In addition, when a third party performs services on behalf of the institution, increased levels of credit, liquidity, transaction, and reputation risk can result.
During widespread telecommunications outages, considerable challenges emerge regarding real-time communications and cross-industry interdependencies with core processors and other third-party service providers, including ATM and EFT business partners. For financial institutions and their branch offices, timely connectivity with significant vendors, suppliers, service providers, and business partners is critical in order to conduct routine banking transactions. Therefore, redundant systems and manual operating procedures should be an integral part of financial institutions' and service providers' BCP. For example, alternate methods for processing EFT through Internet based systems, proprietary software, or correspondent bank relationships should be established to ensure timely transmission of customer transactions. To ensure that employees understand cross-industry interdependencies and manual operating procedures, comprehensive, enterprise-wide testing should be performed.
Redundant telecommunications links can also be established with the service provider through the development of a contractual arrangement that allows either party to switch its connection to an alternate communication path. For example, either party could use permanent virtual circuit or switched virtual circuit technology, which re-routes the communication path around a problem location either permanently or temporarily, as deemed necessary, and assists in re-establishing timely connectivity between the service provider and the institution.
Reliance on correspondent financial institutions or other third parties for liquidity needs also represents a critical aspect of the BCP process. In the event of an area-wide disaster, existing arrangements with cash providers and delivery services may not be feasible. Therefore, management should establish procedures for securing, storing, delivering, and distributing cash despite having limited power, telecommunications, staff, and security available.
Vendor Due Diligence
To ensure timely recovery of operations, management should routinely perform vendor due diligence.As part of this due diligence process, management should inquire about the physical paths used by the service provider to ensure that system redundancies have been properly implemented. Institutions should also review the service provider's BCP and ensure that critical services can be restored within acceptable timeframes based upon the needs of the institution. The contract with the service provider should address the service provider's responsibility for maintenance and testing of disaster recovery and contingency plans. Financial institution management should request a copy of the service provider's BCP test results and audit reports to determine the adequacy of business continuity plans and the effectiveness of the testing program. If possible, the institution should consider participating in the service provider's testing process. If the service provider fails to perform satisfactorily during a service disruption, management should determine whether the institution has sufficient resources and capacity to perform these processes internally or if alternate vendor arrangements should be considered.
Transaction Processing and Report Distribution
Alternate methods of transaction processing and report distribution represent another important element of recovery for serviced institutions. During area-wide disasters, remote image capture systems, using a VPN connection, may allow financial institutions to scan daily items and electronically deliver the imaged information to their service provider for processing without having to physically transport the daily work. In addition, the financial institution may use remote capture software and a secure Internet connection to retrieve various reports needed for operations.
Many financial institutions contract with third-party service providers and other vendors for disaster recovery assistance. These arrangements can be cost-effective for smaller institutions since the cost of maintaining a dedicated recovery site can be substantial. When contracting with third-party providers for recovery services, institutions should consider:
- Staffing-What kinds of technical support personnel is the service provider obligated to make available onsite to assist institution employees in getting the recovery site operating?
- Processing Time Availability-Assuming that other clients are also using the same recovery site, how much processing time is the institution entitled to on a particular computer system? Is the institution guaranteed a sufficient amount of processing time to handle the volume of work that will need to be done at the site?
- Access Rights-Since most back-up sites can be used by numerous clients, does the institution have a guaranteed right to use the site in case of an emergency? Alternatively, does the service provider accept clients on a first-come, first-serve basis until the recovery site is at full capacity? When the back-up site is oversubscribed, is there a limit on the amount of time each client can use the facility?
- Hardware and Software-Is the recovery site equipped with the precise computer hardware and software that the institution needs to continue operations? Will the institution be notified of changes in the equipment at the recovery site?
- Security Controls-Does the recovery site have sufficient physical and logical security to adequately protect the institution's information assets?
- Testing-Does the contract with the service provider permit the institution to perform at least one full-scale test of the recovery site annually? Does the service provider perform tests of its' own BCP and submit test reports to customer financial institutions?
- Confidentiality of Data-In the event other businesses are also using the recovery site, what steps will the service provider take to ensure the security and confidentiality of institution data?
Has the service provider entered into an appropriate contract with the financial institution that addresses the requirements of the "Interagency Guidelines Establishing Information Security Standards"?
- Telecommunications-Has the service provider taken appropriate steps to ensure that the recovery site will have adequate telecommunications services (both voice and data) for the number of personnel that will be working at that site and the volume of data transmissions that are anticipated?
- Reciprocal Agreements-In the event the institution has a reciprocal agreement with another financial institution, does the other institution have sufficient excess computer capacity to ensure that the affected institution's work will be done? Are the hardware and software at the recovery site compatible with the affected institution's systems? Will the institution be notified of changes in equipment at the recovery site? Will the site be available in the event of an area-wide disaster?
- Space-Does the recovery site have adequate resources to accommodate the institution's employees by providing basic necessities and enabling them to conduct business?
- Paper Files and Forms-Does the recovery site maintain a sufficient inventory of paper-based files and forms that are necessary to perform business functions?
- Printing Capacity/Capability-Does the recovery site maintain adequate printing capacity to meet the demand of the affected institution?
- Contacts-Who is authorized to initiate use of the back-up site? Who does the institution contact at the back-up site, and how much lead time is needed prior to the financial institution's arrival at the back-up site? How much will it cost to activate the back-up site?
Internal Systems and Business Processes
The failure of critical systems or the interruption of vital business processes could prevent timely recovery of operations. Therefore, financial institution management must fully understand the vulnerabilities associated with interrelationships between various systems, departments, and business processes. These vulnerabilities should be incorporated into the BIA, which analyzes the correlation between system components and the services they provide.
Various tools can be used to analyze these critical interdependencies, such as a work flow analysis, an organizational chart, a network topology, and inventory records. A work flow analysis can be performed by observing daily operations and interviewing employees to determine what resources and services are shared among various departments. This analysis, in conjunction with the other tools, will allow management to understand various processing priorities, documentation requirements, and the interrelationships between various systems.
The analysis of internal interdependencies will become even more important during a disruption, particularly if the financial institution is required to relocate to another facility and comparable systems are not available. For example, financial institutions sometimes develop stand-alone programs, called stovepipe applications, in attempt to solve an immediate problem without regard to interoperability issues. While these applications may work well within the institution's environment, they may not easily integrate with other applications or systems. Therefore, when performing business continuity planning, management should be aware of the processes that are dependent upon these stand-alone programs and consider their impact on recovery strategies.
While every financial institution is unique and has its own risk profile, management should consider the following issues when determining critical interdependencies within the organization:
- Key personnel;
- Vital records;
- Shared equipment, hardware, software, data files, and workspace;
- Production processes;
- Customer services;
- Network connectivity; and
- Management information systems.
Appendix D: Pandemic Planning
Appendix F: Business Impact Analysis Process