Appendix D: Managed Security Service Providers
Background and Purpose
A growing number of financial institutions (FIs) are partially or completely outsourcing the security management function to third parties, typically known as Managed Security Service Providers (MSSPs). FIs engage MSSPs due to increasingly sophisticated threats, cost pressures, and absence of internal expertise. The services that MSSPs provide present additional risks FIs are required to manage.
The purpose of this appendix is to identify the risks associated with the MSSP engagement and offer guidance to assist FIs in mitigating these risks. While the Federal Financial Institutions Examination Council (FFIEC) Information Technology Examination Handbook (IT Handbook) on Information Security Booklet provides related guidance, FIs should pay particular attention to risk management issues that are heightened when serviced by MSSPs. The loss of control that comes with the outsourced security function introduces an element of risk that FIs need to understand and appropriately manage. The following subsection, MSSP Engagement Criteria covers numerous engagement criteria and related contract considerations institutions should consider when engaging an MSSP.
In addition to the normal vendor management responsibilities, a successful engagement with an MSSP should include:
- A contract with mutually agreed upon Service Level Agreements (SLAs);
Strategies for ensuring transparency and accountability that include:
o Regular communication between the FI and the MSSP on
matters inncluding change control, problem resolution,
threat assessments, and MIS reporting,
o Descriptions of processes for physical and logical controls over
FI data; and,
- Periodic review of the MSSP's processes, infrastructure, and control environment through offsite reviews of documentation and onsite visitations.
Types of Managed Security Services
Following are some of the many types of security-related services offered by MSSPs:
Network Boundary Protection
Using technology such as firewalls and virtual private networks (VPNs), the MSSP protects the FI's network perimeter. The MSSP should provide device monitoring of connections to external third parties such as Internet Service Providers.
Management of Intrusion Detection and Prevention for Networks and Hosts
Intrusion detection systems (IDS) and intrusion prevention systems (IPS) are automated services that can detect patterns in network traffic and may take action according to a rule set or pattern definition database.
Event Log Management and Alerting
Event log management and alerting is conducted to monitor event logs generated by network devices or computer systems to centralize, filter, and provide management reports on material activity. Alerts can be set for highly sensitive events or activities.
Anti-virus and Web Content Filtering Services
Managed antivirus protection provides organizations with malware protection that helps safeguard FIs from new threats. The malware definitions are updated frequently to help recognize the new threats.
Patch Management and Security Software Management
MSSPs can identify and manage network security related software systems and components requiring regular security updates; conduct compatibility testing before deployment; deploy the updates uniformly; and provide reporting on the status and effectiveness of the security software as implemented.
Security Incident Response and Management
MSSPs can assist an organization in building an incident response team or providing a turnkey incident response in the event of a breach.
Data Leak Prevention
MSSPs can help identify all methods of data ingress and egress, and establish systems that monitor and enforce appropriate controls.
MSSPs can provide services to ensure the security of messages into and out of the FI.
Information Security Consulting Services
Security consulting by MSSPs may include risk assessment, vulnerability assessment testing, penetration testing, compliance tools, education and training, and attestation services.
Description of Managed Security Services Arrangements
Managed security services arrangements can include the following four deployment models:
Full Outsourcing: Under this model the MSSP performs the following functions autonomously.
o Manage all network connections at customer premises;
o Manage network platforms;
o Update rules and thresholds over networking devices;
o Analyze data and necessary escalation responses; and
o Provide client reports or alerts on outcomes of the managed
Co-managed: Under this model the FI and MSSP use the same infrastructure and have access rights and responsibilities on platforms.
o Typically involves client-owned network equipment on their
o Includes common security event monitoring tools and data loss
prevention solutions; and
o IDS/IPS events are reported to the MSSP and the FI consults
with the MSSP providing primary services during off hours.
Split Processing: Under this model, the MSSP performs some functions and the FI performs others.
o Most commonly used with firewall and network devices where the
MSSP monitors log data, health and capacity with the FI pushing
system updates, rules changes, or configurations;
o Vulnerability assessment and analysis where the MSSP and FI
each test applications and platforms; and
o Sometimes used when multiple MSSPs are employed.
Consulting: Consulting services provided by MSSPs can include assisting with risk assessments, initial system configuration, policy formulation, compliance (PCI, GLBA, and SOX), forensics, penetration testing, application security testing, application code review, social engineering, physical security, and management reporting.
Effective governance is fundamental for understanding and managing the risks involved when outsourcing to MSSPs. Critical areas include availability, integrity, and confidentiality of FI data. The costs to procure, operate, and manage service delivery, including review for compliance with the SLAs, should be part of the overall contract.
A risk assessment must be performed as part of, or in conjunction with, the due diligence review when an FI is considering outsourcing security services. Concerns about vendors become especially important as security practices that were traditionally conducted in-house are outsourced to an MSSP. The MSSP risk assessment should guide the FI as it develops, implements, tests, and maintains the information systems security program.
Financial Institution Requirements
Gathering necessary information internally and from the potential MSSP is necessary to identify potential threats, vulnerabilities, and controls. Documentation of the risk assessment is especially important to help ensure coordination, consistency, and standardization between the FI and the MSSP. The identification of information systems and the ranking of sensitive data and applications at the MSSP should be part of the risk assessment process. Coordination is also necessary to help ensure that vulnerabilities are identified and processes are validated through testing.
Risk Considerations for Managed Security Services
The reliance on MSSPs may significantly increase an FI's risk profile. Increased risk can arise from poor planning, lack of oversight and control, and/or poor MSSP performance or service. To control these risks, the FI should exercise appropriate due diligence prior to entering an MSSP relationship and maintain effective governance during the relationship.1
Below are risk elements that should be considered in an FI's MSSP risk assessment.2 The risks identified are relevant regardless of the type of MSSP arrangement.
Risk Elements Pertaining to Managed Security Services
According to the FI's risk profile, the following risks should be considered:
o Decline in business reputation and customer confidence;
o Liability under business partnership agreements;
o False sense of security by FI management;
o Diverse offshore legal, geo-political, and cultural risk;
o Impact on competitive advantage when valuable intellectual
property or proprietary information is stolen;
o Reputational damage should the MSSP fail to provide the
o Heightened legal and regulatory issues;
o Dependence on an outside organization for critical services;
o Loss of the FI experience, knowledge, and skill development; and
o Vendor financial condition decline.
Information Security Infrastructure
To optimize service availability while mitigating risks, the following should be considered:
o Complexity of network infrastructure and deployment of agents;
o Information security breaches and data loss;
o Loss to the FI for failing to comply with applicable regulations
o Downtime due to lack of resilient MSSP infrastructure; and
o Loss of the FI's key control requirements due to MSSP's "one
size fits all" products.
Access Management and Control
Ensure FI and MSSP user access is monitored, controlled, and assessed for inappropriate or inadequate:
o MSSP access of FI data;
o User access controls;
o Segregation of duties;
o Control and oversight of MSSP activity by the FI; and
o Attestations of MSSP access to FI systems and data.
Protection Against Malware
To protect the computing environment of malicious software the MSSP should have the following:
o Current antivirus/malware protection;
o Strong patch and/or configuration management policies
o Timely identification of compromised devices; and
o Appropriate endpoint protection tools.
Data and Media Handling
To foster adequate data and media handling protection, consider if the MSSP has:
o Proper application configuration;
o Secure data storage and/or processing by MSSPs;
o Adequate access and integrity controls;
o Appropriate encryption;
o Adequate key management for encrypted data; and
o Sufficient data retention.
Application Development and Systems Integration
An FI should confirm that application development and change management are performed securely by an MSSP. The following should be considered:
o Configuration specifications;
o Change management processes at the MSSP and/or at the FI;
o Logging and monitoring; and
o Recertification of software and permissions.
Business Continuity and Disaster Recovery
An FI should confirm that MSSPs can provide resilient services in the event of an outage or disruption. Risks that FIs should identify and address include:
o Incompatible continuity plans and unrealistic disaster recovery
o Insufficient distance between datacenter and backup datacenter
(or recovery cite) for disaster recovery;
o Inadequate disaster recovery testing and postmortem report
(Disaster recovery is not in line with disaster recovery needs.);
o Poor communication between the FI and MSSP during
a disaster; and
o Inadequate capacity of the MSSP to service all clients during an
Incident Response Management
An FI should identify, monitor, and manage incidents in coordination with the MSSP. Risks that FIs should identify and address include:
o Undefined roles and responsibilities between the FI and the
o Untimely reporting of incidents and/or data breaches;
o Failure to take appropriate steps to contain and control the
o Failure to notify the FI's customers or regulators on FI's behalf
per contract agreement;
o Failure to perform joint incident response table-top testing with
o Overdependence on the MSSP for incident response; and
o Legal issues arising from a security incident involving both
Awareness and Training
An FI should determine that all parties are aware of and trained in processes and MIS reports. Potential risks to be addressed include:
o Insufficient training or expertise at either the MSSP or FI; and
o Inadequate MSSP personnel screening practices.
Request for Information and Request for Proposal
Request for Information (RFI) and Request for Proposal (RFP) are part of a deliberate and intentional process associated with engaging an MSSP. This type of evaluation should be completed in accordance with the FI's strategic plan and tactical approach to security. For example, the strategic plan should determine what security functions to maintain in-house, whether to contract a sole provider, or split services between providers. The RFI, the initial formal step in selection, must define FI objectives for the service needed. These objectives primarily are to be based on the FI's configuration (OS, security, network, and servers) and security policies. The FI should also consider the MSSP's staffing, certifications, training, transition process, and incident response methodology.
It is essential for the FI to coordinate with the MSSP regarding configuration and staff resources. This will be important not only to initial selection through the RFI/RFP and contracting process, but as the relationship evolves. It is important that the MSSP be a cultural fit with the FI. 3 MSSP specific contract language will require modification to RFIs and RFPs based on the FI and vendor configuration. See subsection, MSSP Engagement Criteria for RFI/RFP examples specific to MSSPs.
Initial Due Diligence
An FI considering an MSSP engagement must perform adequate due diligence to validate that the vendor is capable of managing security services that are aligned with their risk profile. Management should consider performing an onsite visitation to determine if the servicer has the appropriate experience and control environment to meet the FI's needs, how long the MSSP has been in business, the MSSP's staffing, the MSSP's incident response methodology, etc.
When performing an onsite visitation, the FI should determine if the MSSP can ensure the security of their data. Pertinent entity and operating information should be obtained to facilitate the vendor selection process. Discussions with management should focus on the risk elements noted in the risk assessment section with emphasis on determining that the MSSP has the necessary expertise and experience to service the FI and will provide sufficient metrics for the FI to assess compliance with the contract.
The time the MSSP has been operating and if there are any expected changes (e.g., merger, acquisitions, expansion/growth, etc.), that could impact contracted services should be determined. The number of clients the MSSP services and number of FI clients also should be identified. If the MSSP does not have FI clients, it may indicate the vendor is seeking to enter into an unfamiliar business area. Before accepting this risk, the MSSP's familiarity with pertinent regulatory requirements such as GLBA, SOX, and FFIEC guidance must be validated.
When evaluating the MSSP's expertise, the following should be considered:
Current and unbiased customer testimonials and/or references;
Use of current monitoring and risk management technologies;
The MSSPs ability to:
o Generate timely MIS reporting and incident notification;
o Maintain confidentiality, integrity, and availability of FI data; and
Manage prospective services for the defined contract term.
If the MSSP does not perform all services in-house, FIs should determine which services are to be outsourced, the quality of vendor management exercised by the MSSP, and whether the service provider(s) is/are offshore.
To fulfill its duties, an MSSP may be required to install software and/or hardware in an FI's data center. What data will be collected, reviewed, stored, and secured by the MSSP should be defined with established SLAs based on business requirements. The dialog between the MSSP and the FI should focus on identifying services that preserve security.
The initial due diligence process is a key method to determine if an MSSP can provide the necessary services to an FI. Each of the above recommendations should be considered in deciding if the MSSP is viable and has the ability to fulfill the terms of the engagement.
In any MSSP arrangement, the contractual expectations and obligations of each party should be clearly defined, understood, monitored, and enforced. FI managers who have a strong understanding of MSSP risks and mitigating controls should be involved in contract development. Legal representatives with the expertise to assess the enforceability and legitimacy of MSSP contract terms should review contract provisions and be included in contract negotiations. The alignment of contract provisions with FI security policies and procedures creates a strong foundation for the development of comprehensive MSSP agreements.
Although most contract requirements for MSSPs are similar to those of other outsourcing arrangements, FIs should consider the following provisions when developing a formal contract with an MSSP.
Scope of Service
Contract discussion should include:
Specific services provided, timelines for implementation, and explicit responsibilities of the MSSP and the FI;
The right to modify existing services performed under the contract;
The type and frequency of reports available;
Activities the MSSP is allowed to conduct when operating within the FI network;
Handling of confidential data;
Ownership of data generated by proprietary security or third-party monitoring tools owned by the MSSP; and
Access rights granted to the MSSP as it relates to FI network systems.
Service Level Agreements
Well defined SLAs provide the framework for establishing the expectations and metrics for the effective delivery of service such as levels of availability, performance, or support. When working with MSSPs, attention should be given to the engagement criteria in Appendix A.
Contract Term and Renewal
The role of the MSSP relationship and how the length of the contract integrates with the FI's overall business strategy and objectives should be defined. Long-term contracts may limit flexibility and consideration should be given to whether to accept automatic contract renewal provisions.
FIs should consider including termination rights for a variety of conditions including material breach, critical performance failure, and material non-performance. Grounds for termination should be clearly defined and agreed on by the FI and service provider. If the contract is terminated for cause, the MSSP should cover damages. The FI's exit strategy should consider post-termination rights including:
Transfer of data in the FI's preferred format;
Transfer of FI data or assets from the MSSP and all subcontractors;
Assistance from the service provider to migrate services in-house or to another provider;
Right to purchase non-proprietary tools used by the MSSP to provide the services; and
Timely response to the FI's post-termination requests.
Managing the Relationship
While the initial due diligence is critical to managing the MSSP relationship, ongoing monitoring and oversight is equally important. Risks of the MSSP relationships are generally similar to risks of other outsourcing arrangements that need to be addressed within the FI's vendor management program, but the MSSP relationship has some attributes that may call for a heightened level of (or more targeted) education and training.
Education and Awareness
Effective MSSP oversight requires an FI to maintain adequate in-house technical expertise. This enables the FI to monitor and maintain acceptable risk exposure and confirm the MSSP is fulfilling contractual obligations. Education and awareness for FI employees is necessary to help ensure:
The MSSP is effectively managing the relevant information security risk;
Personnel understand the processes, procedures, and protocols of the MSSP, including the use of subcontractors; and
FI management understands:
o What data the MSSP is collecting and who has access to the
o Information in audit reports and security testing of the MSSP;
o How to measure a successful relationship.
Given the high risk and trust of the relationship, the FI should verify that the MSSP is appropriately managing the contracted security services on its behalf. The following should be addressed in the FIs education and awareness program:
Training, education, and awareness provided by the MSSP to FI employees;
Identifying and understanding accountable and responsible parties at the FI and MSSP;
Maintaining the expertise needed to understand metrics and reporting provided by the MSSP; and
Training frequency for FI employees.
FI management should have a monitoring process to attest to the MSSP meeting its contractual obligations. This typically entails reviewing items such as SLAs, Key Performance Indicators (KPIs), Key Risk Indicators (KRIs), security event notification, incident response, and any other metrics relative to performance. These items should be included in MSSP reports, and FIs should perform supplemental monitoring as necessary to evaluate contractual performance.
Ongoing Oversight of MSSP Relationship
The critical services provided by MSSPs require a high level of FI oversight throughout the lifecycle of the contracting relationship. Processes should include maintenance of controls established as part of the initial due diligence, including:
o MSSP provided MIS reports,
o MSSP audit reports, including SSAE 16 and other independent
assessment reports, and
o Penetration testing and vulnerability assessment test results;
Performing periodic onsite visitations of the MSSP;
Monitoring the MSSP's internal risk assessment process; and
Discussing any concerns related to the above items with MSSP management.
Contingency & Event Planning
Business Continuity Planning
To avoid a gap in service in the event of an MSSP outage, the FI should:
Review the MSSP's business continuity plans for the ability to provide continuous services to the FI;
Confirm that MSSPs have tested their business continuity plans at least annually and have forwarded a summary to the FI; and
Include critical MSSPs in the FI's tabletop exercise or other business continuity testing.
To assess that the FI is fully prepared to respond to incidents, the FI should:
Develop and maintain an incident response plan which includes a remediation process clearly defining roles and responsibilities between the MSSP and the FI;
Establish and review processes and procedures to handle communications to and from the MSSP;
Establish and define event types and response procedures; and
Include the MSSP in testing of the incident response plan.
To prevent gaps in service associated with MSSP failure, the FI should:
Maintain awareness of alternate providers;
Develop policies and procedures to outline FI data ownership;
Have a clear understanding of service provider roles and responsibilities;
Assess the MSSP for dependencies with critical services; and
Consider using multiple vendors to provide various MSSP services.
Demarcation of Responsibility
Along with general monitoring and oversight of the MSSP, FIs should have involvement in the operational and policy activities associated with the MSSP. Examples include:
Policy and Procedures
Outsourcing certain security activities does not diminish the need for adequate security polices at the FI. They should coordinate their information security program with the policies, standards, guidelines, and procedures of the MSSP.
The incident response function needs to be coordinated and clearly defined between the FI and MSSP. Notification and escalation requirements regarding incident response should be clearly documented and aligned between the FI and MSSP. The definition of a reportable event should be clear and unambiguous.
Assess controls/methods and audit trails related to the FI's systems, devices, and data being managed by the MSSP.
Typically the MSSP will place devices within the FI (e.g., firewall, IDS, etc.) which the MSSP may own and/or control. FIs should consider appropriate physical security of such devices regardless of ownership and/or control.
There should be a clear process to communicate changes implemented by either the FI or MSSP. Changes can have a material impact on the security environment, and both parties should undergo an adequate change control review. Advanced notification of any changes should be provided whenever possible.
The FI should maintain awareness of data the MSSP is collecting, how it is stored, and how it is used. The FI should maintain its data or logs separate from other MSSP clients. The MSSP's data collection and security event classification processes should be defined and understood to help in corroborating the integrity of the FI's data and in establishing a more effective log review process.
Metrics and Reporting
The MSSP should provide regular reporting on agreed on performance metrics to the client FI. It is important that qualified FI personnel review these reports to attest that the security controls of MSSPs are operating as expected. Metrics and reporting should include security:
Events potentially affecting the FI;
Statistics specific to the FI;
Operational statistics and conditions specific to the FI.
Cloud computing is an emerging trend in which some of the IT industry's biggest players are investing significant resources. Cloud computing in general is a migration from owned resources to shared resources in which client users receive information technology services on demand from third-party service providers via the Internet "cloud." In cloud environments, a client or customer will relocate their resources such as data, applications, and services to computing facilities outside the corporate firewall, which the end user then accesses via the Internet.
Cloud-based MSSP services may be implemented as part of Internet access services. Examples of "in-the-cloud" services include carrier-based denial of service protection, virtual firewall services, and carrier-provided URL blocking.
When an MSSP offers services that use a cloud computing architecture, the same risks that are specific to non-cloud-based security services apply. However, there are a few additional risk considerations that should be assessed when moving to a cloud computing environment. Areas for FIs to consider when an MSSP uses cloud computing in their managed security services environment include:
Protecting data in transit to avoid data leakage;
Securing data at rest so that one data breach within the cloud does not breach the other customer data within the cloud;
Maintaining compliance with applicable regulatory requirements;
Complying with foreign government privacy laws when outsourcing is performed offshore;
Segregating customer data appropriately to comply with audit and legal requirements; and
Avoiding sharing of authentication credentials to prevent the impersonation of users.
Financial institutions' challenges in dealing with high profile network security breaches, changing technology, malware, system maintenance costs, complexity, and uncertainty surrounding network security have resulted in an increased use of MSSPs. While FIs can leverage the expertise of the MSSP, managing this relationship can be an additional challenge, particularly when MSSPs have access to confidential or sensitive information that requires increased protection. In addition, FIs can have high levels of risk exposure in the event that an MSSP cannot comply with service level agreements.
As with all outsourcing arrangements FI management can outsource the daily responsibilities and expertise; however, they cannot outsource accountability.
Appendix C: Foreign-Based Third-Party Service Providers
MSSP Engagement Criteria