IV.A.1 Key Testing Factors
Management should consider the following key factors when developing and implementing independent tests:
- Scope. The tests and methods utilized, in the aggregate, should be sufficient to validate the effectiveness of the security process in identifying and appropriately controlling the risk from information security-related events.
- Personnel. Technical testing is only as good as the personnel performing and supervising the test. Management should review qualifications of testing personnel to verify testers' capabilities are adequate to support the test objectives.
- Notifications. Management should consider whom to inform within the institution about the timing and nature of the tests. The need for protection of institution systems and the potential for disruptive false alarms should be balanced against the need to test personnel reactions to unexpected activities.
- Confidentiality, integrity, and availability. Management should carefully control information security tests to limit the risks to confidentiality, integrity, and system availability. Because testing may uncover sensitive customer information, management should use appropriate safeguards to protect such information. Management should ensure that employee and contract personnel who perform the tests or have access to the test results have passed appropriate background checks and that contract personnel are appropriately bonded. Because certain tests may pose more risk to system availability than other tests, management should have personnel who perform those tests maintain logs of testing actions. Those logs are helpful if the systems react unexpectedly.
- Confidentiality of test plans and data. Because knowledge of test planning and results may facilitate a security breach, the institution should carefully limit the distribution of testing information. Management should restrict test plans and data only to those individuals involved in the testing. Results should be made available in a usable form only to those responsible for following up on tests. Additionally, management should require contractors to sign nondisclosure agreements and to return information they obtained in their testing to the institution.
- Frequency. The institution's ITRM process should determine the frequency of independent testing. Factors that may increase testing frequency include changes to network configurations, changes to or additions of systems and applications, significant changes in potential attacker profiles and techniques, and results of other testing. For instance, management should have a testing process for security and usability over the life cycle of testing (during development, before placing a new or modified system into production, and periodic testing of the production system or application).
- Proxy testing. Proxy testing refers to testing that is conducted on like systems and with like interfaces, rather than the actual system, to avoid disruptions on a system that may be too critical for a comprehensive continuity test. Proxy tests are conducted using the same hardware and operating software, are sometimes used as a replacement for actual tests, and should provide similar results. Independent testing of a proxy system is generally not effective in validating the effectiveness of a security process. Proxy testing, by its nature, does not test the operational system's policies and procedures or its integration with other systems. It also does not test the reaction of personnel to unusual events. Proxy testing may be the best choice, however, when management is unable to test the operational system without creating excessive risk.
IV.A Assurance and Testing
IV.A.2 Types of Tests and Evaluations