Defining an Incident Response Plan

All members of your IT environment should be aware of what to do in the event of an incident. The CSIRT will perform most actions in response to an incident, but all levels of your IT staff should be aware of how to report incidents internally. End users should report suspicious activity to the IT staff directly or through a help desk rather than directly to the CSIRT.

Every team member should review the incidence response plan in detail. Having the plan easily accessible to all IT staff will help to ensure that when an incident does occur, the right procedures are followed.

To instigate a successful incident response plan, you should:

·         Make an initial assessment.

·         Communicate the incident.

·         Contain the damage and minimize the risk.

·         Identify the type and severity of the compromise.

·         Protect evidence.

·         Notify external agencies if appropriate.

·         Recover systems.

·         Compile and organize incident documentation.

·         Assess incident damage and cost.

·         Review the response and update policies.

These steps are not purely sequential. Rather, they happen throughout the incident. For example, documentation starts at the very beginning and continues throughout the entire life cycle of the incident; communication also happens throughout the entire incident.

Other aspects of the process will work alongside each other. For example, as part of your initial assessment, you will gain an idea of the general nature of the attack. It is important to use this information to contain the damage and minimize risk as soon as possible. If you act quickly, you can help to save time and money, and your organization's reputation.

However, until you understand the type and severity of the compromise in more detail, you will not be able to be truly effective in containing the damage and minimizing the risk. An overzealous response could even cause more damage than the initial attack. By working these steps alongside each other, you will get the best compromise between swift and effective action.

Note: It is very important that you thoroughly test your incident response process before an incident occurs. Without thorough testing, you cannot be confident that the measures that you have in place will be effective in responding to incidents.

Making an Initial Assessment

Many activities could indicate a possible attack on your organization. For example, a network administrator performing legitimate system maintenance might appear similar to someone launching some form of attack. In other cases, a badly configured system might lead to a number of false positives in an intrusion detection system, which could make it more difficult to spot genuine incidents.

As part of your initial assessment, you should:

·         Take steps to determine whether you are dealing with an actual incident or a false positive.

·         Gain a general idea of the type and severity of attack. You should gather at least enough information to begin communicating it for further research and to begin containing the damage and minimizing the risk.

·         Record your actions thoroughly. These records will later be used for documenting the incident (whether actual or false).

Note: You should avoid false positives whenever possible; however, it is always better to act on a false positive than fail to act on a genuine incident. Your initial assessment should, therefore, be as brief as possible, yet still eliminate obvious false positives.

Communicating the Incident

Once you suspect that there is a security incident, you should quickly communicate the breach to the rest of the core CSIRT. The incident lead, along with the rest of the team, should quickly identify who needs to be contacted outside of the core CSIRT. This will help to ensure that appropriate control and incident coordination can be maintained, while minimizing the extent of the damage.

Be aware that damage can come in many forms, and that a headline in the newspaper describing a security breach can be much more destructive than many system intrusions. For this reason, and to prevent an attacker from being tipped off, only those playing a role in the incident response should be informed until the incident is properly controlled. Based on the unique situation, your team will later determine who needs to be informed of the incident. This could be anyone from specific individuals up to the entire company and external customers. Communication externally should be coordinated with the Legal Representative.

Containing the Damage and Minimizing the Risks

By acting quickly to reduce the actual and potential effects of an attack, you can make the difference between a minor and a major one. The exact response will depend on your organization and the nature of the attack that you face. However, the following priorities are suggested as a starting point:

1.       Protect human life and people's safety. This should, of course, always be your first priority.

2.       Protect classified and sensitive data. As part of your planning for incident response, you should clearly define which data is classified and which is sensitive. This will enable you to prioritize your responses in protecting the data.

3.       Protect other data, including proprietary, scientific, and managerial data. Other data in your environment might still be of great value. You should act to protect the most valuable data first before moving on to other, less useful, data.

4.       Protect hardware and software against attack. This includes protecting against loss or alteration of system files and physical damage to hardware. Damage to systems can result in costly downtime.

5.       Minimize disruption of computing resources (including processes). Although uptime is very important in most environments, keeping systems up during an attack might result in greater problems later on. For this reason, minimizing disruption of computing resources should generally be a relatively low priority.

There are a number of measures that you can take to contain the damage and minimize the risk to your environment. At a minimum, you should:

·         Try to avoid letting attackers know that you are aware of their activities. This can be difficult, because some essential responses might alert attackers. For example, if there is an emergency meeting of the CSIRT, or you require an immediate change of all passwords, any internal attackers might know that you are aware of an incident.

·         Compare the cost of taking the compromised and related systems offline against the risk of continuing operations. In the vast majority of cases, you should immediately take the system off the network. However, you might have service agreements in place that require keeping systems available even with the possibility of further damage occurring. Under these circumstances, you can choose to keep a system online with limited connectivity in order to gather additional evidence during an ongoing attack.
In some cases, the damage and scope of an incident might be so extensive that you might have to take action that invokes the penalty clauses specified in your service level agreements. In any case, it is very important that the actions that you will take in the event of an incident are discussed in advance and outlined in your response plan so that immediate action can be taken when an attack occurs.

·         Determine the access point(s) used by the attacker and implement measures to prevent future access. Measures might include disabling a modem, adding access control entries to a router or firewall, or increasing physical security measures.

·         Consider rebuilding a fresh system with new hard disks (the existing hard disks should be removed and put in storage as these can be used as evidence if you decide to prosecute attackers). Ensure that you change any local passwords. You should also change administrative and service account passwords elsewhere in your environment.

Identifying the Severity of the Compromise

To be able to recover effectively from an attack, you need to determine how seriously your systems have been compromised. This will determine how to further contain and minimize the risk, how to recover, how quickly and to whom you communicate the incident, and whether to seek legal redress.

You should attempt to:

·         Determine the nature of the attack (this might be different than the initial assessment suggests).

·         Determine the attack point of origin.

·         Determine the intent of the attack. Was the attack specifically directed at your organization to acquire specific information, or was it random?

·         Identify the systems that have been compromised.

·         Identify the files that have been accessed and determine the sensitivity of those files.

By performing these actions, you will be able to determine the appropriate responses for your environment. A good incident response plan will outline specific procedures to follow as you learn more about the attack. Generally, the nature of the attack symptoms will determine the order in which you follow the procedures defined in your plan. Since time is crucial, less time-consuming procedures should generally be carried out before more lengthy ones. To help determine the severity of the compromise, you should:

·         Contact other members of the response team to inform them of your findings, have them verify your results, determine whether they are aware of related or other potential attack activity, and help identify whether the incident is a false positive. In some cases, what might appear to be a genuine incident on initial assessment will prove to be a false positive.

·         Determine whether unauthorized hardware has been attached to the network or whether there are any signs of unauthorized access through the compromise of physical security controls.

·         Examine key groups (domain administrators, administrators, and so on) for unauthorized entries.

·         Search for security assessment or exploitation software. Cracking utilities are often found on compromised systems during evidence gathering.

·         Look for unauthorized processes or applications currently running or set to run using the startup folders or registry entries.

·         Search for gaps in, or the absence of, system logs.

·         Review intrusion detection system logs for signs of intrusion, which systems might have been affected, methods of attack, time and length of attack, and the overall extent of potential damage.

·         Examine other log files for unusual connections; security audit failures; unusual security audit successes; failed logon attempts; attempts to log on to default accounts; activity during nonworking hours; file, directory, and share permission changes; and elevated or changed user permissions.

·         Compare systems to previously conducted file/system integrity checks. This enables you to identify additions, deletions, modifications, and permission and control modifications to the file system and registry. You can save a lot of time when responding to incidents if you identify exactly what has been compromised and what areas need to be recovered.

·         Search for sensitive data, such as credit card numbers and employee or customer data, that might have been moved or hidden for future retrieval or modifications. You might also have to check systems for non-business data, illegal copies of software, and e-mail or other records that might assist in an investigation. If there is a possibility of violating privacy or other laws by searching on a system for investigative purposes, you should contact your legal department before you proceed.

·         Match the performance of suspected systems against their baseline performance levels. This of course presupposes that baselines have been created and properly updated.

When determining which systems have been compromised and how, you will generally be comparing your systems against a previously recorded baseline of the same system before it was compromised. Assuming that a recent system shadow copy is sufficient for comparison might put you in a difficult situation if the previous shadow copy comes from a system that has already been attacked.

Note: Tools such as EventCombMT, DumpEL, and Microsoft Operations Manager (MOM) can help you determine how much a system has been attacked. Third-party intrusion detection systems give advance warning of attacks, and other tools will show file changes on your systems.

Protecting Evidence

In many cases, if your environment has been deliberately attacked, you may want to take legal action against the perpetrators. In order to preserve this option, you should gather evidence that can be used against them, even if a decision is ultimately made not to pursue such action. It is extremely important to back up the compromised systems as soon as possible. Back up the systems prior to performing any actions that could affect data integrity on the original media.

Someone skilled in computer forensics should make at least two complete bit-for-bit backups of the entire system using new, never-before-used media. At least one backup should be on a write-once, read-many media such as a CD-R or DVD-R. This backup should be used only for prosecution of the offender and should be physically secured until needed.

The other backup can be used for data recovery. These backups should not be accessed except for legal purposes, so you should physically secure them. You will also need to document information about the backups, such as who backed up the systems, at what time, how they were secured, and who had access to them.

Once the backups are performed, you should remove the original hard disks and store them in a physically secure location. These disks can be used as forensic evidence in the event of a prosecution. New hard disks should be used to restore the system.

In some cases, the benefit of preserving data might not equal the cost of delaying the response and recovery of the system. The costs and benefits of preserving data should be compared to those of faster recovery for each event.

For extremely large systems, comprehensive backups of all compromised systems might not be feasible. Instead, you should back up all logs and selected, breached portions of the system.

If possible, back up system state data, as well. It can take months or years until prosecution takes place, so it is important to have as much detail of the incident archived for future use.

Often the most difficult legal aspect of prosecuting a cyber crime is collecting evidence in a manner acceptable to the particular jurisdiction's laws of evidence submission. Hence, the most critical component to the forensic process is detailed and complete documentation of how systems were handled, by whom, and when, in order to demonstrate reliable evidence. Sign and date every page of the documentation.

Once you have working, verified backups, you can wipe the infected systems and rebuild them. This will enable you to begin running your operation again. The backups provide the critical, untainted evidence required for prosecution. A different backup from the forensic backup should be used to restore data.

Notifying External Agencies

After the incident has been contained and data preserved for potential prosecution, you should consider whether you need to start notifying appropriate external entities. All external disclosures should be coordinated with your Legal Representative. Potential agencies include local and national law enforcement, external security agencies, and virus experts. External agencies can provide technical assistance, offer faster resolution and provide information learned from similar incidents to help you fully recover from the incident and prevent it from occurring in the future.

For particular industries and types of breaches, you might have to notify customers and the general public, particularly if customers might be affected directly by the incident.

If the event caused substantial financial impact, you might want to report the incident to law enforcement agencies.

For higher profile companies and incidents, the media might be involved. Media attention to a security incident is rarely desirable, but it is often unavoidable. Media attention can enable your organization to take a proactive stance in communicating the incident. At a minimum, the incident response procedures should clearly define the individuals authorized to speak to media representatives.

Normally the public relations department within your organization will speak to the media. You should not attempt to deny to the media that an incident has occurred, because doing so is likely to damage your reputation more than proactive admission and visible responses ever will. This does not mean that you need to notify the media for each and every incident regardless of its nature or severity. You should assess the appropriate media response on a case-by-case basis.

Recovering Systems

How you recover your system will generally depend on the extent of the security breach. You will need to determine whether you can restore the existing system while leaving intact as much as possible, or if it is necessary to completely rebuild the system.

Restoring data presumes, of course, that you have clean backups#&8212;backups made before the incident occurred. File integrity software can help pinpoint the first occurrence of damage. If the software alerts you to a changed file, then you know that the backup you made before the alert is a good one and should be preserved for use when rebuilding the compromised system.

An incident could potentially corrupt data for many months prior to discovery. It is, therefore, very important that as part of your incident response process, you determine the duration of the incident. (File/system integrity software and intrusion detection systems can assist you in this.) In some cases, the latest or even several prior backups might not be long enough to get to a clean state, so you should regularly archive data backups in a secure off-site location.

Compiling and Organizing Incident Evidence

The CSIRT should thoroughly document all processes when dealing with any incident. This should include a description of the breach and details of each action taken (who took the action, when they took it, and the reasoning behind it). All people involved with access must be noted throughout the response process.

Afterward, the documentation should be chronologically organized, checked for completeness, and signed and reviewed with management and legal representatives. You will also need to safeguard the evidence collected in the protect evidence phase. You should consider having two people present during all phases who can sign off on each step. This will help reduce the likelihood of evidence being inadmissible and the possibility of evidence being modified afterward.

Remember that the offender might be an employee, contractor, temporary employee, or other insider within your organization. Without thorough, detailed documentation, identifying an inside offender will be very difficult. Proper documentation also gives you the best chance of prosecuting offenders.

Assessing Incident Damage and Cost

When determining the damage to your organization, you should consider both direct and indirect costs. Incident damage and costs will be important evidence needed if you decide to pursue any legal action. These could include:

·         Costs due to the loss of competitive edge from the release of proprietary or sensitive information.

·         Legal costs.

·         Labor costs to analyze the breaches, reinstall software, and recover data.

·         Costs relating to system downtime (for example, lost employee productivity, lost sales, replacement of hardware, software, and other property).

·         Costs relating to repairing and possibly updating damaged or ineffective physical security measures (locks, walls, cages, and so on).

·         Other consequential damages such as loss of reputation or customer trust.

Reviewing Response and Updating Policies

Once the documentation and recovery phases are complete, you should review the process thoroughly. Determine with your team which steps were executed successfully and which mistakes were made. In almost all cases, you will find some processes that need to be modified so you can better handle future incidents.

You will find weaknesses in your incident response plan; the point of this post-mortem exercise? You are looking for opportunities for improvement, which should initiate a whole new round of the incident response planning process.