Writing Incident Response Runbooks

Incident response runbook (aka. playbook, “use case”) is a written guidance for identifying, containing, eradicating and recovering from cyber security incidents.

Introduction

Incident response runbook (aka. playbook, “use case”) is a written guidance for identifying, containing, eradicating and recovering from cyber security incidents. The document is usually the output of the preparation phase of the SANS Incident Response process. We are going to talk about a “Phishing Incident Response Playbook” in this article.

Runbooks are similar to Standard Operation Procedures (SOPs) because these also feature simple step-by-step instructions that make any process predictable, repeatable and measurable. The directions are prepared with a certain audience in mind, for instance, junior-level analysts. The runbook approach enables certain outsourcing activities as well, such as allowing Managed Security Service Providers (MSSP) to carry out certain tasks.

The following guide walks you through the writing process of a simple runbook for handling e-mail based social engineering campaigns. By the end of this article, you will understand the general principles behind effective runbooks.

Backstory

In the following example, you are working as the head of Incident Response Team (IRT) for a fictional company named Royal Acacia. This business is a successful distributor of artesian honey across the United States. The vast majority of the company’s revenue is generated by its busy e-commerce website.

The online shop’s popularity is due to the smart utilization of latest trends and technologies. Acacia’s DevOps team is developing a scalable micro service infrastructure, which allows the business to roll new features out on a weekly basis. As a result, the shop takes credit card, Apple Pay, and Bitcoin payments. The company also operates a large data warehouse collecting every type of sensitive data from its customers.

As the management is concerned over data security, they hired a third-party auditor firm to carry out an organization-wide threat assessment. The assessors had identified customer and credit card data as the most valuable assets of Royal Acacia. According to the firm’s report, the greatest threat to these is data theft enabled by social engineering. It is also discovered that the sole control in place – the company’s simple Bayesian SPAM filter – is not sufficient to tackle e-mail based social engineering attempts.

In order to protect the sensitive data, your management decides to support data protection with incident response activities. They engage you as the head of IRT to onboard a process for mitigating phishing campaigns.

Preparation

You realize that the preliminary design phase is crucial for the success of your IR activities.

“In plain English, careful planning always pays in the end.”

The first to determine is what the company should be protecting and from what. This will provide the goals and objectives for your runbook. It is also important to understand how social engineering attempts are going to be detected and what the appropriate responses to them.

Goals and Objectives

Every runbook should have a clearly stated objective. This not only justifies the existence of your runbook but also demonstrates its the business value. The lack of purpose will result in an ineffective runbook wasting everyone’s time.

In your example, the management is concerned about the confidentiality of customer data and social engineering. The goal of your runbook is going to revolve around data protection from social engineering-based attacks against Royal Acacia’s employees.

Detection and Response

In order to detect social engineering attempts, it is essential to understand how phishing campaigns are executed first. As for the response, you must be familiar with the relevant kill chain model at your organization.

You decide to go for a quick win for now with regards to detection, so you let end-users report incoming phishing attempts to [email protected].

Building a Kill Chain

For building the kill chain, you just need to systematically go through the necessary steps of carrying out a successful phishing attack. Remember, there are multiple avenues to trick an employee, so this step should be repeated as needed.

You heard a lot about fake invoice emails spreading malicious software recently, so you build one potential chain of events as the following:

  1. The attacker crafts an “unpaid invoice” email with a URL in it and sends the message to several users at your organization
  2. The phishing email is received by the company’s SMTP server
  3. Email goes through the company’s simple SPAM filter
  4. The phish is moved into the end user’s mailbox
  5. User notifies the new email in the mailbox and reads it
  6. User clicks on the link in the malicious email
  7. A website opens up and offers an “.exe” file (disguised with a pdf icon) to be downloaded
  8. User thinks the file is a genuine invoice and opens it
  9. The file was a dropper that downloads some additional content from the Internet
  10. The additional content automatically installs
  11. The malware opens a permanent connection to a certain IP address
  12. Attacker connects to the PC through this permanent tunnel
  13. Attacker enumerates the available file shares and downloads every Office document is available

Assessing the Controls

According to the kill chain model, the successful data theft can be prevented if we can break the chain at any point. The good news is your IT already has a plenty of tools available to stop the chain of events.

So you take the list from above and expand it with your existing controls as the following:

  1. The attacker crafts an “unpaid invoice” email with a URL in it and sends the message to several users at your organization
    No control available
  2. The phishing email is received by the company’s SMTP server
    Reject emails based on subject line, sender IP, content or other attributes
  1. Email goes through the company’s simple SPAM filter
    Flag “bad” emails to teach the SPAM filter
  2. The phish is moved into the end user’s mailbox
    Royal Acacia IT is capable of removing emails already in the inboxes
  3. User notifies the new email in the mailbox, the user opens and reads it
    You can warn your users of the emerging phishing campaign
  1. User clicks on the link in the malicious email and loads in the web browser
    The user’s computer queries the DNS and the HTTP request goes through the web proxy. You can blackhole DNS requests and block (a pattern of) URLs on the proxy.
  2. A website opens up and offers an .exe file to be downloaded
    You can block certain file types or files with certain attributes to be downloaded on the web proxy
  3. User thinks the file is a genuine invoice and opens it
    Fresh AV signatures can be pushed to the endpoints
  4. The file was a dropper that downloads some additional content from the Internet
    You can block the file based on a certain pattern on the proxy
  5. The additional content automatically installs
    Again, fresh AV signatures can be pushed to the endpoints
  6. The malware opens a permanent connection to a certain IP address, attacker controls the infected PC through this channel
    Proxy logs can help identify these types of connections that you can trace back to the source host on your network
  7. Attacker enumerates the available file shares and downloads every available Office document onto the malware infected PC
    You identify that anomaly detection would probably the appropriate control here, but the company does not have this capability yet
  8. Attacker instructs the infected PC to upload the file to Dropbox

You can block this action or detect retrospectively in your proxy logs.

Not bad, you can control almost each step with an existing control. Time to write the actual playbook now!

Your First Playbook

The following playbook is an example for handling certain types of phishing campaigns. This playbook should be peer-reviewed, trained and practiced before your incident response team uses it.

It is also worth to mention that playbooks should be constantly evolving documents. As your IRT uses the process on a regular basis, some actions may become ineffective and others are missing. Collect feedback from your team and amend the process as accordingly. If the whole playbook becomes irrelevant, consider retiring the process.

ID

10001-INV-USER-SOCIAL

Title

Mitigation of Phishing Campaigns

Date

<document last modification date>

Owner

<your name>

Objective Statement

This runbook provides instructions for handling end-user reported phishing campaigns against Royal Acacia employees. The goal is to prevent the introduction of backdoors into the company infrastructure, which may lead to data theft.

The process aims to lower the success rate of the phishing campaign by blocking emails and backdooring attempts. As this makes financially unsustainable to attack Royal Acacia on the long run, the malicious actor will move on to an easier target.

Scope and Applicability

Phishing emails against Royal Acacia employees

Methodology and Procedures

The following paragraphs elaborate the instructions to be carried out by the IRT.

Handling Employee Phishing Reports – Incident Response Part

  1. Check the [email protected] inbox at least every hour for employee reports
  2. If a new report comes in, create a new ticket in JIRA
  3. Obtain the original email from the end-user and attach it to the ticket
  4. Validate the email whether it is a phish
    1. Check the hostname part of all links in the email on https://otx.alienvault.com or another feed like Virustotal or IBM Xforce Exchange
    2. Check the URL inside the link by hovering over it, is it the same? (e.g. <a href=”www.badurl.com”>www.ebay.com</a>) ?
    3. Does the email try to evade SPAM filters?
    4. Does the sender seem to be forged?

If the email is a false positive, thank the user for the report and resolve the ticket. Otherwise, continue with the instructions below.

Investigation Step 1: Get IoCs

Collect and record the IoCs below into the JIRA ticket.

  1. Full download URL(s) from the email
  2. Hostname from URL
  3. Visit http://www.kloth.net/services/nslookup.php and get the IPs belonging to the hostname
  4. Do reverse lookup and record the PTR record
  5. Fire up a Linux VM, curl the URL to retrieve the file
    1. Calculate the md5 and sha256 hashes
    2. Pack the file with “zip” on the VM and encrypt it with the password: “virus”
    3. Copy the zipped file to your desktop with scp
    4. Attach file and hashes to the JIRA ticket
  6. Subject link of the email
  7. Sender of the email
  8. Hostname and IP address of the sender’s SMTP server

Investigation Step 2: Is it a Campaign?

Search JIRA for emails with similar senders, subjects, contents or URLs

  1. If this is the first phish in a new campaign, skip to the next paragraph
  2. If the phish is part of a campaign
    1. Create a master ticket (if necessary) in JIRA
    2. Relate new ticket to master ticket
    3. Associate related tickets with the master ticket

Alerting Employees

If this is the fifth ticket related to the master ticket in JIRA, we need to warn our end-users of the emerging threat.

  1. Go to \dc1irttemplatesphishing and take the pre-approved email template
  2. Fill in the appropriate details
  3. Send the template to our internal PR on [email protected]
  4. Give our PR team a call on #12345 and let them now we have a pre-negotiated incident situation here and the company-wide alert should go out within an hour

Block Emails on the SMTP Server

As the phisher can easily change the subject line or sender of the emails, try to find a common pattern in the email headers of the related emails. For instance, all emails might share the X-Mailer: and X-PHP-Script: headers.

  1. View the source of the original phishing emails under the master JIRA ticket
  2. Try to find common patterns

If you manage to find a some common attributes, contact IT on [email protected] and request to block all incoming emails based on the identified pattern.

Flagging “Bad” Emails

Send the original email to IT on [email protected] asking them to feed it to the company’s Bayesian SPAM filter.

Removing Emails from User Inboxes

Check the SMTP logs whether the same email has been delivered to other users. Engage IT in removing similar emails from the affected employee mailboxes.

  1. Go to https://splunk.royalacacia.com
  2. Search for the subject line from the original phish
  3. Search for the sender email address from the original phish
  4. If you identify other recipients in the SMTP logs
    1. Export affected recipients into a CSV file
    2. Contact IT on [email protected] and ask them to remove the phishes from the affected mailboxes

Blackholing DNS

We have a pre-approved process with IT to hijack certain DNS requests on the domain controllers by resolving them to 127.0.0.1

  1. Go to https://otx.alienvault.com or similar threat information services.
  2. Pivot on hostname and collect related hostnames
  3. Contact IT on [email protected] and send them the list of hostname IoCs to be blackholed

Blocking Download URL

This process will block the dropper to be downloaded if a user clicks on the malicious URL in the phish.

  1. Try to identify a pattern in the URL. For example if the URL is: http://hackedwebsite.com/dl.php?campaign=ra&id=12731342834923919
  2. Create a reasonable regex like: ^.+/dl.php?.*campaign=ra.$
  3. Contact the proxy team on [email protected] to block URLs based on the pattern

Report Malware Sample to McAfee (or similar services)

Send the sample to McAfee to get the updated malware signature, which will block users from opening files already downloaded (unless the malware file mutates).

  1. Send the malware sample from the JIRA ticket to [email protected]

Deploy AV Signatures (your choice of AV)

If McAfee replies with the additional signature file, we push it out to the endpoints as the following:

  1. Take the dat file received from McAfee and attach it to the ticket
  2. Send the dat file to [email protected] and ask them to push it immediately

Summary

In this article, we have learned about the basics of playbooks. These documents are meant to provide practical instructions to handle certain incident scenarios.

Playbooks are living documents and will never be comprehensive. You will always need to adjust, improve or retire them as the threat landscape changes. Remember, because unexpected things could also happen during an incident, you will also need to improvise on the spot. The playbook is not written in stone.

In this part of the series, we focused on the prevention steps to stop phishing emails from getting in. In the follow-up article, we are going to develop a counterpart of this playbook that handles situations when phishing emails succeed. Head on to the second part of the series, which is dealing with the ever-growing threat of ransomware attacks.

This article has first appeared at Demisto. The author is an expert at Iron Bastion, a boutique cyber security firm in Sydney, Australia.

Gabor

Gabor Szathmari is a cybersecurity expert and digital privacy enthusiast. In his professional life, Gabor helps businesses, including many small and mid-size legal practices, with their cybersecurity challenges at Iron Bastion.