Threat Hunting — Practical Guide (Part 2) | Codelivly

Part 2 of "Hunt Evil": hands-on threat-hunting techniques, detection workflows, tools, and real case examples to help you find and remediate advanced network threats.

As we discussed in the Part 1 , adversaries will come in many forms and will deploy
a wide variety of different Tactics, Techniques and Procedures (TTPs). In order to defend
yourself, you must know your enemy. Similarly to how you orient your overall hunting
plan, the kinds of techniques you use to hunt will depend largely on what you’re trying to
defend against, which in turn will depend largely on what you’re trying to protect.

High Impact Activity to Hunt For

Here are some high impact activities and TTPs that you can start hunting for

These TTPs are grouped by tactic categories from MITRE’s ATT&CK Matrix™.

Internal Reconnaissance

How attackers determine where they’re going

Host Enumeration

Determines details about a local host, which includes establishing an understanding of local user context and local host configuration. User context lets you, as an attacker, know what user you are logged in as and what privileges are allotted to you. Local host configuration includes information about the host itself, including hostname and IP address.

Network enumeration

Establishes what other hosts are remotely accessible from the local host. Once attackers have compromised an initial host, they will need to determine how to move around the network and where they can go. Network enumeration lets you, as an attacker, see what access the host you are on has and what active connections there are to other systems and assets.

Persistence

How attackers survive a reboot and simple remediations

Scheduled Task Execution

This is the process of queueing up programs or scripts that can be operationalized at a later point. Tasks can be also scheduled remotely, assuming that the attacker has the authentication to use Remote Procedure Call (RPC). This allows attackers to run programs when a system starts up, or according to a schedule.

DLL Injection

Using this technique, adversaries will run malicious code by using another process to load and execute the code. This allows adversaries to hide malicious activity by incorporating it as part of a benign or routine process. It also allows attackers to access a system’s process memory and permissions.

Registry modification

The additional values to the RUN and RUNONCE registry key allow for malware binaries to execute upon system boot and session login. This is the most common technique for persistence seen in the last decade.

Command & Control

How attackers utilize their tools

Common Protocol, Common Port

Using this method, attackers will seek to hide in plain sight by blending in with routine network traffic. Oftentimes this takes the form of using HTTPS, DNS tunneling, and high-traffic ports to establish command and control.

Uncommon Protocol, Uncommon Port

By utilizing this technique, adversaries bypass heavily monitored ports and send data through uncommon ports. This allows attackers to operate with a high degree of stealth, allowing them to evade detection from both human operators and routine detection systems.

Discover:  Building a Malicious Backdoor & C2 Server in Python! 

Lateral Movement

How attackers move around in your network

pass-the-hash-pth">Pass the Hash (PtH)

Using this technique, attackers will capture valid password hashes via a Credential Access technique and use that to authenticate themselves. This allows them to bypass using a user’s cleartext password and enables them to perform actions on both local and remote systems.

Remote Desktop Protocol

A remote desktop allows a user to access a computer’s desktop interface using a remote system. If a system’s remote desktop protocol is enabled and an attacker knows their target’s account credentials, the attacker can use that information to gain access to and exploit the target system.

Shared Webroot

If a firm has an internally accessible website or intranet network, attackers may upload malicious content to said website and then execute it using a web browser. Since the content is run under the permissions of the Web server process, this can result in the attacker gaining local or administrative privileges.

Path Interception

This technique entails placing an executable file in a specific path so that it is mistakenly run by a legitimate application. Examples of this include using unquoted paths, path environment variable misconfigurations, and search order hijacking. This allows adversaries to escalate their privileges if the executable is run as part of a higher privileged process.

Exfiltration

How attackers steal your data

DNS Tunneling

This allows an attacker to transfer encoded data inside of DNS queries. Attackers utilize this method to bypass common security controls (e.g., firewalls) and exfiltrate sensitive data.

SFTP/SCP Exfiltration

This allows for usage of SSL to hide details about the traffic. You will need a proxy that can break SSL to intercept and investigate this type of activity.

Consider what adversary techniques you want to focus on and research them as much as you can. Put yourself in the mindset of an attacker and determine how one would carry out each of these techniques if they were breaking into your network and attempting to access your critical assets. This will help you determine what data sets to look at and what techniques to deploy, which we will cover in the next section.

Do not be overwhelmed! Hunting is, in part, an exercise in prioritization. For every insidious tactic that an attacker can use to compromise a system, there is a technique that a stalwart defender can use to repel them. As in all battles, the advantage of being the defender is that you are on your home turf. It is your network, and if you know it and know how to patrol it, you can foresee and stop even the most sophisticated adversaries.

Four Primary Threat Hunting Techniques

Okay, you have your hunting procedures and you have your tools squared away. Now let’s talk about the actual practice of hunting, what techniques are in your arsenal, and some examples of how you can proactively find the adversaries lurking in your network. This is by no means an exhaustive list, these are just a set of general techniques that can be applied in different ways.

Techniques

#1. Searching

The simplest method of hunting, searching is the process of querying data for specific results or artifacts, and can be performed using many tools. Searching requires finely defined search criteria to prevent result overload. There are two primary factors to keep in mind when carrying out a search: searching too broadly for general artifacts may produce far too many results to be useful, and searching too specifically for artifacts on specific hosts may produce fewer results than may be useful.

#2. Clustering

Clustering is a statistical technique, often carried out with machine learning, that consists of separating groups (or clusters) of similar data points based on certain characteristics out of a larger set of data. Hunters may use clustering for many applications, including outlier detection, due to the fact that it can accurately find aggregate behaviors, such as an uncommon number of instances of a certain occurrence. This technique is most effective when dealing with a large group of data points that do not explicitly share immediately obvious behavioral characteristics.

#3. Grouping

Grouping consists of taking a set of multiple unique artifacts and identifying when multiple of them appear together based on specific criteria. The major difference between grouping and clustering is that in grouping your input is an explicit set of items that are already of interest. Discovered groups within these items of interest may potentially represent a tool or a TTP that an attacker might be using. An important aspect of using this technique consists of determining the specific criteria used to group the items, such as events having occurred during a specific time window. This technique works best when you are hunting for multiple, related instances of unique artifacts, such as the case of isolating reconnaissance commands that were executed within a specific timeframe.

#4. Stack Counting

Also known as stacking, this is one of the most common techniques carried out by hunters to investigate a hypothesis. Stacking involves counting the number of occurrences for values of a particular type, and analyzing the outliers or extremes of those results. The effectiveness of this technique is generally diminished when dealing with large and/or diverse data sets, but it is most effective with a thoughtfully filtered input (such as endpoints of a similar function, organizational unit, etc.). Analysts should attempt to understand input well enough to predict the volume of the output. For example, if you are given a dataset containing 100k endpoints, stack counting the contents of the Windows\Temp\ folder on each endpoint across an enterprise will produce an enormous result set. Friendly intelligence can be used to define filters for your input.

Machine Learning Techniques

In addition to standard hunting techniques like those listed above, you will find that many investigations and procedures can be enhanced and carried out using various machine learning or data science powered techniques. These techniques can involve creating frameworks of feedback given to automated classification systems. This is known as supervised machine learning, which uses labeled “training data” to condition algorithms to make predictions about unlabeled data. This new, unclassified data is what you want the machine to label correctly (based on the training data).

It’s not an absolute necessity that you be able to leverage machine learning techniques in your hunting, but you should be aware of the role they can play, as you will see them referenced in many places including in the rest of this guide. The best hunting tools, such as Sqrrl, should be able to provide you with prebuilt machine learning techniques you can leverage as part of an investigation workflow.

Datasets

The techniques that you use are only a part of planning out your hunt and knowing what you can have at your disposal. You can’t hunt if you don’t have the right data, but what is the right data? The answer to that question will depend on what you’re looking for, but below is a general list of datasets that lend themselves well to hunting and security activities in general

Endpoint Data

Process execution metadata	Contains information on processes run on specific hosts. Critical metadata associated with process execution includes commandline commands/arguments and process filenames and ID.
Registry access data	Contains data related to registry objects, including key and value metadata.
File data	Information on stored files and artifacts kept on a local host. This can include when files were created or modified, as well as size, type, and storage location information.
Network data	Identification of the parent process for a network connection.
File prevalence	Information on how common a file is in your environment.

Network Data

Network session data	Contains information on network connections between hosts. Critical metadata associated with network connections including the source IP address, destination IP address, destination port, start time of the connection, and end time/duration of the connection. This includes Netflow, IPFIX, and similar data sources.
Bro logs	A widely recommended network monitoring tool that collects connection-based flow data and application protocol metadata (HTTP, DNS, SMTP), specialized for security application.
Proxy logs	HTTP data that contains information on outgoing web requests, including Internet resources that internal clients are accessing.
DNS logs	Contains data related to DNS domain resolution activity, including domain-to-IP address mappings and identification of internal clients making resolution requests.
Firewall logs	Connection data that contains information on network traffic at the border of a network, focused on blocked connections.
Switch and Router logs	Internal netflow, also known as east/west traffic, in your environment that shows what is going on inside the network behind your perimeter security

Security Data

Threat Intelligence	A broad category of information that includes the indicators and TTPs used by attackers, as well as the operations and campaigns they carry out.
Alerts	The automated warnings or notifications created by correlation engine tools like a SIEM or IDS, indicating that a given rule set was violated or certain pattern identified, which might indicate a potential incident.
Friendly Intelligence	Another broad category of information about an organization’s own IT infrastructure, security ecosystem, critical assets, employee information, and business processes. Friendly intel helps hunters orient and understand the environment in which they are hunting and contextualize their investigations.

Walkthrough: Hunting for Command &
Control

By now you’re determined how to map out how you’re going to be carrying out your hunts and what you’ll be hunting for, and figured out what techniques and resources are at your disposal. It’s time to finally dive in and start finding evil. Walkthrough: Hunting for Command & Control CHAPTER 8

Even with all this information, the prospect of hunting might still be a little daunting. Perhaps the most effective way of learning how to actually hunt is to learn from examples of applied hunts. In the next two sections, you’ll see 2 example hunt walkthroughs looking for 2 different adversary techniques that you can look through for guidance. Give these a try on your own! You should now have all the information you need at your disposal.

Hunting for Command & Control

Command & Control (C2) Overview

Attackers generally build Command and Control (C2) channels into common protocols (ComPro) or custom protocols (CusPro). This enables remote access for attackers into target networks. A few examples of common protocols include HTTP/S, SSL/TLS, or DNS. Custom protocols are harder to predict, but include techniques such as encrypting packet data with an XOR cipher. Just like with protocols, attackers generally use common network ports (ComPor) or uncommon network ports (UncPor) for their C2 channels. Examples of standard ports include 80/TCP (HTTP), 443/TCP (SSL/ TLS), 53/UDP (DNS). Uncommon ports are difficult to predict, but they typically deviate from ports registered with IANA. Attackers can use any combination of protocols and ports, including:

• Common Protocol + Common Port
• Common Protocol + Uncommon Port
• Custom Protocol + Common Port
• Custom Protocol + Uncommon Port

Example Hypothesis

Hypothesis

Attackers may be operating on a C2 channel that uses a common protocol on a common network port

• Look for unique artifacts pertinent to the protocol you are interested in. For example, if you are interested in identifying C2 in HTTP traffic, then you might consider looking for anomalous domains/URLs/User-Agent strings

Datasets to Explore

Datasets used to hunt for C2 depends on what you are hunting for. For identifying use of custom protocols, focus primarily on network session metadata, including:

• Netflow (“flow” data in general)
• Firewall logs (should log allowed / accepted packets)
• Bro Conn log

To identify use of common protocols, as in this example, focus on application protocol metadata,
including:

• Proxy logs, IIS logs
• DNS resolution logs
• Bro HTTP, SSL, DNS, SMTP logs

Techniques to Use

Indicator Search

The value of this approach will be impacted by the value of the indicator. Locally sourced indicators will generally provide a high value because they tend to be timely and relevant to the network or systems you might be trying to protect. These can be gathered from previous incidents or by internal threat intelligence teams.

Common network session indicators: IP address, Port
Common application protocol indicators: Domain (HTTP, DNS, SSL), URL (HTTP), User-Agent String (HTTP), X.509 Certificate Subject (SSL), X.509 Certificate Issuer (SSL), Email address (SMTP)

Stacking

Stacking is a technique commonly used in many different kinds of hunts. In the case of hunting for command and control activity, a hunter will want to stack for anomalous instances of inbound or outbound traffic. Metadata types that can be used for stacking include: Ports, URLs, X.509 Certs

Machine Learning - Binary Classification

This involves using machine learning to isolate malicious C2 activity. Supervised machine learning uses labeled training data to make predictions about unlabeled data. Given a set of known good and known bad examples, you can create a binary classifier capable of taking in new transactions and deciding if they look more similar to the good training set or the bad training set. After the classifier is trained, you can feed your HTTP (or other network logs) through it and get back much smaller set of records that require analyst attention.

Example Hunt

Hunting for Command & Control

1. What are you looking for? (Hypothesis)	Hypothesis: Attackers may be operating on a C2 channel that uses custom encryption (uncommon protocol) on a common network port Look for: Anomalies in monitored network port channels, i.e. connections that do not have protocol artifacts related to the common port you are looking at. For example, look for connections that have no identifiable HTTP metadata over port 80/TCP
2. Investigation (Data)	Determine what datasets you are using: For identifying use of common protocols, you will want to focus primarily on application protocol metadata, including: Proxy logs, IIS logs DNS resolution logs Bro HTTP, SSL, DNS, SMTP logs
3. Uncover Patterns and IOCs (Techniques)	1. Use a search to identify legitimate protocol connections on a common port you will be inspecting, by looking at protocol metadata If looking at port 80, search for any HTTP protocol records that exist for a given time period 2. Use a second search to identify all network session metadata (e.g., Netflow, Firewall, etc.) on the common port for the same time period used in step 1 3. Using the output of steps 1 and 2, remove the legitimate protocol connections from the session data. This should leave uncommon protocol connections on the common port 4. Take the results of step 3 and stack the data for what is useful to investigating your hypothesis For example: destination IP, bytes transferred, connection duration/length, etc.
4. . Inform and Enrich Analytics (Takeaways)	The destination IP addresses involved in the C2 activity you have discovered can be taken as IOCs and added to an indicator database in order to expand automated detection systems. You can also create packet-level signatures to trigger alerts for cases where the custom protocol you have discovered may appear again.

Walkthrough: Hunting for Internal Reconnaissance

Internal reconnaissance belongs to the 7th and final step of the kill chain: Act on Objectives. Internal reconnaissance is the process of collecting internal information about a target network, so that an attacker can more effectively move through the network and conduct further activities

Process enumeration

After gaining access to a host or network, an attacker will use this process to attempt to establish what processes are running on the local host and the surrounding hosts. The commands used by attackers for process enumeration will depend on whether the attacker is looking for specific services (i.e. critical processes that run at startup and in the background) or general processes. Specifically on Windows, commands that an attacker might use for identifying services, running services, and scheduled processes include but are not limited to:

Identifying Scheduled Processes	Identifying Running Processes	Identifying Scheduled Processes
net start	tasklist	at
sc query	Get-Process (Powershell)	schtasks /query
gsv (PowerShell)	gps (PowerShell)	Get-ScheduledTask (PowerShell)
Get-Service (Powershell)	process (WMIC)	Get-ScheduledJob (Powershell)
service (WMIC)	process (WMIC)	job (WMIC)

Datasets to explore

For internal reconnaissance, there are two major data types that are useful in a hunt: process execution metadata and network connection metadata. In this context, critical metadata associated with process execution includes command-line commands/arguments and process filenames. This metadata should include the name of the host that the process was executed on and the name of the user who executed the process. Critical metadata associated with network connections include the source IP address, destination IP address, destination port, start time of the connection, and end time/duration of connections. For hunting network enumeration with this type of metadata, it’s best to have data that includes internal-to-internal connections between hosts on a local subnet.

Process Execution Data Tools	Network Connection Data Tools
Sysmon	Bro
PowerShell auditing	Netflow
Process creation auditing

Techniques to Use

#1. Searching

Searching for internal reconnaissance commands and patterns can be useful if the search includes thoughtful filtering, especially based on friendly intelligence. To make this technique effective at finding internal reconnaissance, it’s best to have an explicit goal in mind such as searching for command execution of ‘whoami’ on across a particular class of workstations that should not normally execute the command (e.g., C-suite laptops).

#2. Grouping

Grouping for internal reconnaissance commands is similar to searching, except you can review multiple artifacts across multiple assets in one result set. It’s valuable to take commands related to a specific architecture (e.g., Windows), put them into a single group, and look for the execution of the group on a single asset. This technique works best when you are hunting for multiple, related instances of unique artifacts.

#3. Visualizations

Multiple visualizations may be applied to hunting for internal reconnaissance, but for this example, we will focus on one: box plots. Box plots visually describe distribution of data, with a box that represents median values and whiskers that represent high and low values (outliers). It may be useful to visualize the frequency and variety of command execution across hosts, as well as command execution across hosts across time.

Above is a box plot of the number of recon commands executed by workstations and servers. There are 17 hosts in the dataset and each host may have executed up to 8 unique commands related to internal reconnaissance. The three potential points of interest are the two workstation upper outliers and the one server upper outlier.

That's all for threat Hunting .

✨ Enjoyed this article?
We’d love to hear your thoughts! 💬
Stay connected with us on social media for more cybersecurity tips, tutorials, and updates. 🚀

👉 Follow us: [Instagram] | [Twitter/X] | [Telegram] | [Facebook]
🛒 Explore more: [Codelivly Store] – grab exclusive books & guides to level up your hacking skills.