Threat Hunting — Practical Guide (Part 1) | Codelivly

Part 1 of "Hunt Evil": foundational threat-hunting concepts, triage workflows, key data sources, and essential tools to start detecting advanced network threats. Threat Hunting — Practical Guide (Part 1)

You might have heard a lot of buzz around this topic of “Threat Hunting” and want to try your hand at proactive detection. Great! But how does one actually go about building a hunting program?

To begin, let’s clarify what threat hunting is: Threat hunting is the human-driven, proactive and iterative search through networks, endpoints, or datasets in order to detect malicious, suspicious, or risky activities that have evaded detection by existing automated tools.

While there are a number of great resources available about what hunting is and how it can assist you, it might be challenging to cross over from the realm of the theoretical into the practical. As any hunter will tell you, orientation and planning is one of the critical aspects of effective threat hunting. This guide will help you orient and plan by laying out some basic tips and instructions on how to direct your hunting activities. It will also give you direction on how to practically carry them out using a variety of hunting techniques.

Threat hunting has been around for a while, but it has only recently become a focus of modern enterprise Security Operation Centers (SOCs). Hunting can revolutionize the threat detection efforts of an organization, and many have already recognized that proactive hunting needs to play a role in their overall detection practices (a common mantra one often hears is “prevention is ideal but detection is a must”). According to a recent survey on threat hunting conducted by the SANS institute, 91% of organizations report improvements in speed and accuracy of response due to threat hunting. It’s clearly worth your time, but it’s also worth knowing what exactly you’re investing in. Before going any further, let’s take a look at 3 common myths about hunting that will help clarify what it is.

3 Common Myths About Hunting

#1. Hunting can be fully automated

Hunting is not a reactive activity. If the main human input in a hunt is remediating the result of something that a tool automatically found, you are being reactive and not proactive. You are resolving an identified potential incident, which is a critically important practice in a SOC, but not hunting.

Hunting requires the input of a human analyst and is about proactive, hypothesis-based investigations. The purpose of hunting is specifically to find what is missed by your automated reactive alerting systems. An alert from an automated tool can certainly give you a starting point for an investigation or inform a hypothesis, but an analyst should work through an investigation to understand and expand on the context of what was found to really get the full value of hunting. To put this another way, hunters are the network security equivalent of beat cops; they search for anomalies by patrolling through data, rather than investigating a call in from dispatch.

#2. Hunting can only be carried out with vast quantities of data and a stack of advanced tools

Though it may seem like a new term, security analysts across a variety of sectors have been hunting for years. Basic hunting techniques can still be very useful and effective in helping you find the bad guys (e.g. you can perform basic outlier analysis, or “stack counting”, in Microsoft Excel). An analyst who wants to begin threat hunting should not hesitate to dive into some of the basic techniques with just simple data sets and tools. Take advantage of low hanging fruit!

Of course, having purpose-built tools like a Threat Hunting Platform can help you hunt at scale and simplify the more advanced hunt procedures. Sqrrl’s Threat Hunting Platform has been specially created to make the process of fusing different data sets together and leveraging more advanced techniques significantly more simple.

#3. Hunting is only for elite analysts; only the security 1% with years of experience can do it

As you’ll learn, there are many different hunting techniques that have differing levels of complexity. However, not all these techniques take years to master. Many of the same analysis techniques used for incident response and alert investigation and triage can also be leveraged for hunting. The key to getting started is simply knowing what questions to ask, and digging into the datasets related to them. You learn to hunt by doing it, so if you’re an analyst who has never hunted before, don’t be afraid to dive in.

It is also important to keep in mind that successful hunting is tied to capabilities
in three different areas:

Now let’s dive into the first and arguably most important category, your hunting process, and take a look at what should be your first step in creating or expanding your hunting capabilities: determining your hunting maturity

Determining Your Hunting Maturity

As mentioned, there are many different kinds of techniques and practices you
can pursue in hunting. Your hunting maturity is a measure of what kinds of
techniques and data you can work with. To help assess your current hunting
capabilities and determine how you should be aiming to grow them, we’ve
developed the Hunting Maturity Model (HMM).

The Hunting Maturity Model describes five levels of an organization’s proactive detection capability. Each level of maturity corresponds to how effectively an organization can hunt based on the data they collect, their ability to follow and create data analysis procedures (DAP), and their level of hunting automation. The HMM can be used by analysts and managers to measure current maturity and provide a roadmap for improvement. Often these improvements focus on a combination of tools, processes, and personnel.

If you want to determine your current level of hunting maturity, below is a list of questions you can answer to find out. You can then take your maturity level and align it to our suggestions about where you should be focusing your efforts next.

Based on Your Results, What Improvements Should You Focus On?

Getting Started

If you are at this stage, focus on building a set of core security capabilities by establishing a formal SOC:

• Acquire an automated detection system (SIEM, IDS, etc.)
• Create a centralized logging system and start collecting logs (e.g. web proxy, firewall, switches, routers, Bro logs, host endpoint alerts, event logs, AD logs, etc)
• Establish a specialized incident response team (even if it is only a single analyst) which can perform alert resolution and incident investigation
• Acquire external signature feeds and intel feeds that can compliment your automated detection

At HM0 > Next step: Move to begin hunting

If you have a basic automated detection system and an analyst or team who can perform incident
response, you are ready to begin to build out an effective hunting program. Focus on these points to
grow your maturity:

• Good data is the basis of good hunting, ensure regular collection of at least some security data, at least one sources from each data domain (network, host and application)
• Analysts should practice some basic hunting, such as searching for key indicators to find threats in specific datasets

At HM1 > Next step: Develop ability to identify and carry out existing hunting procedures

At HM1, you can do some basic searching and hunting. This is a great start. To move to the next level of
hunting maturity, focus on these points:

• Find and identify published hunting procedures you want to carry out on your network (If you’re not sure where to begin, we recommend threathunting.net)
• Increase the scale of your data collection to include input data required to carry out published hunting procedures that you want to pursue
• Develop a schedule for applying these procedures on a regular basis

At HM2 > Next step: Develop ability to create new procedures

At HM2, you can follow procedures that are outlined and created in other places, which can be highly
useful for protecting your organization against threats you have prioritized. The primary focus for
moving to HM3 should be:

• Create a hunt team that includes both security and data analysis expertise, which can understand and apply a variety of different types of data analysis and hunting techniques
• Begin crafting new hunting procedures based on the security concerns of your organization and the threats that you have seen in the past

At HM3 > Next step: Develop automation for current procedures

A SOC at the “Innovative” level is one that is hunting in an advanced way. Consider sharing the tested and true hunts you develop with the hunting community! To move to the final level of hunting maturity:

• Create a process for fully automating the successful hunting procedures you develop. This will ensure that your hunters are not wasting time by repeating hunts,
  but always finding new things to hunt for.

At HM4 > Next step: Onward and upward!

If you are at HM4, you are running a cutting edge hunting program. From here, your focus is ever onward and upward, especially in growing your team in scale and efficiency, as well as expanding to meet the growing security concerns and needs of your organization.

Of course, the Hunting Maturity Model is just a prescriptive model, and many organizations will sometimes be at varying levels of capabilities: excelling at some criteria and less advanced in others. For example, you might have a tremendous log collection capability, but you might not be utilizing any hunting procedures. This is OK, you still have taken care of some critical aspects of hunting! Just try to focus on shoring up the areas that you are lacking in before you try to expand your other capabilities. If you’re just a fledgling hunting program, fear not! There’s a lot of room to grow and every step will provide you with tangible, actionable benefits!

You now have a better idea of where you stand in the grand scheme of hunting organizations. Now let’s take a look at how you can evaluate the hunting practices that you’re carrying out or will soon be developing, with some basic metrics for measuring performance.

Metrics for Measuring Your Hunting Success

Clearly, hunting has its merits, but it’s not magic. It’s a process that requires people, technology and knowledge.

Key Metrics	Why It’s Important / What to Look For
1. Number of incidents by severity	You will never be able to know for certain how many incidents are lurking in your network until you find them, but ultimately keeping track of the rate at which you find incidents is a worthy metric to maintain context.
2. Number of compromised hosts by severity	Measuring the trend of how many hosts are discovered as compromised over time can help orient analysts to the state of endpoint security on their network. This can include hosts that have had misconfigured security settings on them.
3. Dwell time of any incidents discovered	Whenever possible, try to determine how long discovered threats have been active on your network. This can help you determine if there are steps of the kill chain (or other attack model) you may be focusing on too much. Dwell time has 3 metrics: time from infection until detection, time from detection to investigation, and time from investigation to remediation.
4. Number of detection gaps filled	One high-level goal of hunting is to create new automated detections -- identifying and filling detection gaps should be part of the team’s mission
5. Logging gaps identified and corrected	Gaps in logging or data collection can make it difficult for a SOC to maintain awareness and context, so trying to identify and improve any existing gaps should be an important actionable metric for a hunt team.
6. Vulnerabilities identified	Vulnerabilities can lead to exploitation and exploitation can lead to compromise -- in other words, identifying vulnerabilities is important. It’s always useful to keep track of how many of these you are uncovering.
7. Insecure practices identified and corrected	Insecure practices can lead to unauthorized access and unauthorized access can lead to incidents -- identifying insecure practices can prevent future incidents.
8. Number of hunts transitioned to new analytics	Since you want to create new automated detections, your team should try to transition each hunt into automated detection. Ideally you would want the ratio here to be 1:1. For every successful hunt you carry out you should be attempt to create a new analytic, update a rule, or at least log a new IoC.
9. False positive rate of transitioned hunts	Once you discover a successful way to find something and create a rule or analytic to automate that process, it is useful to keep track of how many false positives have been created by those automated analytics, to see if they require improvements.
10. Any new visibility gained	In addition to discovering an incident and creating new threat intel, a hunt can inform analysts about their own networks, including misconfigurations, and identify friendly intelligence that can be highly useful in future investigations.

Impress the C-suite

Keeping track of these kinds of metrics is one of the major benefits of hunting from a managerial standpoint. Instead of having to wait until your organization is compromised before receiving more resources to grow your security team, hunting can help you show actionable improvements in initiatives that you undertake, as well as create a real, tangible profiles of threats that your organization might be facing. Some metrics, such as number of compromised hosts found and vulnerabilities identified, are critical pieces of information that should be compelling to any executive or board member.

It’s important to note that when you start hunting for the first time, some of these metrics, like number of compromised hosts, might initially be very high. This is a kind of “start up bump” that all hunting programs will run into, because no previous proactive detection efforts have been undertaken. Although it may be overwhelming or seem problematic when looking at these metrics over time, you are finding new incidents you were not finding before but were always there anyway. This is decidedly a good thing

Determining What to Hunt For and How Often

So you have now determined what your hunting maturity level is and what metrics to use to chart your success, but how do you decide what to actually hunt for? While hunting approaches will vary from company to company, the three steps laid out below can help you kickstart your hunting program.

#1. Choose Your Favorite Attack Model

There are many kinds of bad guys out there that may be trying to attack you for a variety of reasons, but the general progression of how most attacks are carried out tend to share many common elements. These are mapped out in what is known as the Cyber Kill Chain. There are several variations of the kill chain, all of which define what actions adversaries must complete in order to achieve their objective while operating within an enterprise network.

The kill chain will help you identity TTPs and attacker behaviors that you should hunt for. For
this example, we will select and use MITRE’s ATT&CK lifecycle.

#2. Identify Most Concerning Activities

After selecting a model, the next step is to go through each of the phases in the model and identify attacker activities that you are most concerned with. Each phase in a model can include multiple categories of higher level tactics that an adversary might employ, which can then be broken down to a number of actual attacker activities, which you will hunt for

For example, the later stages (Control, Maintain, and Execute) of MITRE’s seven-stage
ATT&CK lifecycle include categories like lateral movement and data exfiltration, under which many kinds of activities can exist. Here’s an example list of potential attacker activities and techniques you might identify:

• Malware Beaconing
• DLL Injection
• Pass the Hash (PtH)
• Shared Webroot
• DNS Tunneling

Make sure you are considering activities that are specific to your network environment and
which assets you suspect an attacker would attempt to target. For example, a manufacturer may list potential attacker activities that are specific to their industrial control systems. You should also aim to try and hunt for TTPs you have previously not found, not the ones you canalready detect. Leave those to your automated detection systems.

#3. Build Your Threat Hunting Calendar

After creating a prioritized list of activities for each phase, the next step is to create your
hunting calendar and set a cadence for the frequency of your hunts. It’s important to start near the end of the kill chain, as these are the point where the attacker is about to achieve their objective. That means you want to stop those absolutely. Organize each of the various phases by low, medium and high impact activity.

For example, the rightmost stages of MITRE’s seven-stage ATT&CK lifecycle (Control, Maintain, and Execute) can be considered High Impact activity. The higher impact activity the more that it should be hunted for on a more regular basis. Here’s an example of weekly hunting sprints over two months:

Month 1

• Two weeks hunting High Impact Activity
• Two weeks Medium Impact Activity

Month 2

• Two weeks on High Impact Activity
• One week on Medium Impact Activity
• One week predicting attacks

Once you have that all down, you will be able to determine what your hunting schedule is actually going to look like.

The Hunting Loop

Once you determine how often you want to hunt and what you want to hunt for, you’re just about ready to actually begin hunting. But while the above model can help you formulate how to determine how often you will hunt, what is that practical process of carrying out a hunt in itself?

In order to answer this question, Sqrrl has developed the Threat Hunting Loop, which can guide an analyst in the tactical implementation of a hunt. We have written about this in a number of other places, so we won’t go as in depth on it here. However, since a great deal of the hunting content we’ll cover aligns to this loop, we’ll briefly explain it. The Hunting Loop breaks down into four steps:

In order to answer this
question, Sqrrl has developed
the Threat Hunting Loop,
which can guide an analyst in
the tactical implementation of
a hunt. We have written about
this in a number of other
places, so we won’t go as in
depth on it here. However,
since a great deal of the
hunting content we’ll cover
aligns to this loop, we’ll briefly
explain it.
The Hunting Loop breaks down into four steps:

1. A hunt starts with creating a hypothesis, or an educated guess, about some type of activity that might be going on in your IT environment. Hypotheses are typically formulated by analysts based on any number of factors, including friendly intelligence and threat intelligence, as well as past experiences.
2. A hunter follows up on hypotheses by investigating via various tools and techniques. We’ll discuss tools and techniques in more detail below, but in general, analysts can use these to discover new malicious patterns in their data and reconstruct complex attack paths to reveal an attacker’s Tactics, Techniques, and Procedures (TTPs).
3. Using manual techniques, tool-based workflows, or analytics, a hunter then aims to uncover the specific patterns or anomalies that might be found in an investigation. What you find in this step is a critical part of the success criteria for a hunt. Even if you don’t find an anomaly or attacker, you want to be able to rule out the presence of a particular tactic or compromise. In essence, this step functions as the “prove or disprove your hypothesis” step.
4. Finally, successful hunts form the basis for informing and enriching automated analytics. Don’t waste your team’s time doing the same hunts over and over. If you find an indicator or pattern that could recur in your environment, automate its detection so that your team can continue to focus on the next new hunt. Information from hunts can be used to improve existing detection mechanisms, which might include updating SIEM rules or detection signatures. The more you know about your own network, the better you can defend it, so it makes sense to try to record and leverage new findings as you encounter them on your hunts.

Top Considerations for Effective Tech

Now that you are starting to piece together a solid hunting process, let’s pause for a moment to talk about the kinds of tools you’ll need to use in hunting.

Although various types of tools can be used for hunting (e.g., SIEMs, purpose-built hunting platforms like Sqrrl, open source software, etc.), there are several questions that you should consider when picking a threat hunting tool. Though tools will vary greatly, there are generally 3 criteria that you should consider: how it assists in investigations, what analytics it can leverage, and how it deploys and deals with that data you’ll be hunting through.

Below is a list of questions that can help inform your requirements for and selection of a hunting tool.

Investigation Capabilities	1. Which of the standard hunting techniques does the tool generally enable you to carry out? In the next section, we will cover a list of hunting techniques that you can apply in various detection situations. It’s important that whatever tool you’re considering be able to carry out most if not all of these techniques. Otherwise, you will be limiting the techniques at your disposal. 2. How does the tool support the creation of hypotheses on which to base a hunt? You should always be creating hypotheses yourself, so this isn’t a necessity, but it’s always helpful when a tool can help you come up with where to begin a hunt with some degree of priority. 3. What ability does the tool have to import outside intelligence or custom indicators in order to assist analysts with the investigation of hypotheses? This is critical. Having guidance from intelligence feeds or even your own friendly intelligence channeled into your data can revolutionize the way that you conduct your hunts and help you confirm with greater certainty when you think you’ve found something. 4. What capabilities does the tool have that allow an analyst to pivot through different data sets? Another critically important feature that a tool should have is the ability to pivot through data. You don’t want to get caught up not being able to answer a question because the answer lies in a different part or type of the data that you can’t access. 5. How does the tool support the collection and storage of new Indicators of Compromise that might be found over the course of a hunt? What’s really going to make your hunt worthwhile is if you can take what you find in it and use it to improve your automated defenses. The first step of that is going to be exporting the indicators that you find.
Analytics Supported	6. What kind of analytics does the tool support that will help you facilitate more streamlined proactive investigation? It will be a great relief to have the backup of some analytics to assist you in finding and identifying anomalies that end up being malicious adversary activity, if not to simply add a degree of certainty to confirm a suspicion you might have. 7. Does the tool enable the creation and customization of detection analytics? Being able to create analytics will become immensely valuable as you customize your hunting process to your organization’s needs, and depending on what you’re hunting for. 8. Does the tool utilize any machine learning or data science techniques? Having machine learning capabilities isn’t an absolute necessity, but the more it has the more you will ideally be able to rely on the fidelity of what the tool finds.
Deployment & Data	9. What data sources does the tool support? You should ideally be able to pivot through any data type that you might need or an investigation might require. 10. To what extent is the tool able to scale its data storage capacity? The amount of data that your tool can look through is Important. Can it search petabyte scale levels of data in real time? If it cannot, you might find yourself limited in terms of trying to scope the full extent of a major incident. 11. Through what process does it ingest or stream data? If the tool needs to intake data in order to analyze it, it’s important to know how quickly and smoothly the process of bringing that data into it will be. 12. What integrations with other security tools does it support? Hunting is never an isolated practice. A hunting tool should integrate with automated detection systems, orchestration tools, and any preventative measures.

✅ That’s a wrap for Part 1 of our Threat Hunting series!
We’ve just scratched the surface — in Part 2, we’ll dive deeper into building hunting hypotheses, detection strategies, and real-world use cases. Stay tuned! 🔍

💡 If you found this helpful, don’t miss out on updates:

👉 Follow us: [Instagram] | [Twitter/X] | [Telegram]
🛒 Explore more: [Codelivly Store] – grab exclusive books & guides to level up your hacking skills.