Hunting with Jupyter Notebooks
This documentation describes the tools and workflows that enable threat hunting operations using Jupyter Notebooks.
Secureworks uses Jupyter Notebooks to automate threat hunting procedures. Threat hunting procedures implemented as Jupyter Notebooks can:
- Document the intent and methodology of the hunting procedures
- Fetch evidence, such as alerts and events, from Taegis™
- Manipulate the data into a format that is conducive for analysis
- Present the data to the threat hunter for review
- Allow threat hunters to document findings during analysis
- Escalate relevant information as Taegis™ XDR investigations
Jupyter Notebook Integration ⫘
Secureworks released several open-source projects to integrate Taegis™ and Jupyter notebooks. These tools drive our threat hunting workflows:
|Taegis™ SDK for Python
|Low-level Python client to call Taegis™ GraphQL APIs
|Convenience utilities to interact with Taegis™ from the command line and Jupyter Notebooks
Taegis™ SDK for Python ⫘
The Taegis™ SDK for Python is the official Python client to call Taegis™ GraphQL APIs.
Taegis™ SDK for Python is code generated from the GraphQL schema. GraphQL operations and complex types are represented as Python dataclasses. The static analysis features of Jupyter Notebooks, such as tab and auto-completion, natively integrate with
dataclasses making it easy to browse available operations and construct input arguments from the notebook.
While a deep dive into the fundamentals of GraphQL is outside the scope of this document, it is helpful to understand how GraphQL operations and types map to Python objects in the Taegis™ SDK for Python:
- Each Taegis™ API exposes GraphQL operations that an authorized client is allowed to call. These operations include queries (read), mutations (write), and subscriptions (a protocol for bidirectional communication between the client and server over web sockets).
- GraphQL allows API developers to define types, which are data structures that are used as both arguments to, and return values from, operations. GraphQL queries allow the client to request specific fields on the return type; this improves performance and reduces the number of round-trip API calls required for the client.
- GraphQL APIs are self-describing. Metadata about all of the available GraphQL operations, types, and fields is defined in a schema and is exposed via introspection queries.
In the Taegis™ SDK for Python, each service module represents a logical grouping of GraphQL types and operations based on functionality. For example, the alerts service is a Python module that contains the queries, mutations, subscriptions, and types related to Taegis™ XDR alerts.
See the Taegis™ SDK for Python documentation for further details.
Taegis™ Magic Jupyter Integration ⫘
Taegis™ Magic Jupyter Integration provides a convenience layer to interact with Taegis™ APIs on the command line and from Jupyter Notebooks. Taegis™ Magic alleviates the need for notebook users to implement boilerplate logic, such as pagination, when trying to perform routine tasks. Taegis™ Magic is implemented as an IPython Magic, which is a special kind of macro supported in Jupyter Notebooks.
When Taegis™ Magic is used as a standalone command-line tool, results are returned to the console as JSON.
When invoked from a Jupyter Notebook, results are returned as
pandas DataFrame is a powerful two-dimensional data structure for manipulating and analyzing data in Python.
Returning Taegis™ query results as
pandas DataFrames allows hunters to quickly dissect the data to find evidence of a threat.
Taegis™ Magic can automatically construct XDR investigations while using a Jupyter Notebook.
XDR investigations are a central part of the XDR analysis workflow and data model.
They allow analysts or automated systems to organize relevant security information, such as events, alerts, assets, search queries, and key findings.
XDR investigations are the primary medium to communicate findings and collaborate towards the resolution of a security incident.
We leverage Jupyter Notebooks' native support for rich media output, such as markdown and HTML, to populate the key findings section of XDR investigations.
The Taegis™ Magic Jupyter Integration allows notebook users to stage evidence in an XDR investigation based on resource identifiers found inside
See the Taegis™ Magic Jupyter Integration documentation for further details.
Hunting Workflows ⫘
The following sections describe how we use Jupyter Notebooks to support common threat hunting workflows.
Exploratory Data Analysis (EDA) ⫘
Exploratory analysis is an ad hoc workflow when threat hunters rapidly iterate to test their hypotheses about how to find evidence of some threat in available data.
Threat hunters use Taegis™ Magic from Jupyter Notebooks to query for relevant data, then analyze results as
The relevant information can be escalated as an XDR investigation.
- Load Taegis™ Magics notebook extension.
- Query for data.
- Analyze results.
- Stage evidence for an investigation.
- Create investigation.
Automating Hunting Procedures ⫘
If a threat hunter identifies a useful way to find evidence of some threat from the EDA workflow, then they can formalize the steps taken as a threat hunting procedure. In this context, a threat hunting procedure is a Jupyter Notebook that is intended to be reused and shared with other analysts.
Unlike a notebook used for free-form EDA, a threat hunting procedure notebook should contain canned language in markdown to help future threat hunters and customers understand the methodology. It should also include additional metadata to help organize the procedure in a catalog of hunting procedures managed under version control. See the Metadata section below for more information about suggested notebook metadata for threat hunting purposes.
Once a threat hunting procedure has been formalized in a Jupyter Notebook and committed to version control, other analysts can instantiate a copy of that notebook to repeat the procedure.
- Create new notebook.
- Add notebook metadata to catalog hunting procedure.
- Add markdown cells to create desired report structure.
- Add code cells to fetch and analyze relevant evidence.
- Add code cells to automate investigation creation.
- Commit notebook under version control.
Hunting at Scale ⫘
Hunting Fan Out
We treat threat hunting procedure notebooks as discrete units of work, usually written to function on a per-tenant basis. But we are able to run notebooks at scale across many tenants by taking a fan-out/fan-in approach.
We run a hunting procedure across many individual tenants, then review them in aggregate. Taegis™ Magic enables hunting at scale through its ability to cache query results. Once the threat hunting procedure notebook has finished executing across the tenants in scope, threat hunters can parse the cached query results. For tenants that do not have any evidence related to the threat, we can automatically create a null findings investigation to show our work even though no threats were identified. For tenants that have interesting search results related to the threat, we then assign it to a threat hunter for manual review. The threat hunter can simply open the already-executed copy of the notebook containing the cached query results.
- Select target tenants.
- Select hunting notebook.
- Execute notebook in parallel across tenants.
- Review results in aggregate.
- Assign to threat hunters for human analysis and/or create null findings investigations.
Managing Hunting Procedures ⫘
Version Control ⫘
We recommend storing threat hunting procedure notebooks under version control, such as
Version control systems provide a plethora of benefits for threat hunting procedure notebooks, such as allowing multiple hunters to share and collaborate on content and tracking change history for a notebook over time.
Many popular hosted version control systems, such as GitHub and Gitlab, also provide essential workflow tools in the form of issue tracking and peer review mechanisms.
Jupyter Notebooks can be challenging to store under version control because they are JSON documents, which are difficult to compare line-by-line.
Some popular hosted version control systems have special support for notebook files to address these inconveniences.
Alternatively, you can use tools like
jupytext to convert notebooks into a structured markdown document format.
jupytext to store all threat hunting notebooks in
MyST markdown format.
Linting is a form of static analysis that reads source code and identifies potential problems, such as syntax and formatting errors.
In the context of threat hunting notebooks, linting is extremely useful to ensure that threat hunting procedure notebooks contain necessary metadata, markdown content, and code content.
pytest as a CI job to enforce conventions and quality standards across threat hunting notebooks in a given project.
In our experience, different threat hunting teams have unique needs and preferences, so a one-size-fits-all approach will likely meet resistance.
It is helpful to write general purpose
pytest fixtures to read notebooks, then ask each hunting team to implement their own linting rules.
Jupyter Notebooks contain multiple kinds of metadata. The official Jupyter Notebook specification defines several required metadata fields, which determine the behavior of the notebook and how it is rendered in the browser. Jupyter Notebooks also support arbitrary JSON objects in the metdata for custom use cases.
In the context of threat hunting procedures, we use Jupyter Notebook metadata:
- To organize and categorize the repository of notebooks based on their content
- To determine how a given cell should be rendered in the resulting XDR investigation
Notebook-Level Metadata ⫘
Required Notebook Metadata ⫘
All Jupyter Notebooks must contain
This metadata tells Jupyter which kernel to use when evaluating code cells.
Kernels correspond to a programming language, such as Python, and a runtime environment where necessary dependencies are located.
We recommend curating a shared Jupyter kernel for threat hunting purposes.
This kernel should contain the
pandas, and any other Python packages commonly used in threat hunting procedures.
Hunting-Specific Notebook Metadata ⫘
Beyond the required metadata, we recommend the following notebook-level metadata for each threat hunting procedure notebook:
- A unique identifier, such as a UUID
- A title for the hunting procedure
- A short description of the threat that the notebook is intended to hunt for
- A list of relevant MITRE ATT&CK technique IDs
- A list of data sources or other dependencies to perform the hunt
- A list of arbitrary string tags for custom labels
This information is useful when threat hunters need to search through a repository containing many threat hunting procedures.
Cell-Level Metadata ⫘
Each cell in a Jupyter notebook has its own metadata. The notebook interface is typically responsible for setting the required metadata fields. Cell metadata is used to determine how the browser should render the cell.
The most common kind of cell metadata that is relevant to users are cell
Tags are a JSON array of arbitrary string labels.
The notebook web interface allows users to conveniently add and remove cell tags from web GUI.
In the context of threat hunting, there are two categories of custom tags that we recommend using:
- Papermill Parameters: The
papermillmodule looks for a cell tagged with
parametersto read default values when parameterizing a notebook. When overriding the default parameter values at runtime, a new cell containing the injected parameters will be appended immediately following the cell tagged as
parameters. If a notebook does not contain a cell with the
parameterstag, then the injected parameters cell will be prepended to the start of the notebook.
- Export Tags: The
nbconvertmodule uses cell tags when exporting a notebook into other formats, such as markdown. The
TagRemovePreprocessoris responsible for parsing the notebook and determining which parts of a cell should appear in the exported output text. While the
TagRemovePreprocessorcan work with arbitrary—or localized—tags, the following tags are conventional:
|Removes the code input from the cell in the export
|Removes the code output from the cell in the export
|Removes both inputs and outputs from the export
remove_cell tag is commonly used to display guidance that is only intended for the threat hunter and not the customer.
This guidance is removed when exporting the notebook as the key findings section of an XDR investigation.
The following is an example of using the
nbconvert CLI to export a notebook without any cells tagged with
jupyter nbconvert --to markdown --no-input my-hunting-notebook.ipynb --TagRemovePreprocessor.enabled=True --TagRemovePreprocessor.remove_cell_tags remove_cell
Markdown Cells ⫘
Jupyter Notebooks natively support rendering HTML and markdown text. This feature allows notebooks to mix human-readable language with programmatically executable code.
In the context of threat hunting notebooks, the markdown cells are used for two important purposes:
- To provide instruction to analysts who are running the notebook
- To populate the key findings section of an XDR investigation
The markdown content of a hunting notebook can be exported and used to populate the key findings section of an XDR investigation. Therefore, we include canned language that describes the what, why, and how of the threat hunting procedure as markdown text. We generally structure hunting notebooks into the following sections using markdown headings:
|Identifies the threat hunting notebook or procedure
|High-level description of the threat and the performed hunting procedures
|Technical description of the performed hunting procedures
|Written by analysts upon completing hunting procedures
|Suggestions for remediation guidance, prevention, etc.
The structure and content of the markdown text should be adapted for the specific procedure and hunting use case.
For more information about XDR Investigation best practices, see our Knowledge Base article series.
Code Cells ⫘
Jupyter Notebooks can evaluate and execute code for many interpreted programming languages, the most popular of which is Python.
In the context of threat hunting notebooks, the code cells contain the automation for threat hunting procedures. These procedures typically involve:
- Importing any dependencies required to perform the threat hunting procedure
- Fetching data from a remote system, such as calling a Taegis™ API
- Transforming results into a format that is convenient for analysis
- Presenting the data to the analyst for ad hoc analysis
- Selecting data of interest for inclusion in an XDR investigation
- Creating an XDR investigation to document the threat hunting procedure and findings