FAQ: Generic Events and Normalized Data

What is changing with XDR generic events? ⫘

Secureworks® Taegis™ XDR will now only store data that is not normalized as generic events. This reduction in generic events improves performance for any XDR feature that uses generic events.

When does this change take effect? ⫘

If you have custom rules, saved searches, or reports using generic events, then the duplication of generic events will be turned off for your tenant on March 16th, 2023.

If you do not have custom rules, saved searches, or reports that rely on the generic events, then this update should not have an impact on your tenant and the duplication of data will turn off on February 16th, 2023.

What features are impacted by this change ⫘

Searching all events using the Generic schema—To search across all event data types, use @raw or original_data. If you know the specific schema that your events are normalized to, target your search at an appropriate logical type or build directed queries for the appropriate event types.

Reports—If you have reports that use generic events you must determine if additional reports against normalized datasets are needed to cover your extant use cases.

Custom Rules—If you have custom rules that relied on generic events, you must update them to take normalized events into account.

Saved Search Queries—If you have saved search queries that use from generic you must determine if additional saved searches need to be created to cover additional event types.

Does this remove existing generic events? ⫘

No, but historical duplicate generic events do age out according to the applicable retention policy.

How can I evaluate if my custom rules need to be updated? ⫘

The following search can be used to find historical custom alerts which would no longer be created after March 16th.

from alert
  metadata.creator.detector.detector_id contains 'app:event-filter-ql'
  and generic.normalized_resource_ids is not null

Each unique alert title requires a different rule to duplicate.

Open an alert and view the source event’s normalized data. Look for the JSON object normalized_resource_ids. This object contains a reference to the normalized version of this event. The normalized schema type is contained in event://priv:scwx.<normalize schema type>

Create a new custom alerting rule and replace “generic” in the query with the schema type from the normalized_resource_ids field.

The above assumes that the generic rule has been in place long enough to have triggered on to be impacted events. To perform a more holistic check use the custom rule’s query as specified in the next question.

How do I test my queries in saved search, custom rules, or reports to determine if they are impacted? ⫘

To test, copy your existing query into an advanced search and add the following criteria: normalized_resource_ids is not null. Any events that return contain a normalized_resource_ids object that lists the normalized version of the event. These generic events are the ones that will no longer be populated. You can then write new search queries using the normalized event type seen in the normalized_resource_ids object of these events.

Does this impact log retention? ⫘

No, the normalized event already contains the same data as the duplicate generic event. These events also have the same data retention policy.

How do I know if a generic event has a normalized event? ⫘

For events that occurred before data duplication was turned off, you can find out by looking under the generic event’s normalized data tab: the object normalized_resource_ids contains the event IDs. The following advanced search query returns all generic events with a normalized event counterpart: from generic normalized_resource_ids is not null. The following image shows that the generic event has a normalized Auth event:

Generic Event with Normalized Event Counterpart

Can you give an example of the duplicate data stored in generic? ⫘

As an example, network connection logs from firewalls were stored as netflow events and were duplicated as generic events as well. Going forward only the netflow event will be stored in XDR and there will not be a duplicate generic event; the netflow event will contain the original un-altered message in the original_data field.

Can you give an example of how a generic query can be accompanied by additional queries for reporting, custom rules, and saved search? ⫘

If there is a saved query or report with the following query:

from generic where original_data contains 'admin'

Alternative queries depending on the desired outcome:

• @raw contains 'admin' — Search across all schemas where original_data contains admin
• @user contains 'admin' — Search across all schemas for normalized username fields
• from auth where @user contains 'admin' — Search across only auth events for username
• from process where @user contains 'admin' — Search across only process events for username

Do normalized events already contain the raw event in the original_data field? ⫘

Yes, all normalized data also contains the original un-altered message in the original_data field. For this reason, it is no longer necessary to also keep a copy of the same message in the Generic schema. Going forward, Generic events will only store messages that are not normalized into specific schema; the @raw operator will efficiently search all original_data in all schema, including Generic events.

For more information on generic events please reference How Are Generic Events Generated.