🌙
 

Subscribe to the Taegis™ XDR Documentation RSS Feed at .

Learn more about RSS readers or RSS browser extensions.

Custom Parser Syntax

Statements

!SAMPLE=...

A sample message. Everything to the right of the = is interpreted literally, all the way up to a newline. This field is optional, but strongly encouraged.

!SCHEMA=...

This is the schema for this message type, for example scwx.nids, scwx.netflow, scwx.auth. If not specified, the schema from the parent or closest ancestor is used.

!CONFIRMWITH

This is either PATTERN or EXPRESSION. This works in tandem with !CONFIRMSTRING to determine if a message matches this parser. If set to PATTERN, then CONFIRMSTRING is a regex pattern. If set to EXPRESSION, then CONFIRMSTRING is an expression that evaluates to True/False.

!CONFIRMSTRING=

See !CONFIRMWITH.

!DISABLED=

This disables the parser. The parser is completely removed from the runtime catalog. This is useful when you don’t yet know how to handle a message but want to capture minimal documentation of its existence.

!IMPORT=

Import another parser into this parser at the current line. Variables are shared between the importing and imported parser. This allows repeating lines of parser code to be consolidated into one place.

!IMPORTONLY

Indicates that this parser is only for import (via !IMPORT). With extremely rare exceptions, all imported parsers should be !IMPORTONLY. This flag exempts the parser from many validation rules (For example, it doesn’t have to have a parent parser, no CONFIRMWITH/CONFIRMSTRING, etc.)

!TRIMALLOFF

This disables the default behavior of running TRIM_ALL() for all parsers. In some cases, this causes problems as TRIM_ALL() removes leading or trailing braces ({ and } and also [ and ]), which leads to incorrect data for Json fields.

!SANITIZEALLOFF

This disables the default behavior of running SANITIZE_ALL() for all parsers.

Functions

SPLIT(data, delimiter, makeGreedy)

Splits data into tokens separated by delimiter. For example, if (optional) makeGreedy is "true" then the data of 0,,2 with a delimiter of , is evaluated to [0,2] instead of [0,'',2].

Example

data = "aaa,bbb,ccc,eee"
values1 = SPLIT(data, ",", FALSE)
OUTPUT1$ = values1[3]

#OUTPUT1$: eee (String)
data = "aaa,bbb,ccc,,eee"
values1 = SPLIT(data, ",", FALSE)
values2 = SPLIT(data, ",", TRUE)
OUTPUT1$ = values1[3]
OUTPUT2$ = values2[3]

#OUTPUT1$: NULL (null)
#OUTPUT2$: eee (String)

SPLIT_NAME_VALUES(data, delimiter, separator, quoteChar)

Splits data into a collection of name/value pairs where delimiter separates the pairs and separator separates the name vs value. quotechar indicates the character for quoting the value.

Example

data = "User: Unknown, InitiatorPackets: 2, ResponderPackets: 1, InitiatorBytes: 120, ResponderBytes: 66"
dict = SPLIT_NAME_VALUES(data, ",", ":", "\\")
OUTPUT$ = dict["InitiatorBytes"]

# OUTPUT$: 120 (String)

JSON(data)

Converts data into a json object that can be accessed with square brackets containing a json path. See https://goessner.net/articles/JsonPath/ and https://github.com/ohler55/ojg.

Example

data= "{ \"store\": { \"book\": [ { \"category\": \"reference\", \"author\": \"Nigel Rees\", \"title\": \"Sayings of the Century\", \"price\": 8.95 }, { \"category\": \"fiction\", \"author\": \"Evelyn Waugh\", \"title\": \"Sword of Honour\", \"price\": 12.99 }, { \"category\": \"fiction\", \"author\": \"Herman Melville\", \"title\": \"Moby Dick\", \"isbn\": \"0-553-21311-3\", \"price\": 8.99 }, { \"category\": \"fiction\", \"author\": \"J.R. R. Tolkien\", \"title\": \"The Lord of the Rings\", \"isbn\": \"0-395-19395-8\", \"price\": 22.99 } ], \"bicycle\": { \"color\": \"red\", \"price\": 19.95 } } }"
json= JSON(data)
OUTPUT$ = json["$.store.book[*].author"]

# OUTPUT$: "Nigel Rees","Evelyn Waugh","Herman Melville","J. R. R. Tolkien" (String)

Example usage for JSON keys that contain dots:

data= "{ \"store\": { \"book\": [ { \"id.category\": \"reference\" } ] } }"
json= JSON(data)
OUTPUT$ = json["$.store.book[0][\"id.category\"]"]

# OUTPUT$: reference(String)

CEF(data)

Parses data as a CEF-formatted message. The header fields can be accessed with an integer and the named fields can be accessed by name.

Example

!SAMPLE=Nov 6 07:49:03 10.42.0.1 %helloWorld: CEF:0|Check Point|VPN-1 & FireWall-1|Check Point|Log|Address spoofing|Unknown|act=Drop cs3Label=Protection Type cs3=IPS

values = CEF(originalData$)
OUTPUT1$= values[2]
OUTPUT2$= values["act"]
OUTPUT3$= values["Protection Type"]

# OUTPUT1$: VPN-1 & FireWall-1 (String)
# OUTPUT2$: Drop (String)
# OUTPUT3$: IPS (String)

LEEF(data, delimiterOverride)

Parses data as a LEEF-formatted message. The header fields can be accessed with an integer and the named fields can be accessed by name. Optionally, a delimiter override may be specified. LEEF extensions should be either tab-separated or they should indicate an alternate delimiter in field 6 of the header. The override parameter should be used when you know that a device is not compliant with the standard.

DATETIME(data, fmt, handle2DigitYear)

Converts a string to a time value for fields like EventTimeUsec$. Also accepts time.Parse format strings (optional). If handle2digitYear is TRUE, an appropriate year is chosen; usually the current year with an edge case around the new year.

Example

data = "Sep 21 2018 17:35:54"
OUTPUT1$ = DATETIME(data, "Jan 02 2006 15:04:05")
OUTPUT2$ = data

# OUTPUT1$: 2018-09-21 17:35:54 +0000 UTC (time)
# OUTPUT2$: Sep 21 2018 17:35:54 (String)

IS_PRIVATE_IP(string)

Returns boolean if the passed in (IP address) string is in the private IP range. Currently only supports IPv4 and tests against the private IP ranges defined in RFC1918.

Example

data1 = "10.0.0.1"
data2 = "11.0.0.1"
OUTPUT1$ = IS_PRIVATE_IP(data1)
OUTPUT2$ = IS_PRIVATE_IP(data2)

# OUTPUT1$: true (bool)
# OUTPUT2$: false (bool)

IS_VALID_IP(string)

Returns boolean if the passed in string is a valid IP address, leveraging net.ParseIP.

Example

data1 = "10.0.0.1"
data2 = "999.255.255.255"
data3 = "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
OUTPUT1$ = IS_VALID_IP(data1)
OUTPUT2$ = IS_VALID_IP(data2)
OUTPUT3$ = IS_VALID_IP(data3)

# OUTPUT1$: true (bool)
# OUTPUT2$: false (bool)
# OUTPUT3$: true (bool)

REPLACE(data, oldString, newString)

Replaces all occurrences of oldString with newString.

Example

data = "aaaBBBaaaCCC"
OUTPUT$ = REPLACE(data, "aaa", "zzz")

# OUTPUT$: zzzBBBzzzCCC (String)

REPLACE_REGEX(data, pattern, newString)

Replaces all occurrences of oldPattern with newString.

Example

data = "aaaBBBaaaCCC"
OUTPUT$ = REPLACE_REGEX(data, "a+", "z")

# OUTPUT$: zBBBzCCC (String)

STRLEN(string)

Returns the length of the passed in string. On error, returns -1; if NULL Type passed, returns 0 (zero, no error).

Example

data = "1234567890"
OUTPUT$ = STRLEN(data)

# OUTPUT$: 10 (int)

UPPERCASE(string)

Returns the passed in string with all Unicode letters mapped to their upper case; just an interface/wrapper for strings.ToUpper().

Example

data = "aaabbbccc acme"
OUTPUT$ = UPPERCASE(data)

# OUTPUT$: AAABBBCCC ACME (String)

LOWERCASE(string)

Returns the passed in string with all Unicode letters mapped to their lower case; just an interface/wrapper for strings.ToLower().

Example

data = "AAABBBCCC ACME"
OUTPUT$ = LOWERCASE(data)

# OUTPUT$: aaabbbccc acme (String)

SANITIZE_ALL()

Cleans up null/empty values in event field variables. For example, all of these are set to null: " ", "N/A", "n/a", "null", "nil", "-". This function is run by default on all parsers unless disabled with !SANITIZEALLOFF.

Example

data = "N/A"
OUTPUT$ = data

# OUTPUT$: NULL (null)
!SANITIZEALLOFF
data = "N/A"
OUTPUT$ = data

# OUTPUT$: N/A (String)

TRIM(data)

Removes whitespace, quotes, braces etc.

Example

data = " aaa bbb bcc "
OUTPUT1$ = "---" + data
OUTPUT2$ = "---" + TRIM(data)

# OUTPUT1$: --- aaa bbb bcc (String)
# OUTPUT2$: ---aaa bbb bcc (String)

TRIM_ALL()

Removes whitespace from the beginning/end of all event field variables. This function is run by default on all parsers unless disabled with !TRIMALLOFF.

Example

!TRIMALLOFF
data = " aaa bbb bcc "
OUTPUT2$ = data

# OUTPUT2$: aaa bbb bcc (String)

ADDFIELD(collection, fieldName, fieldValues)

Adds a field to an array of objects. The values of the field for each object are specified by fieldValues (also an array). The name of the new field is specified by fieldName. If collection is NULL, a new array of objects is created, each with a single field (fieldName) with the provided values.

Example

keys = ["httpSourceName", "httpSourceId"]
values = [json["$.httpSourceName"], json["$.httpSourceId"]]

eventMetadata$.record$ = ADDFIELD(NULL, "key$", keys)
eventMetadata$.record$ = ADDFIELD(eventMetadata$.record$, "value$", values)

# event_metadata = {
#     "httpSourceName": json["$.httpSourceName"]
#     "httpSourceId": json["$.httpSourceId"]
# }

URL_PARSE(url, silent)

Parse a URL.

Tip

For more on working with parsing, see Creating, Editing, and Enabling a Custom Parser in Taegis™ XDR.

If silent is true, this does not throw an error in the case of the URL being invalid, and instead nulls all fields. For badly formatted URLs, it always attempts to extract as much as possible. The expected passed in URL format is one of:

scheme:opaque?query#fragment
scheme://userinfo@host/path?query#fragment

Examples

http://user:password@192.1.1.1:8080/1/asdfasdfasdf.html?key=value&key2=value2#topOfTheMorning
hTtps://Example.com:443/here//is/path.html?a=1+6&x=%2f%2Fkey=%41%0Avalue&b=ddd#top
https://example.com/foo/bar/bar/../baz.html?a=1&b=2
example.com/foo/bar/bar/../baz.html?a=1&b=2

If the scheme is not provided (for example, example.com/index.html instead of http://example.com/index.html), then http is assumed and returned in the scheme value.

The resulting collection object contains the following values, if possible, given the URL:

Examples

data = "hTtps://Example.com:443/here//is/path.html?a=1+6&x=%2f%2Fkey=%41%0Avalue&b=ddd#top"
urlParts = URL_PARSE(data, FALSE)
OUTPUT$ = urlParts["path_raw"]

# OUTPUT$: /here//is/path.html (String)

CONTAINS(string, substring)

Wraps golang's strings.Contains(string,subString), returns a bool.

Example

data     = "aaabbbccc acme"
OUTPUT1$ = CONTAINS(data, "roadrunner")
OUTPUT2$ = CONTAINS(data, "acme")

# OUTPUT1$: false    (bool)
# OUTPUT2$: true    (bool)

IDX_OF_TLD(string)

Returns an int64 that signals where in the string the top-level domain is at for indexOfTopPrivateDomain$. If -1 is returned set IsTopPrivateDomainParsed$ to false, otherwise set IsTopPrivateDomainParsed$ to true.

Example

OUTPUT0$ = IDX_OF_TLD("aaa http://example.com")
OUTPUT1$ = IDX_OF_TLD("http://example.com")
OUTPUT2$ = IDX_OF_TLD("")

# OUTPUT0$: 0    (int)
# OUTPUT1$: 0    (int)
# OUTPUT2$: -1    (int)

PARSE_ERROR(string,string)

ParseError explicitly raises an error in the parser .parameters[0] errText if coercion.EvaluateAsString() is passed. parameter[1] is an optional string that is cast into a boolean via ParserValue.BoolValue() to denote if a generic event should be created. It defaults to true if not provided. The message does not normalize to any other schema.

Examples

Creates a Generic Event
test = IF someVal != "Expected_value" THEN PARSE_ERROR("bad data received") ELSE "ok"
Doesn’t Create a Generic Event
tenantId$ = TENANT_LOOKUP("ngav_id", vals["Account"], PARSE_ERROR("Unable to find Taegis tenant id for Deep Armor account " + vals["Account"],"False"))

TENANT_LOOKUP(label, value, default)

Looks up the tenant id in Taegis Tenant Manager based on a label and a value from the message. If no tenant is found, the specified default expression is evaluated. Note that if a tenant is found, the third parameter is not evaluated. This gives the caller the option to provide a default value or to use the PARSE_ERROR() function to raise an error.

Example

tenantId$ = TENANT_LOOKUP("VendorName", messageValues["customerId"], PARSE_ERROR("Customer Id not on file"))

BASE64_DECODE(string)

Returns plain text string of a base64 encoded string input.

Example

OUTPUT$ = BASE64_DECODE("aG1lZXBcISBobWVlcFwh")

#OUTPUT$: hmeep\! hmeep\!    (String)

INT(string, base)

Returns integer of a number string with specific base

Example

OUTPUT$ = INT("4e0", 16)

#OUTPUT$: 1248    (int)

STRING(valueType)

Attempts to cast the variable input into a string representation.

# In some cases "key" can be a string, empty (NULL), an array, or even map.
key = json["$.requestParameters.key"]
# By calling STRING() you guarantee objectKey is set with a value.
objectKey$ = STRING(key)
# Note: ParserValue.StringValue() isn't used directly because addition logic breaks when appending two valuetype.OBJECT to make a list (addition operator).
# valuetype.LIST, valuetype.OBJECT, and valuetype.JSONDATA returns the json string representation all others are cast to their string analogs.

 

On this page: