Can I parse a specific field in my JSON log with a fastmatch or regex log parser in Panther?

Last updated
Save as PDF

QUESTION

How do I pull in my CloudWatch logs and keep the extra info like logGroup and logStream while still parsing the message field with fastmatch or regex? Can I transform or choose one field of my JSON file to use as my log input?

ANSWER

There are a few ways to parse your text fields with grok patterns, like Panther does with fastmatch log parsing either with Cribl or from inside your detection code.

Cribl

Cribl's grok command can take in the log field and transform it into separate fields.

Python

You can use Python's string split method to parse the log in your detection, if your fields are separated evenly with whitespace or commas (CSV). Otherwise, we recommend using the Cribl parsing method with grok.

The following examples are designed to work with a field that is shaped like this with '-' meaning no value:

{
    "log": "2023-05-10 16:03:05:094 field1 field2 - field4 field5 is all the rest of the line pulled in greedily"
}

String Split Method

Example:

log = {}
log_fields = event.get(log).split(' ', 6)
# Reassemble the timestamp
log['timestamp'] = "{} {}".format(log_fields[0],log_fields[1])
for index in range(2,len(log_fields)):
    # Can be named individually for better naming scheme.
    field_name = "field{}".format(index)
    if log_fields[index] is "-":
        log[field_name] = None
    else:
        log[field_name] = log_fields[index]

PyGrok

You can instead create a custom global helper function using pygrok. (You will need to request this library be installed on your Panther instance by following the instructions in this article.) This will organize the return value into JSON and keep the timestamp together when supplied with the correct pattern definition.

See the full list of grok patterns here.

Example:

from pygrok import Grok

grokparse_text_field(event):
    text = event.get('log')
    pattern = '%{TIMESTAMP_ISO8601:timestamp} %{WORD:field1} %{WORD:field2} %{WORD:field3} %{NOTSPACE:field4} %{GREEDYDATA:field5}'
    grok = Grok(pattern)
    return grok.match(text)