How to resolve "Failed to infer schema... error found in byte" when inferring schema in Panther

Issue

When trying to infer a schema from a sample of log files, Panther presents the following error: Failed to infer schema: failed to parse line [1] as JSON: ReadString expects '' or n, but found , error in #1 byte of ...|{|..., bigger context ...|{|...

Resolution

• Case 1: Inferring JSON files

To resolve this issue, please ensure your log files are written in JSONL format. This means that each JSON event object must fit on a single line in the file. There are many preprocessors that offer this functionality (i.e. Cribl), but in a pinch, any computer with a bash shell can convert the files using the following steps:

1. Place all of your sample log files into a separate directory (if they aren't already).
2. Open a bash terminal (or equivalent), and change directory into the folder with the sample logs.
3. Run the following bash script:

for f in azure_*; do (cat "${f}"; echo) >> sample_logs.jsonl; done

You should now be able to use the file sample_logs.jsonl to infer a schema.

• Case 2: Inferring CSV files with headers

To resolve this issue, please ensure that you are using stream type Lines when inferring the schema. Stream type Auto will throw the above error when inferring CSV files with headers.

Cause

• Case 1: This issue is because currently Panther doesn't support parsing JSON objects which span multiple lines.
• Case 2: This issue is because Panther does not support stream type Auto for inferring CSV files with headers.