How to resolve "Failed to infer schema... error found in #1 byte" when inferring schema in Panther
Issue
When trying to infer a schema from a sample of log files, Panther presents the following error: Failed to infer schema: failed to parse line [1] as JSON: ReadString expects '' or n, but found , error in #1 byte of ...|{|..., bigger context ...|{|...
Resolution
To resolve this issue, please ensure your log files are written in JSONL format. This means that each JSON event object must fit on a single line in the file. There are many preprocessors that offer this functionality (i.e. Cribl), but in a pinch, any computer with a bash shell can convert the files using the following steps:
- Place all of your sample log files into a separate directory (if they aren't already).
- Open a bash terminal (or equivalent), and change directory into the folder with the sample logs.
- Run the following bash script:
for f in azure_*; do (cat "${f}"; echo) >> sample_logs.jsonl; done
You should now be able to use the file sample_logs.jsonl
to infer a schema.
Cause
This issue is because currently Panther doesn't support parsing JSON objects which span multiple lines.