When trying to ingest zst compressed data from my S3 log source into Panther I see an error message such as:
failed to read JSON array
or
error found in #1 byte of ...|(\ufffd/\ufffd\u0000|...Double check to ensure your zst compressed files were not compressed using a dictionary. If they were compressed with a dictionary, please modify the system that sends these files to S3 such that they are compressed without using a dictionary.
If you are ingesting compressed data, Panther will try to decompress the files prior to classifying the events in your files. If Panther doesn't detect that your files are compressed, it will continue to try to read them as is. For this reason, if you have compressed files that Panther fails to decompress, you might see errors that look like Panther is attempting to read a binary file.
Specifically for zst compression, this could happen if you compress your files using a dictionary. Panther does not support decompressing files that were compressed using a dictionary, and therefore you might see errors like the above. See here for more details on Panther's supported compression formats.
If you did not compress your input files with a dictionary, please reach out to Panther's support team and share how these files were compressed.