Can I exclude logs from ingestion into Panther?

QUESTION

Are there any options to filter out logs as they are ingested into Panther?

• Can I filter based on specific values of specific fields?
• Can I filter based on file names?

ANSWER

Depending on the type of log source, Panther has a few options for filtering logs.

• If you have an S3 log source, you can filter incoming logs based on the prefix of the objects.
• If you have a CloudWatch Logs log source, you can filter incoming logs based on the information in the logs via a pattern filter.

Other than CloudWatch Logs log sources, Panther doesn't offer a native way to exclude logs based on information in the logs, but this could be done using a filtering tool like Cribl.

Potential alternate solutions:

• ​​​​If you can designate an S3 prefix for the data you'd like to exclude, do so and then configure the Log Source to exclude that prefix.
  • This also supports wildcards, so if you would like to exclude, for example, all CloudTrail Digest files, you can provide an exclude filter like AWSLogs/*/CloudTrail-Digest
• If what you want to do is redact certain pieces of information from the ingested data, you can exclude the sensitive fields from your schema. This way, parts of the payload won't be stored. Please note if you do this, that we will store the full payload if there are any classification errors, and today we don't provide an option to delete that. Also, we store raw data in our archive for 90 days, so omitting the fields from a schema may not do what you want if you want to avoid storing these things anywhere in the Panther instance.