Are there any options to filter out logs as they are ingested into Panther?
Can I filter based on specific values of specific fields?
Can I filter based on file names?
The easiest way to do this is to use Panther's built-in raw event filtering, documented here. While that method of filtering was designed for this purpose, the options below may also fit your use case.
You can add the required:true tag in your schema to prevent events from parsing if they miss these specific fields.
If you have an S3 log source, you can filter incoming logs based on the prefix of the objects.
If you have a CloudWatch Logs log source, you can filter incoming logs based on the information in the logs via a pattern filter.
If you can designate an S3 prefix for the data you'd like to exclude, do so and then configure the Log Source to exclude that prefix.
This also supports wildcards, so if you would like to exclude, for example, all CloudTrail Digest files, you can provide an exclude filter like AWSLogs/*/CloudTrail-Digest
You can also exclude specific object names. For example, using the exclusion filter testing*.txt will exclude objects like testing2.txt and testing3.txt. Similarly, using *.txt will exclude all the TXT files in your specified S3 directory.
Please note that:
Wildcards work only for the exclusion filters
You cannot append an asterisk at the end of your filter (e.g., /test/* is not valid))
To redact certain pieces of information from the ingested data, you can exclude the sensitive fields from your schema. This way, parts of the payload won't be stored. Please note if you do this, that we will store the full payload if there are any classification errors, and today we don't provide an option to delete that. Also, we store raw data in our archive for 90 days, so omitting the fields from a schema may not do what you want if you want to avoid storing these things anywhere in the Panther instance.