When trying to onboard a new custom S3 log source the following issue occurs:
No data is being pulled from the log source.
To resolve this issue:
- Did you already have your data in the S3 bucket before initially setting up the log source?
Any data that existed in your S3 bucket prior to setting up the log source will not be ingested automatically. The workflow is to create/connect your bucket to your console before you start uploading. If you need to re-ingest these files we do have a tool called the
s3snstool which you can use. For more information on how to backfill your logs using this tool, you can check our KB article.
- Is there new data coming into the S3 bucket?
Add new data to the bucket and see if "Last Received Data" updates for your new log source on the
- Does the data coming into the bucket match the log type's schema?
Check for log classification errors by clicking on your new custom S3 log source and clicking "Health" on the upper left-hand side of your Panther Console, or by using the following Data Explorer Query:
select * from panther_monitor.public.classification_failures where p_occurs_since('48 hours') -- adjust as needed and p_source_label = 'your-new-s3-log-source' order by p_parse_time desc limit 10
- Was the source set up via the CloudFormation wizard or manually?
Check to make sure all IAM roles for the source have permissions and are correctly hooked into Panther.
If the log source was configured manually, check the CloudFormation or Terraform template to ensure all roles have the correct permissions to write notifications.
This issue occurs when
- No new data is coming into the log source's S3 bucket
Panther only pulls data that comes into the bucket after the source has been onboarded. Old data will not be pulled in.
- The new data coming into the S3 bucket doesn't match the schemas for any of the log types associated with that log source
This can happen when logs do not include required fields or are not formatted the way specified in the schema and cannot be parsed. (JSON vs CSV, etc.)
- When set up manually and editing the CF or Terraform template, values can get deleted or roles may have conflicting names with existing AWS IAM role names. It's good to double-check when debugging that the template is complete.