Issue

When trying to onboard a new custom S3 log source the following issue occurs:

No data is being pulled from the log source.

Resolution

To resolve this issue:

• Did you already have your data in the S3 bucket before initially setting up the log source?
  Any data that existed in your S3 bucket prior to setting up the log source will not be ingested automatically. The workflow is to create/connect your bucket to your console before you start uploading. Please try re-uploading your files. If you need to re-ingest these files we do have a tool called the s3sns tool which you can use. For more information on how to backfill your logs using this tool, you can check our KB article.

• Is there new data coming into the S3 bucket?
  Add new data to the bucket, then see if "Last Received Data" and "Last Data Ingested" update for your new log source on the Configure > Log Sources page. If "Last Data Ingested" has a timestamp, you can look for the data in the "Schemas" tab and click "View Data". That will bring you to the Investigate > Search page, where you may need to use the drop-down menu to change the date range to see your newly ingested data.

• Does the data coming into the bucket match the log type's schema?

Check for log classification errors by clicking on your new custom S3 log source and clicking "Health" on the upper left-hand side of your Panther Console, or by using the following Data Explorer Query:

select * from panther_monitor.public.classification_failures
where p_occurs_since('48 hours') -- adjust as needed
and p_source_label = 'your-new-s3-log-source'
order by p_parse_time desc
limit 10

• Was the source set up via the CloudFormation wizard or manually?
  Check to make sure all IAM roles for the source have permissions and are correctly hooked into Panther.
  If the log source was configured manually, check the CloudFormation or Terraform template to ensure all roles have the correct permissions to write notifications.
   

Cause

This issue occurs when

1. No new data is coming into the log source's S3 bucket
   Panther only pulls data that comes into the bucket after the source has been onboarded. Old data will not be pulled in.

2. The new data coming into the S3 bucket doesn't match the schemas for any of the log types associated with that log source
   This can happen when logs do not include required fields or are not formatted the way specified in the schema and cannot be parsed. (JSON vs CSV, etc.)

3. When set up manually and editing the CF or Terraform template, values can get deleted or roles may have conflicting names with existing AWS IAM role names. It's good to double-check when debugging that the template is complete.