Can I backfill the logs of a new log source into Panther?
I have a new log source for an organization I need to monitor. Would it be possible to backfill the logs into Panther for the last XX days?
Yes, it is possible to backfill logs into Panther, there are two ways to backfill logs:
- If you have an S3 log source, you can use Panther's ops tool called
s3snsto tell Panther to re-ingest specific objects from your S3 bucket.
- Follow the instructions here to download s3sns.
The tool works by specifying your S3 bucket, and a prefix of your desired objects to be re-ingested. You will also specify the SNS topic that your S3 bucket sends notifications to. The tool will send notifications to the SNS topic for each object it finds at the prefix you specified. Then Panther will receive that notification and will read the object from your S3 bucket.
Here is an example of how to use this tool:
s3sns -account <YOUR_AWS_ACCOUNT_ID> -region <THE_REGION_YOUR_SNS_TOPIC_IS_IN> \ -topic panther-notifications-topic \ -s3path s3://yours3bucketname/optionalprefixcangohere
- For other log sources, we can initiate the backfill on your behalf.
- To initiate this process, please raise a request to our Support team and provide them with the log source ID and the exact timestamp that you want the backfilling to start from. Once your request is submitted, the Support team will engage the engineers, who may have some additional questions prior to starting the procedure.
Please note that there may be limitations on the time range depending on the log source. For example, the GitHub puller can backfill data up to a specific amount of time.