Can I backfill the logs of a new log source into Panther?

QUESTION

I have a new log source for an organization I need to monitor. Would it be possible to backfill the logs into Panther for the last XX days?

ANSWER

Yes, it is possible to backfill logs into Panther, there are two ways to backfill logs:

If you have an S3 log source, you can use Panther's ops tool called s3sns to tell Panther to re-ingest specific objects from your S3 bucket.
- Follow the instructions here to download s3sns.
- The tool works by specifying your S3 bucket, and a prefix of your desired objects to be re-ingested. You will also specify the SNS topic that your S3 bucket sends notifications to. The tool will send notifications to the SNS topic for each object it finds at the prefix you specified. Then Panther will receive that notification and will read the object from your S3 bucket.
  Here is an example of how to use this tool:
```
s3sns -account <YOUR_AWS_ACCOUNT_ID> -region <THE_REGION_YOUR_SNS_TOPIC_IS_IN> \
    -topic panther-notifications-topic \
    -s3path s3://yours3bucketname/optionalprefixcangohere
```
For other log sources, we can initiate the backfill on your behalf.
- To initiate this process, please raise a request to our Support team and provide them with the log source ID and the exact timestamp that you want the backfilling to start from. Once your request is submitted, the Support team will engage the engineers, who may have some additional questions prior to starting the procedure.

Please note that there may be limitations on the time range depending on the log source. For example, the GitHub puller can backfill data up to a specific amount of time.

Also, please note that the data must be stored in such a way that they're accessible to s3sns in order for the tool to succeed. For example, items stored in AWS's Glacier Flexible Retrieval cannot be backfilled. To backfill these items, you must restore a temporary copy of the item to its S3 bucket for a specified duration. When you do this in an S3 bucket that is setup for ingestion into Panther, Panther will ingest these items automatically, without need for s3sns.