How do I make panther_analysis_tool upload operations faster? Even when uploading a small amount of content, the upload is slow and takes a long time.
Here are some things that can help reduce the time spent waiting for panther_analysis_tool (PAT) to complete:
Exclude saved queries from the upload. Currently Panther saves these one at a time on the backend.
If what you're doing is updating detection code, use PAT test and validate. This way, when you merge a branch to main in GitHub Actions or your preferred CI/CD system, the upload command runs in the background.
Only upload the rules that have changed.
To upload only the changed rules, you can add -path ./rules to your command like so:
pipenv run panther_analysis_tool upload --path ./rules --filter Enabled=true --skip-testsWhen we upload the rules/ directory, if a rule has not been updated, nothing happens. This effectively ensures that only the files that have changed are updated.
If your CI/CD pipeline only runs the upload process, you can set your workflow to monitor only the rules folders:
paths:
- 'rules/**'
Check if your GitHub actions is set to use directory-based upload (ie. for dir in rules/*; do). This method processes directories one by one, and also re-uploads the global helpers and data models for each rule directory. Which could be why it's slow. You can refer to the example GitHub workflow YAML in our docs.