How to make Panther Analysis Tool upload faster

QUESTION

How do I make panther_analysis_tool upload operations faster? Even when uploading a small amount of content, the upload is slow and takes a long time.

ANSWER

Here are some things that can help reduce the time spent waiting for panther_analysis_tool (PAT) to complete:

• Exclude saved queries from the upload. Currently Panther saves these one at a time on the backend.
• If what you're doing is updating detection code, use PAT test and validate. This way, when you merge a branch to main in GitHub Actions or your preferred CI/CD system, the upload command runs in the background.

• Only upload the rules that have changed.

  • To upload only the changed rules, you can add -path ./rules to your command like so:

pipenv run panther_analysis_tool upload --path ./rules --filter Enabled=true --skip-tests

When we upload the rules/ directory, if a rule has not been updated, nothing happens. This effectively ensures that only the files that have changed are updated.

• If your CI/CD pipeline only runs the upload process, you can set your workflow to monitor only the rules folders:

  paths:
    - 'rules/**'
  

   
  • Check if your GitHub actions is set to use directory-based upload (ie. for dir in rules/*; do). This method processes directories one by one, and also re-uploads the global helpers and data models for each rule directory. Which could be why it's slow. You can refer to the example GitHub workflow YAML in our docs.