QUESTION

After running PAT commands such as pat migrate, pat test, or pat upload, I'm seeing test failures or validation errors. How do I diagnose and resolve common PAT issues?

For general information on the PAT workflow, see Panther's PAT documentation.

ANSWER

1. Identify the error type

PAT errors generally fall into one of three categories. Identifying which type you're dealing with will help narrow down the cause:

Schema / validation errors:

The YAML file contains a field that the current version of PAT does not recognize, or is missing a required field.

'BaseVersion' not in list of valid keys: ['AnalysisType', 'Enabled', ...]

Module / import errors:

A Python file references a module or function that PAT cannot find in the current environment.

No module named 'panther_base_helpers'
ModuleNotFoundError: No module named 'panther_event_type_helpers'

Test logic errors:

A test case produces an unexpected result — the rule returns True when False is expected, or throws a runtime exception.

[FAIL] [rule] false
[FAIL] [rule] NameError: name 'parser' is not defined

2. Check your PAT version

Many errors are caused by a mismatch between PAT versions — for example, running an older version of PAT locally or in CI/CD against a detections repo that was migrated using a newer version.

Open up Terminal and check your local PAT version:

pat --version

If you are running PAT in a CI/CD pipeline (e.g. GitHub Actions), also check the version installed there. The version used locally and in CI/CD should match. If your pipeline installs PAT via pip, verify the pinned version in your workflow file.

3. Check whether your local files are current

After running pat migrate, some files in your repo may be stale even if the migration appeared to complete successfully. pat migrate updates rule .py files and test YAML, but does not always update associated files such as:

• Data model files (e.g. aws_cloudtrail_data_model.py, aws_vpcflow_data_model.yml)

• Global helper files (e.g. panther_base_helpers.py, panther_gsuite_helpers.py)

Stale data models cause UDM field lookups (e.g. event.udm("event_type")) to return None in test context. Stale helper files cause import errors or unexpected runtime behavior.

To check whether a file is current, compare your local version against the latest in the panther-analysis GitHub repository. If your local version differs, replace it:

# Verify the URL is valid before using it (should return HTTP 200)
curl -I https://raw.githubusercontent.com/panther-labs/panther-analysis/main/<path/to/file>

# Replace the file
curl -o <local/path/to/file> \  https://raw.githubusercontent.com/panther-labs/panther-analysis/main/<path/to/file>

After replacing files, run git diff to review what changed before committing.

4. Check whether pat merge can resolve the issue

If a specific managed rule is failing after migration, try:

pat merge <RuleID> pat test --filter RuleID=<RuleID>

pat merge compares your local version of a rule against the current version in panther-analysis main. If the rule has changed upstream, it will prompt you to accept the update.

Important: pat merge only acts when a rule has no BaseVersion set. If pat migrate has already stamped a BaseVersion on the rule, pat merge will return does not need merging even if the local rule logic is outdated. In this case, remove the BaseVersion line from the rule's .yml file and re-run pat migrate <RuleID> to force a re-migration.

5. Isolate the failing test

When a test fails with an unexpected result rather than an error, narrow down the cause by examining what the rule logic is actually doing:

1. Check what pat test outputs — The full test output failure often includes the actual return value, dedup string, alert context, and title, which can reveal where the logic is going wrong.

2. Check whether the issue is in the rule or the test data — compare your local rule .py and test YAML against the current versions in panther-analysis main. If the test data was altered by pat migrate (e.g. actor emails, event field names, or timestamp formats), the test expectations may no longer match the rule logic.

3. Check for UDM mapping issues — if a rule uses event.udm() and the test is returning an unexpected result, verify that the relevant data model file is current and that the test event data contains the fields the data model maps from.

6. Gather information for Support

If you are unable to resolve the issue after following the steps above, gather the following and reach out to Panther Support:

• The full pat test output (not just the summary)

• The PAT version (pat --version)

• The content of the failing rule's .py and .yml files

• The content of any data model or global helper files the rule depends on

• Whether the issue appeared after running pat migrate, and if so, what command was used (including any flags such as --auto-accept)

• Whether the issue occurs locally, in CI/CD, or both