Panther policy failing unexpectedly when using AWS Tags
Issue
I want a Panther policy to only monitor a subset of my AWS resources, so I've applied AWS Tags to the resources that I don't want Panther to watch, as a sort of way to deny-list the resources that should not cause the policy to fail. However, from time to time the policy fails anyhow, on these exact resources. Panther seems to read the Tags data intermittently; some of the ingested data for this resource contains the Tag, and some of it does not. When I look at the resource in AWS, the resource appears to have the desired Tags.
Resolution
To resolve this issue, check whether the resource is shared by multiple AWS accounts. If it is, make sure both accounts maintain the same Tag information for that resource. One way to do this is to use the Resource Access Manager. If you've done this and the issue persists, contact Panther support for more assistance.
Cause
Shared AWS resources can undergo regular changes of ownership, where each AWS account "checks in" on the resource, and changes its Tags. When the Tags change, Panther notices and updates its policies accordingly. If the Tags normally "deny-list" a resource, but then the Tags are removed, the resource falls back into consideration by the policy.