How do I resolve "Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type" when setting up an S3 source for Panther?
Issue
I'm setting up a Panther S3 source and manually setting up the IAM role on my own. I'm trying to add a new filter to send All object create events from aws/ to my Panther SNS topic (panther-notifications-topic), but I see the error:
Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.
My existing event configuration filters look like this:
Resolution
Event notifications are configured via object key name filtering, name filters or just filters.
When creating event notifications, avoid creating multiple filters that use overlapping prefixes and suffixes. Otherwise, your configuration will not be considered valid.
To resolve this issue:
-
You can use overlapping object key name filters with different event types. For example, you can create a notification configuration that uses the prefix
image/for theObjectCreated:Putevent type and the prefiximage/for theObjectRemoved:*event type. -
You may want to use a fanout method to send event notifications to multiple destinations via a single SNS topic.
-
This is described in a Medium article linked here. Note that this is a third party web site and the solution described here is not officially supported by Panther.
-
See below for more information on using prefixes and suffixes.
Prefixes & suffixes
-
A prefix includes an object if a key starts with the specified characters;
-
A suffix includes an object if the key ends with the specified characters
Imagine your bucket contains these S3 objects:
images/myfile.jpg
images/myfile.png
logs/vpcflowlogs/.../foo.log.gz
In this example, both images/ and logs/ can both be considered prefixes. An example suffix would be "jpg," "png" or any subset of those characters, i.e. "pg" or "ng."
Valid event notification configuration examples
In each example, assume you are trying to set up two distinct notifications for All object create events using the filters shown. Your bucket contains objects that look like this:
test/.../log.json
test2/.../image.jpg
Non-overlapping prefixes
This configuration is valid because the prefixes used in each notification do not overlap.
Non-overlapping suffixes
This configuration is valid because the suffixes used in each notification do not overlap.
Overlapping prefix, non-overlapping suffix
This configuration is valid because while the prefixes overlap, the suffixes used in each notification do not overlap.
Invalid event notification configuration examples
Overlapping root prefix
There cannot be another notification for this event type because your bucket's root prefix overlaps with any other prefix (the same is true if you use a suffix instead of a prefix - the root suffix also overlaps with any other suffix). A wildcard (*) prefix/suffix is assumed when none is specified.
You may see the error that prompted this question: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.
Overlapping prefixes
This configuration is not valid because the prefixes overlap.
Name Event types Filters Destination type Destination
Notification 1 All object create events test/ SNS topic topic-1
Notification 2 All object create events test/ SNS topic topic-2
Overlapping suffixes
This configuration is not valid because the suffixes overlap (no prefix in Notification 1, and "on" is a substring of "json" in Notification 2).
Name Event types Filters Destination type Destination
Notification 1 All object create events , .on SNS topic topic-1
Notification 2 All object create events test2/, .json SNS topic topic-2
Overlapping prefixes & suffixes
This configuration is not valid because both the prefixes and suffixes overlap.
Name Event types Filters Destination type Destination
Notification 1 All object create events test/, .json SNS topic topic-1
Notification 2 All object create events test/ SNS topic topic-2
Cause
This is typically caused by a known AWS S3 name filtering issue.