How can I get more fine-grained health alarms for my Panther log source?

Issue

I haven't seen logs processed for one of my log types or one of my S3 prefixes in my log source, but my log source is still reporting healthy. Why?

Resolution

To troubleshoot this issue, consider the following:

1. Does your log source have more than one log type?
   • To create a drop-off alarm on a per-log type basis, we can create a scheduled query. The scheduled query will look for activity n+1 days ago and return the most recent event; the associated rule can check if that p_event_time is >= 5 days ago. (If we try to see if no logs returned 5 or more days ago, the scheduled query will not trigger the scheduled rule as the payload would be empty.)

      SELECT
      p_event_time,
      p_source_label
     FROM
      panther_logs.public.custom_mylogs
     WHERE
      p_occurs_since('6 days')
     ORDER BY p_event_time DESC
     LIMIT 1 

2. Do you have multiple prefix filters set, and one of them is not pulling data?
   • Double-check that the prefix filter has the correct spelling of the prefix you are trying to pull from your bucket and that it exists as spelled.
   • Wildcards can only be used for exclusion filters, so ensure they are not included in your inclusion filters. See this article for more information on prefix filters.
   • To set up an alert in this case, we can also use a scheduled query that checks against the panther_monitor.public.data_audit database for the prefix that we're targeting and format it similarly to the above.

Cause

This issue occurs because the log source drop-off alarm looks at the overall health of your log source to determine the health status. It can work well for diagnosing if your credentials are expired or if the service you're pulling from drops the connection to Panther completely. If any logs are still being ingested via that transport within the drop-off alarm timeframe, then the log source will show as healthy.