Issues

• When monitoring a Pub/Sub log source in Panther, there is high data latency (multiple hours) between when events occur (p_event_time) and when they are processed (p_parse_time).

• High latency between when data is received and when a detection is triggered and an alert is created.

Understanding Pub/Sub log source

Below is a high-level overview of the Pub/Sub log flow from GCS Notification → GCP Pub/Sub → Panther.

1. Panther subscribes to the Pub/Sub topic and fetches the corresponding GCS objects.

2. Panther parses the log (p_parse_time), then aggregates the logs into S3 objects up to 100 MB. This aggregation is critical for good performance.

3. Panther publishes notifications to an SNS topic referencing the aggregated S3 objects. Two processes run concurrently based on these notifications:

   1. An SQS queue for the "delta loader" listens for notifications from the topic.

      • The "delta loader" reads notifications from the queue and enqueues files with Databricks for Search.

   2. An SQS queue for the "detection processor" listens for notifications from the topic.

      • The "detection processor" listens for notifications from the queue and processes the streaming detections on the data.

      • Alerts are stored in DDB , a DDB stream is used to trigger a lambda for notifications (e.g., Slack , Jira, Webhooks).

See the Log Processing subsystem documentation for more information.

Solutions

To investigate high data latency issues, please check upstream data:

• Compare p_event_time versus p_parse_time in your data. Large differences indicate upstream delays before data reaches Panther, such as delays in your cloud provider's logging pipeline (eg., GCP Pub/Sub has a regional throughput limit).
  

To investigate high latency for detection/alert to be triggered:

1. Check the dedup period and make sure it's not the cause for the delay.

2. Traffic spikes: New log sources or increased log volume can push processing beyond current compute capacity