QUESTION

Why is it so slow when searching across "All tables" in Panther, and how can I improve performance for queries that are taking too long to run?

ANSWER

Doing a free text search across all tables can take a very long time to execute due to the large volume of records being processed. They search every field in every event (within the database, table, and date constraints), including fields nested in complex objects. Performance scales with several factors including the amount of data scanned, how the query is written, the number of sources, and other variables.

To improve search performance, try these optimization strategies:

1. Use specific indicator fields instead of raw text searches

Query specific fields using p_any indicator fields. For example, use p_any_domain_names when searching for a specific domain name instead of performing a raw text search of a single string.

2. Select a subset of tables to search

If you must do a free text search using a string, select a subset of tables to search.

3. Search just the Signals table first

Start by searching the Signals table instead of all Logs. This is much faster since it's a single table with fewer records. While the Signals table doesn't include all data, it can surface relevant cases more quickly. You can then pivot to other indicators or search specific log types based on the results.

4. Search specific log types when possible

Instead of searching all log types, search specific log types based on your findings from the Signals table. This is significantly faster than searching through all available log types.

5. Apply additional query optimizations

• Filter by time using p_event_time

• Avoid SELECT * statements

• Use LIMIT and aggregations where appropriate

• Use the Search tool for indicator lookups