What is the recommended approach to writing/maintaining unit tests in rules that are stateful detections?
While Panther currently doesn't allow you to fully simulate the k-v store, you can test individual components of your stateful detection to ensure your rule works as intended, using function mocks. There are two types of tests for stateful detections, and we'll explain both here: testing what the rule does when specific values are returned from the cache, and testing that your rule writes to the cache when it's supposed to.
For the examples below, we'll write tests for a detection which alerts if a user changes their password after logging in. This type of behavior might indicate an account takeover. The rule logic is as follows:
from panther_oss_helpers import put_string_set, get_string_set # How far apart can the login and password change be? (In minutes) TIMESPAN = 10 def rule(event): # Logic Path 1: Record login events if event.get('type') == 'user.login': # Cache this login, and the timestamp key = 'my_rule_id' + event.get('user_id') # Unique key for this rule and user put_string_set(key, str(event.get('p_event_time'), TIMESPAN * 60) # Return false - don't raise an alert for logins return False # Logic Path 2: If password is changed if event.get('type') == 'user.password_changed': # Check if this user logged in recently key = 'my_rule_id' + event.get('user_id') Unique key for this rule and user if get_string_set(key, force_ttl_check=True): # If a value is returned by get_string_set, then ths user recently logged in! return True # By default, return false return False
Type 1: Testing for specific cache values
When writing a stateful detection, you need to know how it will react for a specific stored value. For our example rule, we should test what happens for the following cases:
- User changed their password shortly after logging in
- User changed their password, but not within 10 minutes of logging in
To accomplish this, we will use function mocks to simulate each case. By specifying values for
get_string_set, we can see what happens in each case:
- Set the return value for
1999-12-31 23:59:59Z(or any timestamp you prefer). This simulates the rule having previously stored a login timestamp from a previous event.
- Set the return value for
get_string_setto an empty string
"". This simulates the case where there's no data in the cache, because we haven't recently seen any login events from this user.
Type 2: Testing whether the rule writes to cache
The other thing we want to test is what happens when we process a login event. Obviously, we expect the rule to write the timestamp to the cache, but how can we test this behaviour? The solution is a workaround which takes advantage of the return value of
To start, we must make a slight modification to the rule as written above, instead of returning
False for the case where we detect a login event, we should instead return something else:
# Logic Path 1: Record login events if event.get('type') == 'user.login': # Cache this login, and the timestamp key = 'my_rule_id' + event.get('user_id') # Unique key for this rule and user # Return false - don't raise an alert for logins return bool(put_string_set(key, str(event.get('p_event_time'), TIMESPAN * 60))
Here's the explanation: since the production versions of all the cache storing functions (
put_dictionary, etc.) all return
None, we can convert that to a boolean so the rule still returns
False. However, this is helpful for testing, because in our unit test, we can override the return value of those functions.
By setting the return value of
True in our unit test, the above code will now return
True if we call
put_string_set. Then, we can configure our unit test to set the expected result to
True, in cases where the rule should write to the cache, and
False in cases where it shouldn't.
Unfortunately, this doesn't allow you to test the precise contents of the cache, but you can ensure that your logic branches are executed when they should be.