How to write unit tests for stateful Panther detections

Last updated
Save as PDF

QUESTION

What is the recommended approach to writing/maintaining unit tests in rules that are stateful detections?

ANSWER

While Panther currently doesn't allow you to fully simulate the k-v store, you can test individual components of your stateful detection to ensure your rule works as intended, using function mocks. There are two types of tests for stateful detections, and we'll explain both here: testing what the rule does when specific values are returned from the cache, and testing that your rule writes to the cache when it's supposed to.

For the examples below, we'll write tests for a detection which alerts if a user changes their password after logging in. This type of behavior might indicate an account takeover. The rule logic is as follows:

from panther_oss_helpers import put_string_set, get_string_set

# How far apart can the login and password change be? (In minutes)
TIMESPAN = 10

def rule(event):
    # Logic Path 1: Record login events
    if event.get('type') == 'user.login':
        # Cache this login, and the timestamp
        key = 'my_rule_id' + event.get('user_id') # Unique key for this rule and user
        put_string_set(key, str(event.get('p_event_time'), TIMESPAN * 60)
        # Return false - don't raise an alert for logins
        return False 
    
    # Logic Path 2: If password is changed
    if event.get('type') == 'user.password_changed':
        # Check if this user logged in recently
        key = 'my_rule_id' + event.get('user_id') Unique key for this rule and user
        if get_string_set(key, force_ttl_check=True):
            # If a value is returned by get_string_set, then ths user recently logged in!
            return True
    
    # By default, return false
    return False

Type 1: Testing for specific cache values

When writing a stateful detection, you need to know how it will react for a specific stored value. For our example rule, we should test what happens for the following cases:

User changed their password shortly after logging in
User changed their password, but not within 10 minutes of logging in

To accomplish this, we will use function mocks to simulate each case. By specifying values for get_string_set, we can see what happens in each case:

Set the return value for get_string_set to 1999-12-31 23:59:59Z (or any timestamp you prefer). This simulates the rule having previously stored a login timestamp from a previous event.
Set the return value for get_string_set to an empty string "". This simulates the case where there's no data in the cache, because we haven't recently seen any login events from this user.

Type 2: Testing whether the rule writes to cache

The other thing we want to test is what happens when we process a login event. Obviously, we expect the rule to write the timestamp to the cache, but how can we test this behaviour? The solution is a workaround which takes advantage of the return value of put_string_set.

To start, we must make a slight modification to the rule as written above, instead of returning False for the case where we detect a login event, we should instead return something else:

    # Logic Path 1: Record login events
    if event.get('type') == 'user.login':
        # Cache this login, and the timestamp
        key = 'my_rule_id' + event.get('user_id') # Unique key for this rule and user
        # Return false - don't raise an alert for logins
        return bool(put_string_set(key, str(event.get('p_event_time'), TIMESPAN * 60))

Here's the explanation: since the production versions of all the cache storing functions (put_string_set, reset_counter, increment_counter, put_dictionary, etc.) all return None, we can convert that to a boolean so the rule still returns False. However, this is helpful for testing, because in our unit test, we can override the return value of those functions.

By setting the return value of put_string_set to True in our unit test, the above code will now return True if we call put_string_set. Then, we can configure our unit test to set the expected result to True, in cases where the rule should write to the cache, and False in cases where it shouldn't.

Unfortunately, this doesn't allow you to test the precise contents of the cache, but you can ensure that your logic branches are executed when they should be.