Skip to main content
Panther Knowledge Base

When adding or removing fields from a custom schema in Panther, what happens to the corresponding columns in the data lake?

  1. Last updated
  2. Save as PDF

QUESTION

When I create a new fields for a custom schema, or remove an existing field, will a corresponding column be created/destroyed in the data lake table for this log type?

ANSWER

Panther manages updates to the data lake tables through edits to your custom schemas. When a new field is added, Panther will create a new column and begin recording event data for that field. When a field is removed, Panther retains the column in the data lake, but ignores this field for new log events which are processed. We retain the column so that the field (and it's information) are still accessible for logs previously ingested.

Do note that when viewing data explorer results, Panther automatically excludes some columns from the result set, if those columns were not present over the time span the query is searching. For example, if you remove the field source_hostname from a custom log source, then any query results from before the change will still include the source_hostname column; but if you query over a time span after the change, this column will be absent from your results.