When adding or removing fields from a custom schema in Panther, what happens to the corresponding columns in the data lake?
When I create a new fields for a custom schema, or remove an existing field, will a corresponding column be created/destroyed in the data lake table for this log type?
Panther manages updates to the data lake tables through edits to your custom schemas. When a new field is added, Panther will create a new column and begin recording event data for that field. When a field is removed, Panther retains the column in the data lake, but ignores this field for new log events which are processed. We retain the column so that the field (and it's information) are still accessible for logs previously ingested.
Do note that when viewing data explorer results, Panther automatically excludes some columns from the result set, if those columns were not present over the time span the query is searching. For example, if you remove the field source_hostname from a custom log source, then any query results from before the change will still include the source_hostname column; but if you query over a time span after the change, this column will be absent from your results.