Removing a field from a schema to reduce classification failures in Panther

Issue

In a certain data source, most data ingests into Panther properly but some events end up unmatched, throwing classification errors, due to a special character or an uncommonly formatted value in one of the fields.

Resolution

To resolve this issue, you'll need to adjust the schema used to ingest data from this log source. If the error occurs in just one field, and this field isn't particularly crucial for your purposes, consider removing the field from the schema. This way, Panther ingestion will ignore this field altogether, so while uncommon formatting won't cause any errors, you also won't be able to use it in other Panther workflows.

Field removal example:

Schema

version: 0
fields:
- name: field_1
      type: string
- name: field_2
      type: string 

Event

{"field_1": "value 1", "field_2": "value 2"}

Parsed Event

{
	"field_1": "value 1",
	"field_2": "value 2",
	"p_event_time": "2023-06-12 06:51:47.329",
	"p_log_type": "XXXXXXX",
	"p_parse_time": "2023-06-12 06:51:47.329",
	"p_row_id": "869d5b40fcac8099bce5f6db1815",
	"p_schema_version": 2,
	"p_source_id": "c3e61bd7-b1aa-4a7a-8a9f-848045719578",
	"p_source_label": "XXXXXXX"
}

This event is now stored in the Data Lake, if you edit the schema afterward, remove field_2 and ingest a new event that will contain field_2:

Schema

version: 0
fields:
- name: field_1
      type: string

Event

{"field_1": "value 3", "field_2": "value 4"}

Parsed Event

{
	"field_1": "value 3",
	"p_event_time": "2023-06-12 06:55:47.316",
	"p_log_type": "XXXXXXX",
	"p_parse_time": "2023-06-12 06:55:47.316",
	"p_row_id": "76bcfcf3f645cab9c8b986dc1807",
	"p_schema_version": 3,
	"p_source_id": "c3e61bd7-b1aa-4a7a-8a9f-848045719578",
	"p_source_label": "XXXXXXX"
}

The new event will only have field_1 populated and the field_2 will not be captured. Keep in mind that the old event which had the field_2 populated will remain in the Data Lake and you will be able to query it, your schema edit will apply only to the new events that will be ingested.

For other schema adjustment options, look for other articles related to "classification failures" in our System Errors and Health Notifications section.

Cause

This issue occurs when the data ingested doesn't match the schema.