Is there a limit to how large a lookup table can be?

QUESTION

I'd like to upload a lookup table to Panther, but there is a lot of data. How big can my lookup table be?

ANSWER

There are a few limitations on your lookup tables, in terms of size:

• No individual row can exceed 64KB in size.

• If uploading through PAT, bulk upload, or our API, it's recommended to keep your data file < 1MB. If your file is larger than 1MB, it's suggested to instead use the S3 sync upload method.

• If syncing via S3, the max table size is 10GB.

• If syncing via GCS, the max table size is 10GB.

• The maximum number of rows is 100 million.

The S3 max table size, and the maximum row number, are negotiable - if you require an increase in either of these figures, contact Support (see 📄 How can I Increase my Lookup Table number or size limit in Panther?). However, the individual row size, and the API upload size, are hard limits that we are unable to increase at this time.