Spice Computation

Hi @woliveiras

SPICE uses columnar storage, which is optimized for analytics and often compresses data by storing columns separately. When many rows share the same values in certain columns (like ColumnA, ColumnB, and ColumnC in your example), SPICE can apply compression effectively.

SPICE also stores metadata, indexing, and other overhead for each file, which can result in a dataset size larger than expected when concatenating multiple files.

Full Refresh vs. Incremental Refresh: If you perform a full refresh, SPICE reloads the entire dataset, potentially leading to increased storage requirements as new, unique data accumulates. On the other hand, using incremental refresh can optimize storage by only appending new rows (e.g., for a new LoadDate) instead of reloading the entire dataset. This method is more storage-efficient when only a portion of your data changes over time.

Please refer the below documentation this might be helpful for you.

1 Like