SPICE data format

Question & Answer
, ,
1

I noticed the SPICE dataset is not quite optimized in data size. Compared to the same data in parquet format, I observed 4x to 10x gap in size efficiency.

What is the data format in SPICE? There is no official document about the data format in SPICE.
The document that I could find is about size calculation. (Importing data into SPICE - Amazon QuickSight)

2

Hello @hoyeon, I do not know for sure since it isn’t in any of the documentation, but it isn’t a compressed file format. It is going to be formatted more similarly to a CSV. If you had a CSV stored in S3 and in SPICE, those data sizes would be roughly the same.