Could someone please help me with the Space or capacity it takes and what would be the approximate time needed by the Spice engine to ingest a dataset of 572 million rows and having 200 columns. I just need an approximate number as I have to do a initial data load with this much volume and post which I’m planning to do an Incremental load on a daily basis. It would be very helpful, if you could also share any best practices when handling with huge volume of data.
Please refer the below documentation this might be helpful for you.
Best Practices for Handling Large Data:
-
After the initial load, implement incremental refreshes to reduce the daily load size. Ensure that your data model is designed to accommodate these incremental updates efficiently.
-
Remove unnecessary columns, especially high cardinality string columns, as they affect performance. Consider splitting the dataset or using smaller datasets for specific dashboards if possible.
1 Like
Hi @Sriselvaraj24,
It’s been awhile since we last heard from you, did you have any additional questions regarding your initial topic?
If we do not hear back within the next 3 business days, I’ll go ahead and close out this topic.
Thank you!
Hi Team,
Thanks for the information. You can close this request.
1 Like