Hi All,
I’ve added a handful of filters on my dataset, but I did not get the loading speed improvements I thought I would get when loading the data into SPICE. Is that expected behavior?
@dmarcus Could you please check your dataset size? Does the filter you applied significantly reduce the data size?
What is the data source? When you run the data ingestion? Is there any other heavy queries running during your data ingestion?
All these factors would impact the loading speed. FYI.
Hi,
QS providing good features to filter on many condition at dataset levels.
SPICE is super fast. its all depend on your source/DB to how much it take time for process data.
It’s is good to exclude unwanted data from your source. This will speed up you data loading sources.
1 Like
In general, it is a good idea to filter out data if the underlying datasouce handles those filters optimally. This will definitely improve the ingestion time after the underlying query execution completes. However, the underlying datasource can have other bottlenecks and hence you would not notice much difference - or in some cases have adverse effects on the loading time. Example, if you are using a relational database and your dataset filter is doing a full table scan, then the underlying query execution time will not improve unless you have the right indexes in place to support your filters.
Additionally, if your filters only affect a small proportion of the rows out of the whole data set, then you will probably not notice the difference either.
1 Like