Dataset too large to import. How can I import certain amount of rows every x hours?

Question & Answer
,
1

Im trying to load data from a redshift cluster but the import fails because the dataset is too large to be imported using SPICE. (Figure 1)

How can I import…for example…300k rows every hour so that I can slowly build up the dataset to the full dataset?

Maybe doing an incremental refresh is the solution? The problem is I don’t understand what the “Window size” configuration means. Do i put 300000 in this field (Figure 2)?

Screen Shot 2022-04-03 at 4.02.21 PM
Screen Shot 2022-04-03 at 4.02.21 PM1920×1074 99.2 KB

2

Hey hasanm,

Thanks for using QuickSight. I think there are two issues here,

  1. get large number of rows into SPICE. Incremental refresh won’t help you in this case. Limits specified here Data source quotas - Amazon QuickSight applies to a single dataset no matter you use incremental refresh or not (in total number of rows)

  2. For setting up “window size” of incremental refresh, please refer to this doc, Refreshing SPICE data - Amazon QuickSight

Thanks,

1 Like