I have an SQL source table with records that are never deleted, but are added and updated. There is a primary key column id.
In QuickSight, I added this as a simple Spice dataset, with an incremental refresh based on updated_at with a one hour window, triggered hourly.
But the dataset keeps adding “new” records, i.e. I have ~70K records instead of ~55K in the source. If I run a full refresh, the correct number of records is shown.
How can I tell QuickSight to respect the primary key and only update the records in an incremental refresh?
Hi @Gabriel_R ,
The incremental refresh is based on the lookback window , following video explains what happens : 2022 QuickSight Learning Series: Keep your data fresh with incremental SPICE refresh - YouTube
Kind regards,
Koushik
1 Like
Thanks for the video Koushik, make sense. I’m horrified 
So it’s by design that records are duplicated?!
PS: See at 18m00s in the video where Karthik explains that duplicates can occur.
This makes incremental refresh useless for so many use cases.
Please consider making this simpler and more powerful, i.e. similar to how Zoho Analytics does it:
Hi @Gabriel_R ,
The records are not duplicated by design , it’s looking into lookback window configuration. Hence the update is not similar to what happens in a datawarehouse scenario .
Thanks for your feedback, I will tag this question as a feature request so that we capture your requirements.
Kind regards,
Koushik