Hello Guys, I have being using incremental loading for the heavy data set on spice. Moreover while loading it with respect to the date time stamp for a refresh time for 30min, with 1 day window as describe below. Incremental refresh configuration Date column: CreatedAt Window size: 1 day Now in…

Hi, Incremental refresh works on identifying 2 types of rows. So during initial full refresh after setting up the configuration, rows that are before CreatedAt are identified as immutable rows and rows after identified as mutable rows. As part of full refresh we store the lookback window (current t…

Hi, you can try with hours option 24 hours. [image]

Thanks for the response, i will surely check and update :wink:

Hi Naveed, The same is not working !! , Still duplicates are observed

Hi All, Can you please check my questions and comment if anyone knows the answers. Thanks in advance :slight_smile:

Hi! I have the same issue. But the problem is - it works right the same Quicksight describe it in their docs. And the way it works looks a but stupid for me (may be I am stupid? ) : " An incremental refresh queries only data defined by the dataset within a specified look-back window. It transfers al…

Not sure if you were able to resolve this but if the date column is also being updated as part of you row updates, you will see duplicates in Spice. This is due to the fact that the timestamp stored by Quicksight to lookup later during previous incremental refresh is changed (with update). The sugge…

I am using static timestamp column , then also it is under the same error. I guess this need to get fix from quicksight technical teams end. How we can raise this issue to the relevant team , any idea?? Moreover, bcs of this is very hard to use this loading.

Hi @adi1994 In case you need them, here are the steps to open a support case . If your company has someone who manages your AWS account, you might not have direct access to AWS Support and will need to raise an internal ticket to your IT team or whomever manages your AWS account. They should be able…

Hi, my case it that.. even if there are no new rows.. based on the lookback window it’s appending all the rows falling in that window. The column i use is “record_updated_at”. Kept an hourly incremental refresh. Won’t it be like based on the lookback window if some record changed in that column sinc…

Duplicates rows are being created via Incremental loading on data set

Q&A

Naveed March 28, 2022, 9:15am 6

Hi,
can you Please verify your data have no duplicates ?

Quick Sight Incremental Dataset Refresh duplicates rows instead of replacing old ones and appending new ones

Topic		Replies	Views
Quick Sight Incremental Dataset Refresh duplicates rows instead of replacing old ones and appending new ones Q&A quick-sight , dataset , Business-Intelligence-Engineer	3	60	June 3, 2026
Rows removed during incremental refresh Q&A spice , error , quick-sight	13	1349	November 2, 2022
Delete duplicate rows appended by incremental refresh Q&A quick-sight , Business-Intelligence-Engineer , tables	9	639	October 12, 2023
Unknown rows getting imported during incremental refresh Q&A date , quick-sight , dataset , qls-spice , Business-Intelligence-Engineer	3	77	May 16, 2024
Incremental refresh created duplicates Q&A data-source , spice , feature-request , quick-sight , dataset	4	622	June 1, 2023

Duplicates rows are being created via Incremental loading on data set

Related topics