We have a data model where records initially have timestamp2 as NULL, and later get updated with an actual timestamp2 value. While querying our source database directly, we only see one record per entry (either with NULL if not yet updated, or with the actual timestamp if updated). However, when using QuickSight’s incremental refresh with the same query, we’re seeing duplicate records for the same entry - one with the original NULL timestamp2 and another with the updated timestamp2 value. This duplication doesn’t reflect our source data’s state and impacts our analysis. Even though our database properly maintains the latest state of each record through updates, QuickSight seems to be treating these updates as new records and retaining both versions in SPICE. We need to maintain records with NULL timestamp2 values when they haven’t been updated yet, but only see the updated version (non-NULL timestamp2) when an update has occurred.
How can we achieve this behavior with QuickSight’s incremental refresh without resorting to full refreshes?
Note: the incremental refresh is not on this column and is on another column that is never null.
Review this video which explain how the QuickSight incremental refresh would work : https://www.youtube.com/watch?v=jl_91ahT-P8 , that should provide you a good starting point to understand 1/ why you have duplicated records 2/ if the incremental refresh works for your use-case .
Based on the description of the issue; it appears that the field that you use for incremental data refresh is probably not the right one.
The video @Koushik_Muthanna has shared is a useful one. It does take a while to digest the way things work. So, if you still have issues you need to provide the details of your incremental refresh schedule and example data to review the issue.
Hi @mahehska,
Since we have not heard back, I’ll go ahead and close out this topic. However, if you have any additional questions, feel free to create a new post in the community and link this discussion for relevant information if needed.