Merging archived dataset with the ongoing datsets (union of datasets)

Hi Team,

I want to merge two datasets to optimize the loading on the archived data again and again(incremental loading is not working effectively) . Like I want to created an archived dataset (static dataset- no loading required) and merge in which ongoing dataset (daily loading) and in the final dataset i want to merge (union of both) to create the respective dashboard.

Let me know if we can do this in any way around. :slight_smile:

Thanks in advance.

Hi @adi1994, It sounds like incremental refresh might be a good fit for your requirement however it seems it is not working as expected. Can you please elaborate on why is it not effective? We currently don’t have an option to union datasets.


Yeah, Thanks for the response.

Dataset is heavy because of merging of different table and with different columns as date time stamp, every time i am running on the desired date-time stamp on hourly basis with setting up to refresh last few hours in the window. It is storing the duplicate entries in the dataset.
Tried in various scheduling but its not working fine.


Hi @adi1994 - At AWS, our roadmap is primarily driven by our customers. Your feedback helps us build a better service. I have noted this in our internal feature request tracker. Thanks!