Incremental refresh with different data sources

Question & Answer
, ,
1

Hello,
I’ve built a dataset that combines different custom SQL’s from 2 data sources.
When I tried to schedule the run time of this dataset I don’t have an incremental refresh option.
It’s a dataset with millions of rows.

Why there is no incremental option and how can I solve it?

Thanks in advance,
Mia

2

Hi @miap,

Thank you for posting. To use incremental refresh, you will have to write a single customSQL which joins the tables instead of performing the join in QuickSight.

Please check the link below for previous discussions on this issue

Please let me know if the options solves your issue.

Regards,
Demola

3

Thanks @demola ,

Is it possible to use a single custom SQL when I join tables from different data sources?

4

Yes you can.

5

How is it possible? each data connection is specific for one data source, and I have to create the custom SQL from a specific data connection…
For example, I want to join tables from Athena and PostgreSQL but there is no option for PostgreSQL DB

image
image765×435 22.5 KB

6

@miap ,

For example, I want to join tables from Athena and PostgreSQL but there is no option for PostgreSQL DB

This is not possible.

You can write a custom sql for Athena , a custom sql for PostgreSQL DB and then join the 2 in QuickSight . In this scenario only full refresh is possible.

image
image1508×346 19.7 KB

I don’t know of another way apart from bringing the data to a central repository and then setting up the incremental refresh process.

Kind regards,
Koushik