Incremental Refresh Error - Dataset is not eligible for incremental refresh

Question & Answer
,
1

Hi,

I encountered an error while refreshing a dataset in Amazon QuickSight. Below are the details of the error message I received.

  • Ingestion Id: c68941e0-7708-4a13-83e2-aab48f10950a
  • Error type: CUSTOMER_ERROR
  • Error details:

Dataset with id: de54b108-bdac-4371-ab96-d74649d17c75 is not eligible for incremental refresh. Configure Incremental refresh correctly.

Could you please advise me on how to resolve this issue?
I would like to know the correct configuration for enabling incremental refresh for this dataset.

Thank you for your support.

2

What is the Data Source (Amazon S3, RDS or anything else)?

Some things to check -

  1. Does the data source have a valid primary key and is incremental refresh supported on it? Additional configuration may be needed if incremental refresh is not supported.
  2. Check if ‘Enable incremental refresh’ setting is turned on in the datasets page.
  3. If the size of the dataset is too large, incremental refresh may not work.
    Thanks!
3

The data source I am using is Amazon Athena.

Incremental refresh was working fine when I used a single table.
However, after I joined two tables (both having timestamp fields) to create a new dataset, incremental refresh stopped working and I received the error.

Is incremental refresh not supported for datasets created by joining multiple tables in Athena?

Thanks!

4

Incremental refresh is supported for datasets created by joining multiple tables in Athena.

Issue may be due to:

  1. Timestamp Field Conflict: When joining two tables with timestamp fields, Athena might struggle to determine which timestamp field to use for tracking changes.
  2. Lack of Unique Identifier: If the joined tables do not have a unique identifier, Athena might not be able to track changes correctly.

Recommendations:

  1. Ensure both tables have Timestamp fields with same data type and format
  2. Add a unique identifier column to each table if not already present.
  3. Explicitly specify which Timestamp field to use for incremental refresh when creating the dataset.

Let us know if this works.
Thanks!

5

Hi @InseokSeo,
It’s been awhile since we last heard from you on this thread, did you have any additional questions regarding your initial post?

If we do not hear back within the next 3 business days, I’ll close out this topic.

Thank you!

6

Hi @InseokSeo,
Since we have not heard back, I’ll go ahead and close out this topic. However, if you have any additional questions, feel free to create a new post in the community.

Thank you!