Dataset Incremental Refresh With SDK

Question & Answer
,
1

I’m trying to setup Incremental Refresh with Boto API call. I have the following code:

        if op_params["import_mode"] == "SPICE":
            schedule_params = {
                "AwsAccountId": op_params["aws_account_id"],
                "DataSetId": op_params["data_set_id"],
                "Schedule": {
                    "ScheduleId": f"{op_params['data_set_id']}Schedule",
                    "RefreshType": op_params["refreshType"],
                    "ScheduleFrequency": {
                        "Interval": op_params["refreshInterval"],
                        "TimeOfTheDay": op_params["refreshTime"],
                        "Timezone": op_params["refreshTimezone"],
                    },
                },
            }

            if op_params["refreshType"] == "INCREMENTAL_REFRESH":
                schedule_params["Schedule"]["IncrementalRefresh"] = {
                    "LookbackWindow": {
                        "ColumnName": op_params["lookbackWindowColumn"],
                        "Size": int(op_params["lookbackWindowSize"]),
                        "Unit": op_params["lookbackWindowUnit"],
                    }
                }

            schedule_response = client.create_refresh_schedule(**schedule_params)

I have these parameter values that get passed into the API call:

quicksightDatasetRefresh:
  refreshType: "INCREMENTAL_REFRESH"
  refreshInterval: "DAILY"
  refreshTime: "00:00"
  refreshTimezone: "UTC"
  lookbackWindowSize: 1
  lookbackWindowUnit: "DAY"
  lookbackWindowColumn: "timestamp"

When trying to deploy this using CDK I get the following error:

Message returned: Parameter validation failed: Unknown parameter in Schedule: "IncrementalRefresh", must be one of: ScheduleId, ScheduleFrequency, StartAfterDateTime, RefreshType, Arn

When I remove IncrementalRefresh and LookbackWindow it complains that there is no LookbackWindow configured. There is no documentation on the schema for creating incremental refreshes.

Any help would be appreciated!

2

Hi @kaveha,
Welcome to the QuickSight community.
To be able to configure an incremental refresh, you’ll have to set the dataset refresh properties by using the PutDataSetRefreshProperties API (Boto3 documentation). And afterwards, you can use the CreateRefreshSchedule API (Boto3 documentation) to create the actual refresh schedule.

The parameter options and structure of each of the API calls can be found in the request Syntax section of the documentation. You’ll have to move your LookbackWindow configuration into the content of the PutDataSetRefreshProperties API call, which should likely already fix your issue.

Did this answer your question? If so, please help the community out by marking this answer as “Solution”!