Problem with describe_data_set_refresh_properties API

Question & Answer
, ,
1

Hello. When calling describe_data_set_refresh_properties, there seems to be two problems:

  1. It does not provide response for datasets that DO HAVE refresh configuration and returns:
    “Dataset refresh properties are not set for this dataset”
  2. The response is NOT accurate:
    For datasets that are set to refresh fully, shows Incremental. The time is not accurate too, shows all as DAY even if Weekly refresh is set.
    {‘RefreshConfiguration’: {‘IncrementalRefresh’: {‘LookbackWindow’: {‘ColumnName’: ‘DateStamp’, ‘Size’: 1, ‘SizeUnit’: ‘DAY’}}}}
    Tested several times on more than 50 datasets
2

Hello @quickmeso, welcome back to the community!

In regards to your first point, I feel like that response is fairly helpful. Is it missing something that you think it should be returning, like does it not include the dataset id?

As for the 2nd item you mentioned, that definitely does sound like an issue. One interesting thing I noticed though when looking through the documentation is I think it may always return ‘IncrementalRefresh’: even if it is a full refresh that you have scheduled. It looks like it is part of the default return from the API. If you have checked your datasets manually and see that some of them should be returning as WEEK for SizeUnit instead of DAY, I would recommend filing a case with AWS Support where we can dive into the details to help you further. Here are the steps to open a support case.

If they are able to provide some valuable information on this issue, please let me know what they discovered, especially if you still need guidance to resolve this. Thank you!

3

Thanks for your response.
For the first one: It does not return the dataset Id. There is a refresh configuration, so it is expected to return something.
For the second: This is definitely an issue when an API returns wrong configuration. This API response is so poor and inaccurate for all datasets that I will work manually until their future release.

4

Hello @quickmeso, I see, you do have a refresh configured but it is returning that you don’t. Well I would definitely recommend filing a case with AWS support so they can help you further. I will also tag this topic on my end. If you have any remaining questions, please let me know. Otherwise I will archive this topic for our support team. Thank you!