I created a topic and in response got the refresh ARN.
Based on that I extracted the refresh ID and checked it status using describe_topic_refresh command. This worked fine and gave me proper result i.e. Running and then eventually completed when indexing is done.
After that if I do refresh (I did refresh now from console) and checked the status of refresh again using describe_topic_refresh , this return me completed always despite refresh is still going on. I used the refresh id that is returned during creation of topic.
My expectation was it will give me the current status of refresh rather than giving status of first refresh.
Question
Why I am not getting proper status after first time?
Do refresh id change on every refresh? If yes how to get the current refresh id?
How to know when was my index refresh last time using API?
I know from console I can see what I want, but there should be possibility from API also. OR is describe_topic_refresh made for just one time use?
The describe-topic-refresh command is designed to check the status of a specific refresh operation identified by a unique refresh ID. Refer
Each time you perform a refresh operation on a topic (whether through creation, update, or manual refresh), a new refresh ID is generated. Refer
When you initially created your topic, you received a refresh ARN and extracted the refresh ID from it, which allowed you to track that specific refresh operation.
However, when you performed a subsequent refresh from the console, a new refresh ID was generated for this operation.
I think, since you continued using the original refresh ID with the describe-topic-refresh command, you were still getting the status of the first refresh operation (which was already completed) rather than the new one.
To get the current refresh ID after performing a new refresh operation, you would need to capture the refresh ID returned when initiating the new refresh.
The RefreshId is a unique identifier that is created as a result of topic creation or topic update.
Regarding tracking when your index was last refreshed using API, i think there is no direct way to get that returns the last refresh timestamp. Refer
The describe-topic-refresh command only provides the status of a specific refresh operation identified by its refresh ID.
To monitor refresh operations properly, you need to capture and use the new refresh ID generated with each refresh operation when calling the describe-topic-refresh API.
Hello @Saurabh, the Topic refresh and the dataset refresh are 2 different operations. If you wanted to see when your dataset refresh has completed, you would want to use the DescribeDataSetRefreshProperties instead. Then, when this refresh completes, the data in your topic will be updated. I think you only need to run topic refreshes if you change the format of the dataset or add a new dataset field. Otherwise, the underlying data should update with the SPICE dataset.
HI @DylanM , I want to refresh the topic index when data is changed which doesn’t happens until topic is refreshed. I know data updates when dataset updates but not topic index. So questions based on new data values are not answered in topic. Question is we have API describe-topic-refresh for knowing topic refresh status which takes refresh ID and there is no way to know what is the refresh ID when dataset is updated which trigger topic refresh based on refresh schedule.
Hello @Saurabh, are you running your dataset refreshes on a schedule? If so, how often are you refreshing it? Why don’t you just schedule topic refreshes for 30 minutes after the dataset refresh? That should resolve the issue you are facing.
Hi @DylanM, My Dataset is in SPICE, So topic refresh is based on that. I am running topic refresh “Weekly”. But there is no way to track it was succeeded or failed via API.
Hello @Saurabh, this is a really interesting problem. I will definitely tag this topic as a feature request, because there is not a clear way to me to find the specific refresh id in order to validate the success of a new refresh.
Looking through the outputs, I am not seeing that it is returned in any of these queries, but these would be my best guess to run to try and find the latest refresh id:
List Topic Refresh Schedules should list the refresh schedules for a single topic, but I am unsure if it will show the ids.
Describe Topic should return a massive page of JSON. Again, I am not seeing anything specifically identifying the refresh id but there may be something useful returned here.
Let me know if you obtain anything useful out of these commands, and we can go from there. Otherwise, I have this tagged as a feature request to provide visibility to our support team.
