Trying to understand topic here. I have several namespaces (each namespace is a client). Each namespace has about 50 datasets. I am running scheduled refresh by creating ingestions which runs in the EC2. If I understand this correctly, I can create topic with list of datasets associated to that namespace and create a refresh schedule (create-topic-refresh-schedule) for that namespace. Then I can eliminate the use of EC2 to run the scheduled job for dataset-refresh.
Ok. Is there a way I can create refresh schedule to a group of datasets instead of creating a refresh schedule for each dataset? My use case is if for a group of datasets. The dataset IDs are prefixed by namespace hash. Each namespace has about 50 datasets and I have hundreds of namespaces. I want to be able to refresh a group of datasets either by prefix or some other grouping mechanism.
Hi @alltej While I do not think there is an out of the box way of creating a refresh schedule for a group of datasets together, it would be useful to understand how you may have setup the process today… From your original post: