I could think of two approaches …
-
You could try to schedule the incremental SPICE refresh frequency after considering average time taken to complete SPICE refresh . Say average incremental refresh 45 min then you could schedule the incremental refresh to run every 1 hr.
-
If multiple datasets needs to be refreshed along with dependency in a regular interval , then would recommend to orchestrate the SPICE ingestion using AWS Step Functions or Amazon Managed Workflows for Apache Airflow .
Thanks
VInod