Managing refresh automations for 447 datasets
I'm working on a data migration project in Aperture with approximately 447 datasets, many of which are reference or lookup tables.
I'm looking at options for automating dataset refreshes and would be interested to hear what approach others have taken when managing a large number of datasets:
Adding datasets individually to a schedule automation
Grouping datasets into workflows and scheduling the workflow
Splitting datasets into multiple workflows (e.g. operational vs reference data)
Any other recommended approaches
My main concerns are maintainability, ease of administration, and avoiding the need to manage hundreds of individual refreshes over time.
Has anyone worked with a similar number of datasets and found an approach that scales well? Are there any best practices, limitations, or lessons learned that would be worth considering before implementing this?
Comments
-
That's a lot, and you have some good questions. Have you looked into using the API to do the datasets refresh?
1 -
Agree API is the way to go once you have that volume of objects to manage
0 -
Thanks for the guidance. Is the API approach is designed to handle refreshing all 447 datasets at scale, or whether there are any practical limits that would require datasets to be split into batches or refreshed manually. Are there any recommended best practices for managing refreshes at this volume?
0 -
APIs are designed for large-scale repetitive tasks. You could request a list of all Datasets, determine which need to be refreshed (based on a name, type, object tag etc.) then loop through the list refreshing each one.
Having said that you could do this in UI manually if that makes more sense. Filter the list of Datasets to those that need a refresh (based on a name, type, object tag etc.), select them all and hit Dataset Refresh
0
