Dataset list with refresh date information
Hello dear community,
We try to list all dataset in all spaces with their last refresh date.
Our Aperture version id 2.8.8
We noticed that the rest API "datasets" allow us to get in it's json response:
- datasets : information of dataset in spaces
- columns: …
- tables: information of dataset (rows account, name …)
- batches: including time loaded and number of rows
- tags …
how it is possible to link batches to datasets in order to get the dataset last refresh date (I guess that time loaded = last refresh ) ?
Or maybe there are other ways to do it
Thanks for your help
Ali
Comments
-
0
-
Hi Ali
A Dataset is just a container that stores settings/schemas etc. The actual data is stored in batches within the Dataset so there might be one batch that gets overwritten or multiple batches added for new data.
Simplifying the API response, each Dataset contains one or more tables which each contain one or more batches:
{
"datasets": [
{
"id": 55, "tables": [
],
"batches": [
{
"timeLoaded": 1628058274391You want to find the maximum timeLoaded from all the batches within a Table within a Dataset, for each Dataset
1 -
A way to do this is to pull the API response back into Data Studio and manipulate it from there
0 -
Thank you Josh for your reactivity.
We expected to use Talend to provide Aperture in it's data drop zone the response of the rest web service.
I have done this step manually, I've uploaded the json response as a new local dataset (file upload)
however, I could not extract the needed information (in a workflow) because the structure of the json is not respected
1 -
Thanks for the file, we will investigate this further, but it is possible there is an update needed to the Json file parser. Some not great options in the meantime, but just in case useful a) use the JDBC option linked above b) write a custom parser (wont require an upgrade so some benefit there) c) parse the JSON into tabular format before loading it
1 -
Update on this request, new JSON parser released:
1
