Dataset list with refresh date information

Slimani

Hello dear community,

We try to list all dataset in all spaces with their last refresh date.

Our Aperture version id 2.8.8

We noticed that the rest API "datasets" allow us to get in it's json response:

• datasets : information of dataset in spaces
• columns: …
• tables: information of dataset (rows account, name …)
• batches: including time loaded and number of rows
• tags …

how it is possible to link batches to datasets in order to get the dataset last refresh date (I guess that time loaded = last refresh ) ?

Or maybe there are other ways to do it

Thanks for your help

Ali

Slimani

Josh Boxer

Hi Ali

A Dataset is just a container that stores settings/schemas etc. The actual data is stored in batches within the Dataset so there might be one batch that gets overwritten or multiple batches added for new data.

Simplifying the API response, each Dataset contains one or more tables which each contain one or more batches:

{

"datasets": [

  {

  "id": 55,
  "tables": [


      ],

      "batches": [

        {

          "timeLoaded": 1628058274391

You want to find the maximum timeLoaded from all the batches within a Table within a Dataset, for each Dataset

Josh Boxer

A way to do this is to pull the API response back into Data Studio and manipulate it from there

https://community.experianaperture.io/discussion/960/aperture-rest-api-calls-as-datasets-within-aperture

Slimani

Thank you Josh for your reactivity.

We expected to use Talend to provide Aperture in it's data drop zone the response of the rest web service.

I have done this step manually, I've uploaded the json response as a new local dataset (file upload)

response_1698136699041.json.txt

however, I could not extract the needed information (in a workflow) because the structure of the json is not respected

Josh Boxer

Thanks for the file, we will investigate this further, but it is possible there is an update needed to the Json file parser. Some not great options in the meantime, but just in case useful a) use the JDBC option linked above b) write a custom parser (wont require an upgrade so some benefit there) c) parse the JSON into tabular format before loading it

Josh Boxer

Update on this request, new JSON parser released:

https://community.experianaperture.io/discussion/1213/aperture-data-studio-2-12-13