Load Timestamp for data loads

Options
Jasper
Jasper Member
in General

Hi,
I would like a timestamp of when data is last loaded in ADS using the 'add dataset' capability. Furthermore, I would like this to reflect the most recent refresh date when we the data is reloaded.

Is there any functionality with ADS to do this ?

Thanks Jasper

Tagged:

Best Answers

  • Ivan Ng
    Ivan Ng Administrator
    Answer ✓
    Options

    Hi Jasper,

    There are a few ways to to get the dates. Please see below:

    • Dataset listing page 'Refreshed' column
    image.png
    • View more information under the Action menu
      image.png
    • Metadata option in the Source step
    image.png

    Does this help? Do let me know if you have further questions. Thanks.

  • Seamus Kyle
    Seamus Kyle Experian Contributor
    Answer ✓
    Options

    Hi, sorry just spotted this conversation.

    Jasper, if I understand you correctly, then the Metadata option in the Source step (mentioned above) should give you exactly what you are asking for.

    Once you select this option you get a new column, on the output of the source step, with the timestamp. This timestamp is the timestamp for when the data was last loaded.

    image.png
    image.png

    If you have a multi batch dataset you can also select Batch ID and you will get the timestamps for when each batch was loaded.

    image.png


    Hope this helps

Answers

  • Jasper
    Jasper Member
    edited January 17
    Options

    Kind of. Ideally it would be a column that is created upon the load of new data (from SQL warehouse). So when I pull in a new dataset, or refresh it, the column appears as an additional column.

    There are workarounds, I can add a timestamp to the SQL query, or create a new version of the data using a workflow where I use the Metadata > Timestamp function.

    Thank you for quick and thorough response.

  • Tanj Jagpal
    Tanj Jagpal Administrator
    edited January 18
    Options

    Hey @Jasper "Ideally it would be a column that is created upon the load of new data (from SQL warehouse). So when I pull in a new dataset, or refresh it, the column appears as an additional column." - it would be great if you could raise this as an idea on the Ideas board. This is a great way to make the Product team aware of the problem you are experiencing and the impact this is currently having on you and your team!

  • Jasper
    Jasper Member
    Options

    ahhh I see, I thought this step gave the current timestamp, but you are saying it give the actual timestamp of when it was last loaded, that actually gives me exactly what I need. Thank you for that Seamus :)

  • Seamus Kyle
    Seamus Kyle Experian Contributor
    Options

    Hi Jasper

    Yes, it definitely gives the timestamp for when the data was loaded. (If you want the current timestamp there are other ways to do that).

    I am glad that this was useful.

Categories