Workflow - Export to Dataset, then Refresh

George Brown
George Brown Member
edited December 2023 in General

We have a workflow that is compiling some results from several steps (into a union), and then exporting to a JDBC external system with a defined dataset.

As a last step, we have linked the export to the source of the defined dataset assuming that once it has exported the data, it would refresh the dataset, and we could see the latest data and use it for charts, reports, etc.

https://ibb.co/nCTWLMh

However we noticed in a recent run of our workflow, it seems that the refresh of the dataset may be running at the same time as the export. We assumed that the workflow would wait for the export to complete before refreshing the dataset, however it seems it isn't. Are we doing something wrong here?

https://ibb.co/KxXMGF9


Tagged:

Best Answer

  • George Brown
    George Brown Member
    Answer ✓

    @Ian Hayden , we ended up going with just a notification that uses the system event "Workflow completed" with a filter for the workflow name. This saves us from having to specify a custom event and adding the fire event step at the end of our primary workflow. We have confirmed it works the same either way. Is there a reason to choose the fire event step with custom event, over the way we did it?

    ADS Workflow 3 - 2022_09_21.png


Answers

  • Ian Hayden
    Ian Hayden Experian Super Contributor

    Hi @George Brown,

    I can't access the images you've attached so I can't quite grasp how you've linked the export to the source of the defined dataset. Would you mind explaining how you've done this?

    Thanks,

    Ian

  • Josh Boxer
    Josh Boxer Administrator

    Unfortunately URL:https://ibb.co is blocked by Experian, if you can paste or attached the images to a comment please

  • George Brown
    George Brown Member

    If I try to paste, I get an error that says "Permission Problem: You need the Garden.Uploads.Add permission to do that."

    I don't see any option for attaching either; just the two buttons for emoji picker or insert media.

    How do I successfully paste or attach?

  • Tanj Jagpal
    Tanj Jagpal Administrator
    edited September 2022
    Hi @George Brown I have now fixed this permissions issue for you. You may need to logout out and in again for the fix to complete.
  • George Brown
    George Brown Member

    @Ian Hayden , below are the screen shots.

    ADS Workflow - 2022_09_21.png


    ADS Workflow 2 - 2022_09_21.png


  • Ian Hayden
    Ian Hayden Experian Super Contributor

    Hi @George Brown,

    This is actually a really creative way of using the product that we had not anticipated! I think what will happen here is one of two things; either the source step's refresh will not run (as it could be optimised out) or it will run at the same time as the other steps. It sounds like you're encountering the latter.

    A better solution (or rather, a solution we have anticipated) would be to add a fire event step (with the fire after workflow option checked) after the export (and instead of the source step), setup a notification on this event that runs another workflow that contains the source step. This ensures that the workflow finishes before the ADS_Duplicate_NK_Check data is refreshed.

    I hope this makes sense - please let me know if not or if you have any other questions.

    Regards,

    Ian

  • George Brown
    George Brown Member

    @Ian Hayden , thanks! I'll work on applying the changes to our workflow and will reach out if I run into any issues or have any questions.

  • Josh Boxer
    Josh Boxer Administrator

    Agree, very interesting and not something I have seen, but will test myself.

    Another approach that might work is a Take snapshot to write the data to a new or existing Dataset.

    Rather than linking the step to the export, I would expect to see it forked from the previous map to target step, so that the step is writing to both outputs.

    -Josh

  • Ian Hayden
    Ian Hayden Experian Super Contributor

    Hi @George Brown

    Your way is better actually, and accomplishes the same thing.

    The dataset refresh should be reliable now.

    Regards,

    Ian

Categories