find duplicate update row limitation

Mahulima
Mahulima Contributor
in General

Is currently is there any limitation in the latest version ?

I am getting error in my existing workflow . Please suggest where I need to change to set the limit

image.png

Tagged:

Answers

  • Josh Boxer
    Josh Boxer Administrator

    Hello, I don't think this is a new thing. There is a setting:

    image.png

  • Mahulima
    Mahulima Contributor

    @Josh Boxer Update record limit for us also set to this

    image.png

  • Josh Boxer
    Josh Boxer Administrator
    edited October 11

    I assume the Workflow you ran exceeded the limit you have set (which I believe is the default value and has been this for a long time)

  • Mahulima
    Mahulima Contributor

    @Josh Boxer we have same default setting in production environment where the workflow is running fine with same amount of data.

    Production we have : 2.14.4

    Dev we have : 2.15.4

    additionally difference is in dev we have ADS and FD in same server and its Production we have it in different servers. This is happening probably because of some bug in this current version I think and we had faced this issue earlier too when out Prod was running fine and upgrade broke are dev . But then again with next version upgrade it was solved. But now it seems that this new version is not compatible again

  • Josh Boxer
    Josh Boxer Administrator
    edited October 14

    "There is a 1 million record processing limit when using the embedded Find Duplicates server. To process volumes above 1 million records, we recommend configuring a separate Find duplicates server instance."

    https://docs.experianaperture.io/data-quality/aperture-data-studio-v2/find-duplicates-step/installing-a-separate-instance/

    If you think there is any issue with your setup then probably best to raise a support ticket. It will be helpful if you can provide the support team with more information like the number of records you are trying to process for example: https://docs.experianaperture.io/more/contact-us/

  • Mahulima
    Mahulima Contributor

    @Josh Boxer Our existing workflow is running for more than 2 years in Production. Also when earlier release broke our Development workflows we were in touch with the Support team for 4+ months and did not find any solution but then next upgrade solved the issue in development . But now again same situation is arising.

  • Mahulima
    Mahulima Contributor

    @Josh Boxer we have 2.5 M + records to Process , when we are running the workflow with all data with default 1 M settings its failing but when we are running with close to 1M records its running successfully. But if I change the Settings and then running in dev with 2.5M+ records also its running fine .

    image.png

Categories