External system drop zone stopped working

James T Sidebotham
James T Sidebotham Contributor
in General

Hi,

The drop zones that I have set up for several subfolders for many files has stopped working, and even after a restart continues to not work. The log contains a lot (a lot!) of these messages:

INFO c.e.d.d.ExternalDirectoryWatcher [ExternalDirectoryWatcher-e3649cf8-de9f-42e2-8eec-edee6e98c157 1369857878] Error fetching file for external system: SAFE-DQ (SFTP) [Environment=Default] [externalSystemUuid=e3649cf8-de9f-42e2-8eec-edee6e98c157] - Specified file path is invalid.

The external system is accessible (I can export files to there). If the reasons why "Specified file path is invalid" is generated, that could help with troubleshooting.

Any ideas?

Thanks

Comments

  • Henry Simms
    Henry Simms Administrator
    edited October 1

    Did anything change on the Data Studio side (eg version upgrade, change in external dropzone config or the external system / credential themselves)? If not, it could be a change in the behaviour or folder structure of the 3rd party SFTP server although difficult to say what.

    One thing to note is that the Dropzone will be monitoring for files in a location based on your External System's Root Directory while files exported to the External System from Data Studio may be using a different Upload Directory if one has been specified. I have a test setup like this:

    image.png

    When I remove the "files" folder from my SFTP server itself, my Data Studio exports still work but my file watchers error with something different to yours:

    INFO c.e.d.d.ExternalDirectoryWatcher [ExternalDirectoryWatcher-fe68fb99-9772-472f-8903-52ab25738f1c 1794115766] Error fetching file for external system: rebex (SFTP) [Environment=Demo] [externalSystemUuid=fe68fb99-9772-472f-8903-52ab25738f1c] - File not found.

    I wasn't able to recreate the Specified file path is invalid error

  • James T Sidebotham
    James T Sidebotham Contributor

    Nothing has changed at all with the configuration or versions, so it's all the more baffling. No folders have changed on the SFTP server. I am wondering if some Firewall rule change could generate this error - I would have to check with the network group to see if something changed and now has impacted this. I am not seeing any 'time out' messages though, but maybe they just don't get logged.

    Is the ExternalDirectoryWatcher an off-the-shelf Java module, or written in-house by Experian?

  • Henry Simms
    Henry Simms Administrator

    It's using the JSCH library for this , which is where the error is coming from.

  • James T Sidebotham
    James T Sidebotham Contributor

    "com.experian.datastudio.datasource.ExternalDirectoryWatcherException: 19001: External system not found"

    This could support the Firewall theory? At least for certain request types (whatever its doing in the watcher) - I can test connection successfully in Aperture and send files there without issue.

  • James T Sidebotham
    James T Sidebotham Contributor

    @Henry Simms When I set up the external system, I had the Root Directory set to / and then in the dataset definitions, I set the Folder to use as dropzone set to FromUsers and this has always worked fine for over a year.

    Obviously I am having errors now, but if I create a new external system definition and set the Root Directory to /FromUsers and the dataset drop zone is set to blank (it shows prefix of /FromUsers) then the dropzone works (for new files I am using for testing).

  • Henry Simms
    Henry Simms Administrator

    Thanks - I've tested various combinationRoot Directory (External System config) vs Folder to use as dropzone (Dataset config), and using a folder structure where parts of the path are set in both settings, but I wasn't able to reproduce your error.

    Something close to your original setup is working as expected for me:
    Root Directory just "\" (my SFTP is on Windows)

    image.png

    Folder to use as dropzone is set to a path relative to root

    image.png

    The dropzone in this setup is working as expected and files are loaded to the Dataset

    image.png
  • James T Sidebotham
    James T Sidebotham Contributor

    What happens if you leave the dataset dropzone configured for "files/drop" but on the SFTP server you delete the subfolder "/drop"?

  • Henry Simms
    Henry Simms Administrator

    In the scenario where the "drop" subfolder is removed I just get:

    2025-10-08 12:15:18,708 INFO c.e.d.d.ExternalDirectoryWatcher [ExternalDirectoryWatcher-fe68fb99-9772-472f-8903-52ab25738f1c 933010869] Error fetching file for external system: rebex (SFTP) [Environment=Demo] [externalSystemUuid=fe68fb99-9772-472f-8903-52ab25738f1c] - File not found.

    I see above that you reported seeing "External system not found". Is that still the case? That suggests that the link between the Dataset and the External System it came from is lost. It would be worth looking at the Sources:

    image.png
  • James T Sidebotham
    James T Sidebotham Contributor

    Thanks for checking. The system did seem to be 'not found', leading me to believe it was a network issue. However, I added the external system again and within each dataset I changed the source to use the new external system (and dropzone), and the files are coming in now.

    I have support looking at the logs, but I am leaning towards some sort of internal repository problem (issues began after a restart) where all the metadata for the dropzones might have been corrupted?

    I am back working now, after re-configuring about 60 dropzones, and fingers crossed don't see this again.

  • Henry Simms
    Henry Simms Administrator

    Glad you're back up and running after updating all dropzones. I've not been able to reproduce but will continue investigating. Can you confirm what version of Data Studio you're on, for reference?

  • James T Sidebotham
    James T Sidebotham Contributor

    Sure, 3.0.8.61

    thanks

Categories