Error mounting volumes when launching ADS
Hiya,
We have recently tried to update our ADS image to the latest version (2.14.1) and are now unable to start the container. We are getting the error "Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/administrant/docker-compose/ApertureDataStudio/server.properties" to rootfs at "/opt/ApertureDataStudio/server.properties": mount /home/administrant/docker-compose/ApertureDataStudio/server.properties:/opt/ApertureDataStudio/server.properties (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"
I have looked at the source folder and the destination folder, in both cases server.properties is a file, and the experian user is the owner with rw permissions.
We have tried removing these files from the system and replacing them using the setup.sh file but this has not affected the issue.
Please can you advise on the possible causes of this error and how we can resolve it.
Answers
-
1
-
Hi @BAmos, sorry to hear the upgrade is not going well. Hopefully get can get this sorted out quickly.
Typically this is caused when you create the container with a mapped file missing from the host - docker will create a directory in the container which remains until the container is removed, and causes the issue observed if the file is then created in the host.
If this sounds like it could be the case, please remove the container and recreate it.
Let me know how you get on.
Regards,
Ian
0 -
Hi Ian,
Thank you very much for your help. I've removed the container and recreated it and the services are now all starting up and running. When trying to log in though it looks like all of our passwords have reset (including Administrator). I'm trying to follow the process for resetting this outlined here: but am encountering the following error on this command.
Command: experian@95ecd0a29594:~$ java -jar servicerunner-2.14.1.jar STARTUP RESETADMINPASSWORD
Error: 15:41:43.768 [main] FATAL com.experian.datastudio.common.exceptions.KillSwitch - A fatal error occurred and the system has been terminated: FAILED_TO_START - 109
Full log preceding the error is in the attached.I'll keep trying to figure this out but please let me know if you have any guidance for this process.
Many thanks,
Ben
0 -
Hi Ben, upgrading will not reset any passwords. In my experience, this is usually caused when the repository is mapped incorrectly so it has created a new repo with the default password.
It appears that you have two very similar locations on your host machine. My guess is you're pointing to the wrong one. Can you check the mappings are correct and they are pointing to the location used previously?
Thanks,
Ian
0 -
Hi Ian, are these the mapping for the volumes in the .yml file?
If not please can you let me know what the mappings are and where to find them please?
Thanks,
Ben
0 -
Hi Ben, yes the mappings are those defined in the .yml file.
Regards,
Ian
0 -
Hi Ian,
Here's the yml file that I believe is launching the application. There is also another below which I believe we made as a back up before running the update which is also below. As far as I can tell these are all mapping the same way. Is there anything in here that looks different to you?
As a note, our engineer who originally set up our instillation has moved on and I'm running this for the first time so there may be some obvious things that I am missing.
Current:
Backup:
Many thanks,
Ben
0 -
Hi Ben,
Ah ok. The most important file in the mapped directory structure is the repository.db file. This lives in the data/repository folder.
Perhaps you could look in both locations to see if these files are different, and if so, use the mappings for the latest/biggest one.
Regards,
Ian
0 -
Hi Ian,
I've had a look at the repository folder in both locations, here's a screen shot of it:
From what I can tell it is only the "repository.db" file that's important here, the others being our backups that we make before updates.
I had a look in the .yml file and couldn't see any lines that map these files. Is that instruction held elsewhere to check?
0 -
Hiya, the other files are important (minus the pre-upgrade ones which are kept just in case of problems during upgrade - so worth keeping for now). They are mapped into the container via the parent /ApertureDataStudio directory.
The /home/administrant directory you were mapping to appears to contain much smaller files (I would guess they are new 'empty' ones), whereas the other directory (in /opt) is much more healthy looking and appears a better candidate, so probably best to repoint all your mounted files/volumes to here instead.
Regards,
Ian
0 -
Hi Ian,
I've tried copying the repository.db file from the op/ location into the docker-compose/ location and have restarted the container. It's launching the applications still, but now I can't access the web UI (although the find duplicates workbench is still accessible). I've looked at the datastudio.log file and the ERROR lines are:
"2024-06-07 16:02:00,912 ERROR c.e.d.s.ServiceRunner [Thread-0] Catching
java.lang.NullPointerException: Cannot invoke "com.experian.datastudio.licensing.LicenseChecker.shutdown()" because "this.licenseChecker" is null
at com.experian.datastudio.servicerunner.subsystem.EventManagementSubsystem.shutdown(EventManagementSubsystem.java:199) ~[servicerunner-2.14.1.jar:2.14.1.169-478903]
at com.experian.datastudio.servicerunner.ServiceRunner$ShutdownHook.stopEverything(ServiceRunner.java:686) ~[servicerunner-2.14.1.jar:2.14.1.169-478903]
at com.experian.datastudio.servicerunner.ServiceRunner$ShutdownHook.run(ServiceRunner.java:667) ~[servicerunner-2.14.1.jar:2.14.1.169-478903]
2024-06-07 16:02:00,912 FATAL c.e.d.s.ServiceRunner [Thread-0] Fatal error encountered: Cannot invoke "com.experian.datastudio.licensing.LicenseChecker.shutdown()" because "this.licenseChecker" is null"and
"2024-06-07 16:03:41,483 ERROR c.e.d.d.ExternalDirectoryWatcher [ExternalDirectoryWatcher-fba4ed90-d4c8-401c-8c54-0a3716f4b801 2089123471] Error fetching file for external system: crdaaperturedatastudio (AZURE) [Environment=Production] [externalSystemUuid=fba4ed90-d4c8-401c-8c54-0a3716f4b801]
java.io.IOException: java.lang.IllegalArgumentException: Invalid connection string.
at com.experian.datastudio.common.utility.IOUtils.toIOException(IOUtils.java:25) ~[common-2.14.1.jar:?]
at com.experian.datastudio.datasource.connection.cloudssystem.AzureConnection.createStorageAccount(AzureConnection.java:290) ~[datasource-0.1.0.jar:?]
at com.experian.datastudio.datasource.connection.cloudssystem.AzureConnection.getStorageAccount(AzureConnection.java:69) ~[datasource-0.1.0.jar:?]
at com.experian.datastudio.datasource.connection.cloudssystem.AzureConnection.getClient(AzureConnection.java:318) ~[datasource-0.1.0.jar:?]
at com.experian.datastudio.datasource.connection.cloudssystem.AzureConnection.getSystemSources(AzureConnection.java:195) ~[datasource-0.1.0.jar:?]
at com.experian.datastudio.datasource.ExternalDirectoryWatcher.fetch(ExternalDirectoryWatcher.java:157) ~[pserver-2.14.1.jar:?]
at com.experian.datastudio.datasource.ExternalDirectoryWatcher.poll(ExternalDirectoryWatcher.java:143) ~[pserver-2.14.1.jar:?]
at com.experian.datastudio.datasource.ExternalDirectoryWatcher.runImpl(ExternalDirectoryWatcher.java:112) ~[pserver-2.14.1.jar:?]
at com.experian.datastudio.common.metrics.MonitoredThread.run0(MonitoredThread.java:56) ~[common-2.14.1.jar:?]
at com.experian.datastudio.common.metrics.MonitoredThread.run(MonitoredThread.java:40) ~[common-2.14.1.jar:?]
Caused by: java.lang.IllegalArgumentException: Invalid connection string.
at com.microsoft.azure.storage.CloudStorageAccount.parse(CloudStorageAccount.java:275) ~[azure-storage-8.6.0.jar:?]
at com.experian.datastudio.datasource.connection.cloudssystem.AzureConnection.createStorageAccount(AzureConnection.java:288) ~[datasource-0.1.0.jar:?]
... 8 more"
Note, this last one repeats multiple times.I'm assuming that the first one is the route cause of this issue, and will be to do with file mapping still.
What would you suggest to do next?Many thanks, and hope you have a great weekend.
Ben0 -
Hi @BAmos, my feeling is that the /opt location is the correct location to be mapping your docker volumes to, rather than copying individual files into the partial/new repository in the …/docker-compose directory.
The first error is to do with the license - not sure what the problem is there, but the license file is probably missing from the mapped volumes. The second error is because the external connection's keys are unable to be decrypted as the required files are not present in that directory.
The /opt files are the good ones by the looks of it. The ones in …/docker-compose are not.
If I were you, I'd just change the mappings in the docker-compose.yml to /opt and see if it all starts properly.
Regards and have a great weekend too,
Ian
0 -
Thanks Ian, really appreciate your help on this. Will let you know how it goes.
0 -
No luck unfortunately, this is how I I've rewritten the .yml file for reference. I'll pick this all back up again on Monday morning.
0 -
@BAmos Hi Ben, it would probably be quicker to resolve this in a call. Could you email me directly at ian.hayden@experian.com and we can set something up for later today.
Regards,
Ian
0
