Avoid creating a dupicate copy of data in media directory #48

annashcherbina · 2019-11-19T23:40:33Z

Thank you for this amazing tool!

I have searched the docs, and perhaps I am missing this, but it seems that a dataset must be copied to the media directory for it to be ingested into higlass with the --no-upload flag. Alternatively, if the flag is not provided, the data will be uploaded to the internal higlass databae and copied to the media folder? I tried to symlink my datasets to the media folder, but it looks like the links are not preserved within the higlass instance. I also tried creating sshfs mounts to datasets on a different server in the media folder, but that didn't work either. Is there a way to avoid copyng the data to visualize it in higlass? I have about 1000 bigwig tracks to load, so would like to avoid creating a redundant copy of the files if possible. Thank you!

pkerpedjiev · 2019-11-20T16:29:14Z

Hey! Symlinks unfortunately don't work because Docker can't follow them back to their origin.

One option is to set the media directory to the directory that has your bigiwig files using the --media-dir option. You can then upload using using the --no-upload function.

annashcherbina · 2019-11-20T17:13:48Z

Thank you for the quick response. I did try the latter option with --no-upload, but the challenge is that the tracks are on a different server than the one that higlass is running on. I tried to create an sshfs mount to the server with the tracks and set that mount to the --media-dir , but looks like the sshfs mount does not get preserved within the docker container either.

It looks like passing the --privileged flag can allow for propagation of sshfs mounts to docker, but when running higlass-manage start to create the server, the docker container gets created implicitly it seems, so there is no way to pass this flag? Is there a way to run higlass-manage start such that the sshfs mountpoint gets parsed as the media dir?

pkerpedjiev · 2019-11-20T17:22:45Z

Is there a way to run higlass-manage start such that the sshfs mountpoint gets parsed as the media dir?

So that's what --media-dir should be doing but if it doesn't work then that's a problem. I know of people that have used it with other network mounts. If the data is accessible via http, there may be another solution but I'll have a double check to make sure it works.

annashcherbina · 2019-11-20T17:45:43Z

With tag v0.6.34:

running the server with higlass-manage:

sshfs annashch@remoteserver:/path/to/my/data /srv/ssd/higlass/media2 
higlass-manage start -t /srv/ssd/higlass/tmp \
               -d /srv/ssd/higlass/data \
               --site-url 171.67.96.244 \
               -p 8989 \
               -n atlas \
               -m /srv/ssd/higlass/media2 \
               --public-data \
               --workers 40 \
               --use-redis \
               --version v0.6.34 \
               --redis-dir /srv/ssd/higlass/redis

Output:

Pulling redis:5.0.3-alpine
done
Pulling latest image... 
done
Data directory: /srv/ssd/higlass/data
Temp directory: ()
Starting... atlas 8989
Traceback (most recent call last):
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/containers/09dacf127143768b22c646dcd6cccc2d1832c56a07d66e2739ec649a22b7b726/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/users/annashch/miniconda3/envs/highglass/bin/higlass-manage", line 8, in <module>
    sys.exit(cli())
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/higlass_manage/start.py", line 105, in start
    redis_port)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/higlass_manage/start.py", line 278, in _start
    detach=True)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/models/containers.py", line 809, in run
    container.start()
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/models/containers.py", line 400, in start
    return self.client.api.start(self.id, **kwargs)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/users/annashch/miniconda3/envs/highglass/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("error while creating mount source path '/srv/ssd/higlass/media2': mkdir /srv/ssd/higlass/media2: file exists")
(highglass) annashch@nandi:/users/annashch/higlass$

running with docker :

docker run --detach \
       --publish 8989:80 \
       --volume /srv/ssd/higlass/data:/data \
       --volume /srv/ssd/higlass/media2:/media \
       --volume /srv/ssd/higlass/tmp:/tmp \
       --volume /srv/ssd/higlass/redis:/redis \
       --name atlas \
       -e SITE_URL=171.67.96.244 \
       --privileged \
       higlass/higlass-docker:v0.6.34

Gives:

docker: Error response from daemon: error while creating mount source path '/srv/ssd/higlass/media2': mkdir /srv/ssd/higlass/media2: file exists.

annashcherbina · 2019-11-20T18:01:18Z

(Sorry, had a typo in my docker command, fixed it above). It seems that case 2 (with docker run) also gives the error.

pkerpedjiev · 2019-11-20T21:18:24Z

Just to be sure, do you get that same error if you don't mount /srv/ssd/higlass/media as an sshfs filesystem?

annashcherbina · 2019-11-20T21:47:53Z

If I don't mount /srv/ssd/higlass/media , there is no error:

Pulling redis:5.0.3-alpine
done
Pulling latest image...
done
Data directory: /srv/ssd/higlass/data
Temp directory: ()
Starting... atlas 8989
Docker started: higlass-manage-container-atlas
sending request 1
Waiting to start (tilesets)...
sending request 2
Waiting to start (tilesets)...
sending request 3
Waiting to start (tilesets)...
sending request 4
Waiting to start (tilesets)...
sending request 5
Waiting to start (tilesets)...
sending request 6
Non 200 status code returned (502), waiting...
sending request 7
Non 200 status code returned (502), waiting...
sending request 8
Non 200 status code returned (502), waiting...
sending request 9
Non 200 status code returned (502), waiting...
sending request 10
Non 200 status code returned (502), waiting...
sending request 11
Non 200 status code returned (502), waiting...
sending request 12
Non 200 status code returned (502), waiting...
sending request 13
Non 200 status code returned (502), waiting...
sending request 14
Non 200 status code returned (502), waiting...
sending request 15
Non 200 status code returned (502), waiting...
sending request 16
Non 200 status code returned (502), waiting...
sending request 17
Non 200 status code returned (502), waiting...
sending request 18
Non 200 status code returned (502), waiting...
sending request 19
Non 200 status code returned (502), waiting...
sending request 20
Non 200 status code returned (502), waiting...
sending request 21
Non 200 status code returned (502), waiting...
sending request 22
Non 200 status code returned (502), waiting...
sending request 23
Non 200 status code returned (502), waiting...
sending request 24
Non 200 status code returned (502), waiting...
sending request 25
Non 200 status code returned (502), waiting...
sending request 26
Non 200 status code returned (502), waiting...
sending request 27
Non 200 status code returned (502), waiting...
sending request 28
Non 200 status code returned (502), waiting...
sending request 29
Non 200 status code returned (502), waiting...
sending request 30
Non 200 status code returned (502), waiting...
sending request 31
public_data: True
Replaced js file
Started

I got around the issue by modifying the source code of start.py to pass "privileged=True" to client.containers.run

This allowed me to run the sshfs mount command from within the running docker container.

pkerpedjiev · 2019-11-20T21:49:52Z

Oh, fantastic! Is there anything else to be done for this issue then?

annashcherbina · 2019-11-20T21:57:29Z

I guess the workaround is ok, but definitely subideal to have to modify the source code and re-run the mount manually from within the running docker container. Is this not a common use case (i.e. the tracks to visualize are not located on the same server that's running higlass)? Would you consider a pull request to provide a "--privileged" flag to higlass-manage start so that there is no need to modify the start.py file to ingest remote tracks?

pkerpedjiev · 2019-11-20T22:15:21Z

Is this not a common use case (i.e. the tracks to visualize are not located on the same server that's running higlass)?

It has certainly come up before. That's actually where the separate --media-dir option for higlass-server came from.

Would you consider a pull request to provide a "--privileged" flag to higlass-manage start so that there is no need to modify the start.py file to ingest remote tracks?

Absolutely! In fact, I'd be very grateful for it. Thanks for taking the time to investigate and make this work!!

pkerpedjiev mentioned this issue Nov 20, 2019

hard-linking fails if source and tmp folder are on different partitions or devices #3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid creating a dupicate copy of data in media directory #48

Avoid creating a dupicate copy of data in media directory #48

annashcherbina commented Nov 19, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019 •

edited

Loading

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

Avoid creating a dupicate copy of data in media directory #48

Avoid creating a dupicate copy of data in media directory #48

Comments

annashcherbina commented Nov 19, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019 • edited Loading

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019

pkerpedjiev commented Nov 20, 2019

annashcherbina commented Nov 20, 2019 •

edited

Loading