We're Hiring!

download original Files takes much time

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

download original Files takes much time

Postby saleht » Wed Jan 10, 2018 12:16 pm

Hi all,

during trying to trace the cause of the issue http://www.openmicroscopy.org/community ... f=4&t=8404

download original files takes really much time
steps:
- version 5.4.1
1. sigin to Omero using Omero web
2. select 5 images (size around 600 MB)
3. trigger download
- issue: Preparing zip for download takes for 600 Mb(this size after zipping ) about 7 min 25 sec which i believe is too much

i think the issue happens because of zipping large files, and that takes time. IS there a way to download all of them without zipping them (like Omero.insight) ?
any suggestions ?
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby wmoore » Thu Jan 11, 2018 12:26 pm

Hi,

Unfortunately it just takes time to download all the files into a temp directory on the web server then build a zip from them, although 7 minutes does seem a little long.

If the number of files is not too large, you can simply click on each individual file in the download dialog (see screenshot) to download each one directly from the OMERO server. This should be much more efficient (download should start immediately) since we don't need to store the files on the web server.

If the number of files is too high, it may be possible to trigger the download of all files using some JavaScript in the developer tools Console to open a new tab for each download link:
e.g.
Code: Select all
$("table a").each(function(){window.open($(this).attr('href'), "_blank")})

but this may be blocked by your browser popup blocker and it could also place a heavy load on your server so should be tried with caution.

Regards,

Will.
Attachments
Screen Shot 2018-01-11 at 11.40.13.png
Screen Shot 2018-01-11 at 11.40.13.png (91.55 KiB) Viewed 3540 times
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: download original Files takes much time

Postby saleht » Fri Jan 12, 2018 9:58 am

Hi will,
thx for your answer

Unfortunately it just takes time to download all the files into a temp directory on the web server then build a zip from them, although 7 minutes does seem a little long.


yeah i noticed that, when i try to download my tmp will be very big,
the question is, can i change the place of tmp folder, i want to move it to bigger partition, becasue root partition has 10 Gb free and this this critical.
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby wmoore » Mon Jan 15, 2018 4:04 pm

Hi,

OMERO.web uses python's tempfile module for creating the temp directory and zip file.
You can specify where this is by setting various environment variables as described: https://docs.python.org/2/library/tempf ... le.tempdir

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: download original Files takes much time

Postby saleht » Thu Jan 25, 2018 9:25 am

Hi will,
i did the following to change the tmp dir
import tempfile
tempfile.mktemp(dir='/../..')
and then checking the python tmp dir
tempfile.gettempdir()
i am getting the new dirctory
but after a while it will come back to old or Default one
i am not sure if what i did right
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby jmoore » Thu Jan 25, 2018 5:43 pm

Hi Saleh,

what do you mean, "after a while it will come back"? After you restart the server(s)? Something else?

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: download original Files takes much time

Postby saleht » Mon Jan 29, 2018 8:55 am

Hi Josh,

No even without a restart, maybe after 3 or 5 mins, will changed again Automatically
Cheers, Saleh
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby jmoore » Mon Jan 29, 2018 1:04 pm

Hi Saleh,

that sounds quite surprising. Can you try outlining your steps here for me, since I'm coming to this thread late?

We're talking about one server that you're running in production, right? Or do you have a test server?

How are you setting tempdir? What value are you using?

Try to give us as much detail as possible.

All the best,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: download original Files takes much time

Postby saleht » Mon Jan 29, 2018 2:20 pm

Hi Josh,

it is actually a test server, but i need to apply this on my production one.
steps:
1. open python terminal by writing python on Linux's terminal
2. import tempfile
3. tempfile.mkdtmp(dir='/data/pythonTmp')
4. tempfile.gettmpdir()


4 th step is to check the tmp directory

i am not sure if these steps is correct
cheers, Saleh
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby jmoore » Mon Jan 29, 2018 2:49 pm

Hi Saleh,

I understand. `tempfile.mkdtemp` does not change the return value of `tempfile.gettmpdir()` but rather returns you a new directory:

Code: Select all
cd /tmp
mkdir -p josh
python -c "import tempfile; print tempfile.mkdtemp(dir='/tmp/josh'); print tempfile.gettempdir()"


prints out:

Code: Select all
/tmp/josh/tmp7_yDQs
/tmp


whereas if I change my environment variable:

Code: Select all
cd /tmp
mkdir -p josh

export TMPDIR=/tmp/josh
python -c "import tempfile; print tempfile.mkdtemp(dir='/tmp/josh'); print tempfile.gettempdir()"


I see:

Code: Select all
/tmp/josh/tmpnK9vpy
/tmp/josh


Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Next

Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron