We're Hiring!

download original Files takes much time

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Re: download original Files takes much time

Postby saleht » Tue Jan 30, 2018 1:27 pm

Hi Josh,
yes you are correct, but for me both of them does not works (doesn't change the python temp dir)
neither
export TMPDIR=/data/tmp
nor
mkdtemp(dir='/data/tmp')
and i need to change my python tmp dir because when i am trying to download original files from OMERO.web, my main partition (Operation system partition) will be almost full and i am trying let python use another partition for copying and compressing the data which will be downloaded from the user.
any suggestions
cheers, Saleh
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby jmoore » Tue Jan 30, 2018 1:50 pm

Hi Saleh,

saleht wrote:but for me both of them does not works (doesn't change the python temp dir)


Don't work for you under what conditions? Can you show us more steps of what you're trying to do? Have you seen the temp files? Are they under the "omero" directory? If so, this is likely a case of needing to set OMERO_TMPDIR before starting up the server. Alternatively, you can symlink ~/omero/tmp to another location.

~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: download original Files takes much time

Postby saleht » Wed Jan 31, 2018 9:57 am

Hi Josh,
on my Test server the results of the command :
Code: Select all
df  -h
is
Code: Select all
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   17G  4,3G   13G  26% /
devtmpfs               3,9G     0  3,9G   0% /dev
tmpfs                  3,9G  4,0K  3,9G   1% /dev/shm
tmpfs                  3,9G  8,5M  3,9G   1% /run
tmpfs                  3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/sdb1               50G  5,2G   42G  12% /data
/dev/sda1             1014M  184M  831M  19% /boot
tmpfs                  783M     0  783M   0% /run/user/0

during the download of original images, the used size of
Code: Select all
/
directory will be always increased .
i have the partition
Code: Select all
/data
has enough empty space, which i want to used as python(OMERO) tmp directory, that is means the used size of
Code: Select all
/
should not any more - during the download procedure- increased.

Code: Select all
ls -la /data
total 32
drwxr-xr-x   6 root  root  4096 16. Jan 09:55 .
dr-xr-xr-x. 18 root  root   256 20. Jun 2017  ..
drwx------   2 root  root 16384 20. Jun 2017  lost+found
drwxr-xr-x.  9 omero root  4096 20. Jun 2017  OMERO
drwxr-xr-x   9 omero root  4096  9. Jan 15:14 OMERO.server-5.4.1-ice36-b75
drwxr-xr-x   5 omero root  4096 31. Jan 10:41 PyTmpDir


Code: Select all
bin/omero admin diagnostics

Code: Select all
===============================================================================
OMERO Diagnostics (admin) 5.4.1-ice36-b75
================================================================================
       
WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to 5.4.3. See http://downloads.openmicroscopy.org/latest/omero for the latest version.

Commands:   java -version                  1.8.0     (/usr/bin/java)
Commands:   python -V                      2.7.5     (/home/omero/omerowebvenv/bin/python -- 2 others)
Commands:   icegridnode --version          3.6.3     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.6.3     (/usr/bin/icegridadmin)
Commands:   psql --version                 9.6.3     (/usr/bin/psql)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 3565, enabled)
Server:     DropBox                        active (pid = 3577, enabled)
Server:     FileServer                     active (pid = 3587, enabled)
Server:     Indexer-0                      active (pid = 3610, enabled)
Server:     MonitorServer                  active (pid = 3579, enabled)
Server:     OMERO.Glacier2                 active (pid = 3592, enabled)
Server:     OMERO.IceStorm                 active (pid = 3600, enabled)
Server:     PixelData-0                    active (pid = 3582, enabled)
Server:     Processor-0                    active (pid = 3620, enabled)
Server:     Tables-0                       active (pid = 3625, enabled)
Server:     TestDropBox                    inactive (enabled)

Log dir:    /data/OMERO.server-5.4.1-ice36-b75/var/log exists
Log files:  Blitz-0.log                    140.0 MB      errors=23   warnings=68 
Log files:  DropBox.log                    25.0 KB       errors=1    warnings=8   
Log files:  FileServer.log                 3.0 KB       
Log files:  Indexer-0.log                  581.0 KB      errors=3    warnings=1   
Log files:  MonitorServer.log              13.0 KB     
Log files:  OMEROweb.lock                  0.0 KB       
Log files:  OMEROweb.log                   4.0 MB       
Log files:  OMEROweb.log.1                 5.0 MB        errors=1    warnings=4   
Log files:  OMEROweb.log.2                 5.0 MB        errors=1    warnings=1   
Log files:  OMEROweb.log.3                 5.0 MB        errors=0    warnings=3   
Log files:  OMEROweb.log.4                 5.0 MB       
Log files:  OMEROweb.log.5                 5.0 MB        errors=0    warnings=1   
Log files:  OMEROweb.log.6                 5.0 MB       
Log files:  OMEROweb.log.7                 5.0 MB        errors=0    warnings=12 
Log files:  PixelData-0.log                219.0 KB      errors=3    warnings=2   
Log files:  Processor-0.log                1.0 MB        errors=0    warnings=22 
Log files:  Tables-0.log                   20.0 KB       errors=0    warnings=4   
Log files:  TestDropBox.log                n/a
Log files:  g_error.log                    1.0 KB       
Log files:  master.err                     9.0 KB        errors=6    warnings=8   
Log files:  master.out                     0.0 KB       
Log files:  Total size                     184.34 MB


Environment:OMERO_HOME=(unset)             
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:OMERO_USERDIR=(unset)         
Environment:OMERO_TMPDIR=/data/PyTmpDir   
Environment:PATH=/home/omero/omerowebvenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
Environment:PYTHONPATH=(unset)             
Environment:ICE_HOME=(unset)               
Environment:LD_LIBRARY_PATH=(unset)       
Environment:DYLD_LIBRARY_PATH=(unset)     

OMERO SSL port:4064                           
OMERO TCP port:4063                           
OMERO data dir:'/data/OMERO'                  Exists? True   Is writable? True
OMERO temp dir:'/data/PyTmpDir'               Exists? True   Is writable? True   (Size: 0)

JVM settings: Blitz-${index}                -Xmx1230m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index}              -Xmx820m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index}            -Xmx1230m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index}           -Xmx820m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

even my
Code: Select all
Environment:OMERO_TMPDIR=/data/PyTmpDir
configured to used
Code: Select all
/data
partition,
still my
Code: Select all
/
parition -during the download - increased.

Code: Select all
bin/omero config get

Code: Select all
Debug=true
omero.data.dir=/data/OMERO
omero.db.name=omero_database
omero.db.pass=********
omero.db.user=db_user
omero.jvmcfg.append=-Dcom.sun.jndi.ldap.connect.pool.timeout=500
omero.jvmcfg.remove=-Dcom.sun.jndi.ldap.read.timeout=5000
omero.ldap.base=dc=ad,dc=hhu,dc=de
omero.ldap.config=true
omero.ldap.new_user_group=default
omero.ldap.password=********
omero.ldap.referral=ignore
omero.ldap.urls=ldap://ldaps.ad.hhu.de:3268
omero.ldap.user_mapping=omeName=sAMAccountName,firstName=givenName,lastName=sn,email=mail
omero.ldap.username=SVC_Omero
omero.security.ignore_case=true
omero.web.application_server.max_requests=0
omero.web.apps=["omero_figure", "first", "omero_fpbioimage"]
omero.web.debug=true
omero.web.open_with=[["Image viewer", "webindex", {"supported_objects": ["image"], "script_url": "webclient/javascript/ome.openwith_viewer.js"}], ["omero_figure", "new_figure", {"supported_objects": ["images"], "target": "_blank", "label": "OMERO.figure"}], ["omero_fpbioimage", "fpbioimage_index", {"supported_objects": ["image"], "script_url": "fpbioimage/openwith.js", "label": "FPBioimage"}]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Figure", "figure_index", {"target": "_blank", "title": "Open Figure in new tab"}], ["Booking", "https://booking-cai.hhu.de/"]]
omero.web.wsgi_timeout=60
omero.web.wsgi_worker_connections=1000
omero.web.wsgi_workers=5


i hope, it is now more clear, if some info still missing to understand the situation, i would like to add it :)
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby jmoore » Wed Jan 31, 2018 10:55 am

Hi Saleh,

thanks for trying OMERO_TMPDIR. After further digging, I've found that this code doesn't use the variable as I was expecting. But I also confirmed that OMERO.web does use TMPDIR as expected. I tested this in docker as below:


Code: Select all
$ git clone git://github.com/openmicroscopy/omero-test-infra

$ git diff
diff --git a/.env b/.env
index 54b6c07..e073aeb 100644
--- a/.env
+++ b/.env
@@ -5,9 +5,9 @@ POSTGRES_IMAGE=postgres
POSTGRES_VERSION=9.6
POSTGRES_PASSWORD=postgres
OMERO_SERVER_IMAGE=openmicroscopy/omero-server
-OMERO_SERVER_VERSION=5.4
-OMERO_SERVER_TCP=4063
-OMERO_SERVER_SSL=4064
+OMERO_SERVER_VERSION=5.4.1
+OMERO_SERVER_TCP=24063
+OMERO_SERVER_SSL=24064
OMERO_WEB_IMAGE=openmicroscopy/omero-web-standalone
-OMERO_WEB_VERSION=5.4
-OMERO_WEB_PORT=4080
+OMERO_WEB_VERSION=5.4.1
+OMERO_WEB_PORT=24080
diff --git a/docker-compose.yml b/docker-compose.yml
index a376a7e..b892800 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -24,6 +24,9 @@ services:
       - CONFIG_omero_upgrades_url=
       - CONFIG_Ice_IPv6=0
       - PYTHONPATH=/opt/omero/web/OMERO.web/lib/python
+      - TMPDIR=/josh
+    volumes:
+      - ./tmp:/josh
     ports:
       - "${OMERO_WEB_PORT}:4080"


Then create the tmp directory and start it up:

Code: Select all
mkdir tmp
chmod 777 tmp
docker-compose up -d


View http://localhost:24080 -- afterwards, I see temp files being created in the new `./tmp`. (I also modified line 130 of that code to use `logger.error` so I could clearly see the message.)


So, I'd suggest one of a two options (both is ok as well):

  • Create a separate volume for your /tmp directory which is what OMERO.web is currently using. (This is a generally good & safe thing to do)
  • Make sure that you're starting OMERO.web with `TMPDIR` set to some other value. If you are using systemd, then the systemd process will need to set up the variable.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: download original Files takes much time

Postby saleht » Wed Jan 31, 2018 12:30 pm

Hi Josh,

Make sure that you're starting OMERO.web with `TMPDIR` set to some other value. If you are using systemd, then the systemd process will need to set up the variable.


yes TMPDIR it works for me now, i did the following and everything works fine
Code: Select all
bin/omero web stop
export TMPDIR=/data/newdir
bin/omero web start


Thx you guys
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Re: download original Files takes much time

Postby mtbc » Wed Jan 31, 2018 8:22 pm

Great, thank you for letting us know Saleh how you got it working!

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: download original Files takes much time

Postby saleht » Thu Feb 01, 2018 9:08 am

Hi Mark

like Josh said update the .service file
Code: Select all
Environment="TMPDIR=/mnt/data/PyTmp"


i mean the OMERO (Python) temporary directory changed, the long time issue -during the download original files -still reproducible. it is my fault that i mixed two issues in one post. but i was blocked because of my root directory running out of space when trying to download. Now i will try to play around time issue.

Cheers, Saleh
saleht
 
Posts: 96
Joined: Wed Nov 16, 2016 1:06 pm

Previous

Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest