We're Hiring!

Upgrade from 4.4 to 5.0

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

Upgrade from 4.4 to 5.0

Postby ppouchin » Thu Feb 27, 2014 5:41 pm

Hello !

I've seen that OMERO 5 is now out, and this is great !
I'm really eager to use it and upgrade our server... !

However, I was wondering one thing: when I first set up the server (years ago), I chose to use the DropBox feature as a way to archive original files (and let them easily accessible to the users).

Of course, I could just keep the "pixels" as they are stored now into OMERO 5, but in order to free some space, I was wondering if there would be a way to use the original files stored in the DropBox to replace the "Pixel" data while preserving all the annotations... like... a python script re-importing all the images and comparing their names to re-annotate them... ? (brute-force style)

I have a feeling it's going to be hard, and probably not worth it... but still... I wanted to ask.
ppouchin
 
Posts: 98
Joined: Thu Dec 02, 2010 2:08 pm

Re: Upgrade from 4.4 to 5.0

Postby jmoore » Thu Feb 27, 2014 7:41 pm

We haven't (yet) developed anything to do this automatically. Do you know the mapping from files under DropBox to the images in OMERO? What file formats do you have? If you have some less critical data that you would be willing to test this on, we could try to come up with a script that would work.

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Upgrade from 4.4 to 5.0

Postby ppouchin » Fri Feb 28, 2014 10:43 am

jmoore wrote:Do you know the mapping from files under DropBox to the images in OMERO?


What do you mean by "mapping"... ? I'm not sure I understand...

Each user's home resides in the DropBox folder, and when they put a file there, it's automatically imported into OMERO. All I know is that the only link between the original files and the images in OMERO is their name: the image name is the full path to the file (e.g. "/data/OMERO/DropBox/<username>/path/to/the/image"), although some images might have been renamed to only keep the relative path from the user's folder...

jmoore wrote:What file formats do you have? If you have some less critical data that you would be willing to test this on, we could try to come up with a script that would work.

Cheers,
~Josh.


In the DropBox folder, we mainly have LIF from Leica (2643) and (old) LSM from Zeiss (2878), but there are TIFF as well (1173). A few PNG (450), JPG (153), MDB (69) and some rare IMS (40) also made it to the server.

For now, we only use 5.1 TB for both DropBox and OMERO. I have access to a bigger server with several TB available, so I may do a backup of the OMERO data (folder & database) before trying anything.
I can also use a VM to do some tests... Although I cannot do that for 2 TB of data...
ppouchin
 
Posts: 98
Joined: Thu Dec 02, 2010 2:08 pm

Re: Upgrade from 4.4 to 5.0

Postby jmoore » Fri Feb 28, 2014 2:55 pm

ppouchin wrote:What do you mean by "mapping"... ? I'm not sure I understand...


By "mapping", I mean knowing how to programmatically take the names under /data/OMERO/DropBox and find the images that were imported from them. If use of Image.name works, that could be a start. Alternatively, you would need to do them in batches, passing the directory and the image IDs to the script.

In the DropBox folder, we mainly have LIF from Leica (2643) and (old) LSM from Zeiss (2878), but there are TIFF as well (1173). A few PNG (450), JPG (153), MDB (69) and some rare IMS (40) also made it to the server.


This is good to know, thanks. Let us have a look at we'll let you know.

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Upgrade from 4.4 to 5.0

Postby ppouchin » Fri Feb 28, 2014 3:04 pm

Thank you for your quick answers. :D

jmoore wrote:By "mapping", I mean knowing how to programmatically take the names under /data/OMERO/DropBox and find the images that were imported from them. If use of Image.name works, that could be a start. Alternatively, you would need to do them in batches, passing the directory and the image IDs to the script.


In the past, I used a small & dirty script to generate some commands in order to force a re-import in case DropBox had ceased to function and imports had been dropped.
It did work, but I've not used that for quite some time.
Also, there was a limit: if a file had been deleted in OMERO but was still in the folder, I was not able to know that it had been imported and deleted.

It did allow me to match images with files, but it may not be helpful in this case.

Edit: Also, I realize now that it might have been a bit faster had I removed matched images from the list of images. Although there might be even faster ways to do that than a double loop... :oops:
Last edited by ppouchin on Fri Feb 28, 2014 3:11 pm, edited 2 times in total.
ppouchin
 
Posts: 98
Joined: Thu Dec 02, 2010 2:08 pm

Re: Upgrade from 4.4 to 5.0

Postby jmoore » Fri Feb 28, 2014 3:06 pm

I'll certainly take a look. Cheers! ~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Upgrade from 4.4 to 5.0

Postby ppouchin » Fri Feb 28, 2014 3:26 pm

Also, a user recently had to delete around 3000 images, and I noticed it took quite some time, and that it did not complete properly. Doing that by batches of around 700~800 worked better.

So I'm curious if we are able to somehow import images in OMERO 5 and copy the annotations from the old images to the newly imported files, it might be necessary to slow both the import and delete processes (through pauses every couple hundreds of images)...

Otherwise, I fear the server may not be able to withstand the load...

But maybe I'm shortsighted and don't see the proper way to go at it...
ppouchin
 
Posts: 98
Joined: Thu Dec 02, 2010 2:08 pm

Re: Upgrade from 4.4 to 5.0

Postby wmoore » Mon Mar 03, 2014 11:38 am

Hi,

You might find some useful info in this post: viewtopic.php?f=4&t=7397 which is mostly about performance of "Move group" but also applies to large Deletes.

I'm afraid we don't have any built-in functionality for replacing one image with a newly imported one.
It should be easy enough to write a script to transfer annotations from old images to new ones.
Here is one I wrote a while back for taking Tags from one Dataset and applying them to the contained Images or other Datasets or Images. https://gist.github.com/will-moore/5777962. However, it doesn't quite fit your requirements, since you'd need to do some matching of images by name to transfer Tags from one to the other.
You'd also lose any info about who added the annotations and when.

NB: I noticed that in your script you filter "Images I own" after retrieving them from OMERO. A better way of doing this would be:

Code: Select all
userId = conn.getUserId()
params = omero.sys.Parameters()
params.theFilter = omero.sys.Filter()
params.theFilter.ownerId = rlong(userId)
images = conn.getObjects("Image", params=params)
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Upgrade from 4.4 to 5.0

Postby jmoore » Fri Apr 04, 2014 12:39 pm

Please follow https://trac.openmicroscopy.org.uk/ome/ticket/12176 to monitor the progress of any 4.4/5.0 migration work.

Thanks,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Upgrade from 4.4 to 5.0

Postby ppouchin » Fri Apr 11, 2014 8:59 am

jmoore wrote:Please follow https://trac.openmicroscopy.org.uk/ome/ticket/12176 to monitor the progress of any 4.4/5.0 migration work.

Thanks,
~Josh


Ok. Thank you.
(I'll try to do some tests in the weeks/months to come...)
ppouchin
 
Posts: 98
Joined: Thu Dec 02, 2010 2:08 pm

Next

Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest