PO Release SOP Page

This page is a place to list all the steps we need to take for the database releases.

This page is under development: Let's not reinvent the wheel every 4 months.

Quality Checks and Internal Review

Before each release, ontology editors and curators should run some qc checks. In fact, these should be done on a regular basis in between releases, but it is crucial to do them immediately before a release.

• Run the reasoner to remove any redundant links.

• Do a search for extra space and odd characters in terms, definitions and dbxrefs (see below:).

• The whole POC team should take a little time to do a internal review. Look for any inconsistencies, typos, broken links etc.

• Check for any lines in the existing association files that reference obsolete or alternate IDs (see below:)

Updating the "Filter Annotation Objects" lists on the AmiGO browser:

• If annotations from new database groups or species are added, we should update the "Filter Annotation Objects" list on the browser

These lists should match what is in our database; as listed on the release notes page.

LC: new in V#16: Jaiswal Lab: Fragaria vesca (strawberry)

Also, new annotations for Physcomitrella (currently coming from Gramene, but it would be good to be able to filter by species).

In Jan 2012 16A release, we added a large set from Physcomitrella from Cosmoss. Mar 2012 release will include new ones from AgBase for cotton (Gossypium spp.) and new species for SGN.

We also need SGN germplasm which has 4503 annotations

JE: This is handled by the freez_hash_misc_keys script. Make the changes in that file, run the script and copy the created file to the cgi-bin/plantontology directory.

Fixing Links to Dbxrefs:

The was an issue that was discussed and worked on a lot for the version #16 release.

For more details of the discussions see the minutes from POC_Meetings_Minutes on: 9-27-11 through 10-18-11.

Our current version of AmiGO has a built-in set of dbxrefs, in a Perl file, and it is not reading the current GO file. In order to update these, this file has to be edited manually.

We decided on the POC_Conf._Call_9-27-11 to use the PO_DBXref.txt file on the SVN to store the updated dbxrefs stanzas. Then the dbxrefs Perl file could be modified based on that.

• An older version of this file is also available through the documentation page on the PO website:

We decided to change the link on the Documentation page to go to the current file on the SVN. This change should have been implemented in the current release.

Note from LC: Do we really need to have this file PO_DBXref.txt on our website, since it is just for our internal use?

Curators names in definitions, etc

- Curators names in definitions and synonyms should be spelled out in full example: GR:Pankaj_Jaiswal and these are not links (unless they are POC:)

POC:curators

If the definition is worked out as a group, then the source should be POC:curators. This should link to a page where all the curators of the POC are listed.

RW created a new dbxref link. Any reference that begins with POC (e.g., POC:curators, POC:wood_curators, POC:Maria_Alejandra_Gandolfo), will link to the Plant_Ontology_curators page.

Po-Refs Page

- Links to references that do not have a PubMed ID, ISBN number, or stable URL should go to the new PO references page

Links to PO:ids in term definitions

Starting in the March 2012 release we are putting the PO:ids in both the term definitions and in the comments. Currently, the links in the comment are working but not the ones in the definitions.

There is a stanza in the dbxrefs file which refers to these, but it needs to be updated in the AmiGO browsers.

Column 16 annotations (for future releases):

The IT team should run a script to check for needed column 16 annotations; see list at PO Suggestions for Col 16 and more info at: PO Annotation Extensions (column 16)

Icons for new relations

New icon has been added for the has_participant relation so it can be included in the upcoming release. March 2012

• Icons need to be stored in the folder on the SVN:

Need to add link to relations page from icons on live browser.

On Relations in the Plant Ontology, add link to SVN for icons. Link to SVN folder at the top of the page, and embed images, linked to external files, next to each relation.

Should share the icons with OBO Foundry, so they can be used consistently across ontologies.

Summary of Changes Wiki Pages

• During the time between the releases, while we are editing the ontology, we will create a page (or pages) to keep track of the changes being made

For example: see October_2011_Release_Page this has links to the "Summary of Changes" page, e.g.: Summary of Changes to PO October 2011

• On this page there should be a list (or a link to a page) of any new terms and terms that have been obsoleted see: New_terms_and_obsolete_terms_for_October 2011_release

• Also should list terms that have been merged, changed definitions, or renamed.

• If possible, it is also good to list new synonyms for existing terms, especially if those synonyms are quite different from the original name. For example, we would want to highlight the cone is a synonym of strobilus, but it is not that important to note that portion of epidermal tissue is a synonym for epidermis.

Preparing Ontology Files for Release:

Dev version

Between releases, editing is done on the developers version of the plant_ontology.obo file located at: OBO Format.

This is the "dev" version and is loaded onto the Dev Browser nightly, along with the SVN version #. No annotations are loaded on this browser.

At this time, editing may continue on the dev file, without disrupting the version on the beta browser (see below)

• plant_ontology.obo

Once the editing and quality checks are completed for the release, the following additional versions of the Plant Ontology file are prepared prior to the release and moved over to a new beta branch (see below).

• plant_ontology_assert.obo
• plant_ontology_assert_basic.obo
• po_anatomy.obo
• po_temporal.obo

LC: Note: These files should not remain in the trunk folder with the dev version of the ontology once the release is out as they are no longer in sync with the editors version of the file and this will be confusing to have them in two places.

In hindsight, I think the additional files should only be generated once the review process is completed. If an Ontology problem is found, these will have to be recreated (see below:)

Preparing alternative versions of the ontology

plant_ontology_assert.obo

This version is the same as plant_ontology.obo with all nonredundant implied links added by a reasoner.

The "assert implied links" panel in OboEdit does not work very well, but this file can be created very easily using Oort.

To create plant_ontology_assert.obo:

• open the desktop GUI for Oort
• browse to the latest version of plant_ontology.obo as the input file
• specify a folder for the output
• on the "Advanced" tab, uncheck all options.
• select only "OBO" as a Write Format
• choose either the HermiT or the Pellet reasoner (results should be the same)
• click run
• rename the output file "po.obo" as "plant_ontology_assert.obo" and save in the appropriate directory

plant_ontology_assert_basic.obo

This version is the same as plant_ontology_assert.obo, but with only is_a and part_of relations. This file can be created using either a filtered save in OboEdit or a query in Oort.

To create plant_ontology_assert_basic.obo in OboEdit:

• open plant_ontology_assert.obo
• select save as, and use advanced save
• set the save path to .../plant_ontology_assert_basic.obo
• check "Filter links" only
• set up filter to "Matches any", "type, have, any field, contains, is_a", and "type, have, any field, contains, part_of". RW can supply the filter as a text file (assert_basic_filter.txt).
• I used Greedy root selection algorithm an Write current ID rules, but I don't think that matters
• add a comment "filtered from plant_ontology_assert.obo to have only is_a and part_of relations"
• save
• re-open the file in OboEdit. You will see a warning that a number of terms have exactly one intersection_of term. These are the cross products that were set up using participates_in, has_participant, or develops_from. These must be removed manually (in the text file), but there aren't many of them. There is probably a way to filter out all intersection_of relations with OboEdit too (they aren't needed, since all implied links have already been asserted).
• remove any intersection_of tags that are causing problems using a text editor (or OboEdit, if it will work) and save the file.

po_anatomy.obo

This file is created by filtering plant_ontology_assert_basic.obo to contain only terms from the plant anatomical entity branch. Instruction below describe how to do this in OboEdit, but in theory it could be done using Oort as well.

To create po_anatomy.obo in OboEdit:

• open plant_ontology_assert_basic.obo
• manually remove any replaced_by or consider relations that go between the two branches of the ontology (these include: gametophyte, seedling, sporophyte). After removing these links, save the file as some dummy name, so that you can use it to create po_temporal.obo.
• open advanced save dialog
• set the save path to .../po_anatomy.obo
• check "Filter terms" only
• set up filter to "Have, namespace, contains, plant_anatomy"
• check "always save properties"
• add a comment "filtered from plant_ontology_assert_basic.obo to have only plant anatomical entity terms"
• save and check by re-opening OboEdit

po_temporal.obo

This file is created by filtering plant_ontology_assert_basic.obo to contain only terms from the plant structure development stage branch. Instruction below describe how to do this in OboEdit, but in theory it could be done using Oort as well.

To create po_temporal.obo in OboEdit: To create po_anatomy.obo in OboEdit:

• open plant_ontology_assert_basic.obo
• manually remove any replaced_by or consider relations that go between the two branches of the ontology (these include: gametophyte, seedling, sporophyte).
• open advanced save dialog
• set the save path to .../po_temporal.obo
• check "Filter terms" only
• set up filter to "Have, namespace, contains, plant_structure_development_stage"
• check "always save properties"
• add a comment "filtered from plant_ontology_assert_basic.obo to have only plant structure development stage terms"
• save and check by re-opening OboEdit

Beta version

Prior to the release, the ontology files will be copied/moved over to a new beta branch of the svn:

• plant_ontology.obo (copied)

these should be moved to beta:

• plant_ontology_assert.obo
• plant_ontology_assert_basic.obo
• po_anatomy.obo
• po_temporal.obo

LC: Perhaps these should be named something like: plant_ontology_beta16.obo to differentiate them from the editor's version (plant_ontology.obo on dev).

This is a temporary location for the curators and reviewers to look over the new version of the ontology for the planned release. This file is loaded onto the Beta Browser along with any annotation files that are being tested.

• We should also put the revised PO front page on the Beta Site for review, prior to publishing.

At this time, editing may continue on the dev file, without disrupting the version on the beta browser (see above)

If a problem is found that needs to be fixed for the release:

In the event that a problem is found in the ontology file beta version prior to the release, the fix should be made in the DEV version and the process of generating the additional files and copying over to the new beta branch should be repeated anew. This ensures that the fixes are not lost from the editor's version in the process of the release. This is why it is important for the editors to do a thorough review and run QC checks prior to copying to beta.

Tbl version files:

We will also prepare tab-delimited files corresponding to the ontology files:

• plant_ontology.tbl (currently named: po_all.tbl)
• po_anatomy.tbl
• po_temporal.tbl

These will contain the PO:id, term name, the definition, any synonyms and the aspect (in po_all.tbl)

Note: These files also include the obsoleted terms. This is indicated in the definition field of those terms.

These files are be located at TBL files on SVN trunk

The link from the PO download page goes to Live tag.

Note: Only the PO_all.tbl file is available at the Live tag.

LC: It would also be good to have a column listing the alt_ids. I also feel that it is confusing to have these located in two different locations- this is a problem if changes are made in the file on live which are not reflected in the trunk version.

OWL version

We also create OWL versions of the files for each release, which are located in the SVN trunk folder at: OWL_Format.

• The owl file should have a version stamp added to the header, updated with each release:

<owl:versionInfo> xml:lang="en" version ##</owl:versionInfo>

LC: Should this correspond to the Ontology file version (ie: #16) or the version of the OWL software?

It currently says "<?xml version="1.0"?> We should have info here about which Ontology version it is as well.

OWL files available on trunk:

• plant_ontology.owl (This corresponds to plant_ontology.obo from the live tag)
• po_anatomy.tbl
• po_temporal.tbl

Only the plant_ontology.owl file is moved over to the live tag.

Preparing the Association files for the Release

• Check for any lines in the existing association files that reference obsolete or alternate IDs (see wikikpage "Changes to the ontology".

can use grep on the command line to search for the term ids that have been obsoleted.

• Inform the collaborating group about what needs to be fixed and and fix those ourselves if necessary.

• Any problems found in the association files should be fixed in the files in the trunk folder and the copied over to beta, replacing the existing file with the problems.

This accomplishes two things: it maintains the version history on trunk and makes sure the fixes are not lost.

Note: As number of annotations and the size of the association files increases, this process will take longer. The version #16 required 46 hours to load. So it is a good idea to do this check before we start the test load.

External Reviews

Once we have completed the editing, the Ontology should be open for review on the Beta browser. This may occur with or without the full set of annotations. Depending on what we want to test. (see below for more details on Beta)

Inviting Specific External Reviewers

For some releases, we may ask specific experts and or collaborators to take part in an External Review. This was done in Aug 2010 for the October 2010 release. The external experts will need at least a couple of weeks and up to one month.

To facilitate the review, the POC may need to make a presentation to them, such as the Plant_Ontology_Webinar-_May_2011_release PO-Physcomitrella presentation, to feature the new terms for non-vascular plants, in particular, Physcomitrella patens.

Add more details here of the external review process...

Beta Release Announcement

An announcement should be sent out on the mailing lists something like this:

"A beta version of the latest release of the PO is now available on our AmiGO Browser (http://beta.plantontology.org/amigo/go.cgi). Along with the Ontology itself the beta browser also is loaded with some test annotation files (not the full set of files at this point).

You can download the ontology files in OBO-Edit format from: http://palea.cgrb.oregonstate.edu/viewsvn/Poc/branches/beta/ontology/OBO_format/

There is a readme.txt file on the same page explaining what the is different among the files available.

Further information on the upcoming release can be obtained from: Links to Release Pages: "Summary of Changes" and "New and Obsoleted Terms"

Please review the Plant Ontology and let us know asap about any important issues you observe."

Loading the Live Version

Once the review and internal checks and fixes are completed, the plant_ontology_assert.obo file and the whole database of annotations are loaded from the beta branch onto the beta browser.

Note that it is the "plant_ontology_assert.obo " file that should be loaded onto AmiGO browser on the live site.

Once the release is live on the Plant Ontology page, the ontology files should be copied from the beta branch to Live Tag.

This is a stable url that will not change, so it can be incorporated into users scripts etc.

In the October 2011 release, we introduced Japanese translations of the term names. Currently, JE is stripping out the Japanese synonyms before he loads the live version onto AmiGO. We had discussed whether or not to remove them from the live release and provide a separate file which included the translated terms, but that was rejected (meeting date: fill in date...). We have had complaints from some of our user groups (eg. Gramene) that their browsers cannot display the Japanese characters . We should reconsider this approach.

Changes to the HTML Files for the PO web site:

Prepare "Release Notes" page

• Each Release and any Interim Data Release should have a unique Release Notes webpage on the PO site detailing the current statistics of the release.
• The old pages are archived Release Notes Archive so that we can keep the track of the changes.

Update PO front page

• Post release announcements
• Update any "Upcoming Events", move older event to the archive page

• Update the "Filter Annotation Objects Counts" for new species, data types in release

This good time to incorporate any changes or fixes to the web pages:

List needed fixes here:

• Documentation page: replace link to old dbxrefs file with a link to the current one on the svn.

Announcements

• Prepare and post announcements for the PO front page, Jaiswal Lab Page, FB page
• Send out announcements to the mailing lists

other places??

Updating the Plant Ontology files on other sites:

The two main sites where people go to download the PO are:

• Bioportal: Bioportal

• OBO Foundry: OBO Foundry

Also featured at:

• Gramene: Gramene

• MaizeGDB: MaizeGDB

• SGN: Solanaceae Genome network (SGN)

• TAIR:

See this page for links to other sites that feature the PO: Links_to_sites_using_the_PO