Tree Annotation - Revision history

Hilmar: /* additional suggestions for MIAPA ontology */

2013-02-03T00:29:52Z

additional suggestions for MIAPA ontology

← Older revision		Revision as of 00:29, 3 February 2013
Line 69:		Line 69:

	== Model for semantic encoding ==		== Model for semantic encoding ==

	~~==additional suggestions for MIAPA ontology==~~
	~~From annotation session on afternoon of 1/31.~~

	* Add a class for parsimony under algorithm. [http://github.com/miapa/miapa/issues/7 Filed as Issue #7]

	* It would be good to generate an instance of useMaximumLikelihood ("Maximum Likelihood algorithm") in MIAPA, so we don't have to create one for each annotation. [http://github.com/miapa/miapa/issues/8 Filed as Issue #8]

	* Alternatively, maybe make classes of software (like PhyML or RAxML) implement ML algorithm, rather than having to assert it for each instance we create. Some software can use multiple algorithms, so we can't do this for each case.
	** ''Note that in OWL classes cannot be asserted to have property values, only instances can. We can put property restrictions with existential quantification on classes, and a OWL reasoner could then infer that an instance must have at least one such property association (and thus a DL query should in principle return the instance), but this wouldn't work in an RDF triple store so that we could then actually query for these things in SPARQL.''
	** ''Note also that there can be multiple swo:implements assertions for a software instance, so multiple algorithms can be easily asserted. However, this wouldn't the also say which of those implemented algorithms was the one utilized for the generation of the tree of alignment. The idea is that this would be evident from the miapa:'Parameter specification'.''

	* Remove class for SILVA. [http://github.com/miapa/miapa/issues/9 Filed as Issue #9]

	* Add new class for set of trees. [http://github.com/miapa/miapa/issues/10 Filed as Issue #10]

	== more annotations ==		== more annotations ==

Hilmar: /* Quick links */

2013-02-03T00:17:41Z

Quick links

← Older revision		Revision as of 00:17, 3 February 2013
Line 3:		Line 3:
	== Quick links ==		== Quick links ==

	* [[Media:FinalPres.pptx\| Enrico's final presentation]]		* Overview: [[Media:FinalPres.pptx\| Enrico's final presentation]]
	** Also see [[:Image:PetersGraphic.jpg\|example annotation graph for the Peters et al. tree and alignment]].		** Also see [[:Image:PetersGraphic.jpg\|example annotation graph for the Peters et al. tree and alignment]].
	* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees		* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees

Hilmar: /* Quick links */

2013-02-03T00:16:57Z

Quick links

← Older revision		Revision as of 00:16, 3 February 2013
Line 3:		Line 3:
	== Quick links ==		== Quick links ==

	* [[Media:FinalPres.pptx\| Enrico's final presentation]]		* [[Media:FinalPres.pptx\| Enrico's final presentation]]
			** Also see [[:Image:PetersGraphic.jpg\|example annotation graph for the Peters et al. tree and alignment]].
	* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees		* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees
	* [[TreestoreMetadataQueryDemonstration]] - report on the workflow and querying		* [[TreestoreMetadataQueryDemonstration]] - report on the workflow and querying

Hilmar: /* Quick links */

2013-02-03T00:14:45Z

Quick links

New page

'''Synopsis''' Annotate a small set of large trees used as sources of phylogenetic knowledge in an automated delivery system for tree-o-life knowledge called "Phylotastic".

== Quick links ==

* [[Media:FinalPres.pptx| Enrico's final presentation]]
* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees
* [[TreestoreMetadataQueryDemonstration]] - report on the workflow and querying
* [[AdvancingMIAPA]] - report on MIAPA-relevant findings and developments
* [http://wiki.tdwg.org/twiki/bin/view/Phylogenetics/MIAPADraft#Reconciled_draft_checklist checklist Reconciled_draft_checklist checklist from TDWG 2011]

== Synopsis ==

We (1) identified a set of 10 large trees useful as phylotastic source trees (2) created free-text annotations (metadata) for citations, sources and methods, (3) encoded the data and metadata as RDF using CDAO and a new ontology, (4) loaded the encoded information into a triplestore, and (5) demonstrated logical querying based on data and metadata attributes. During the hackathon, group members spent their time developing and revising a strategy, interpreting source materials, developing language support, encoding annotations, working out technical bugs in the workflow, and addressing emerging challenges. The tangible outcomes of this exercise include
* a set of 10 source trees (720 to 250,000 species) with low-res metadata (see [[AnnotatedPhylotasticSourceTrees]])
* a demonstration of semantic annotation, data-basing, and querying (see [[TreestoreMetadataQueryDemonstration]])
** a workflow plan for encoding tree data and metadata and loading them into a treestore
** a treestore instance populated with some of these data and metadata
* Advances in Minimum Information About a Phylogenetic Analysis (see [[AdvancingMIAPA]])
** a new "MIAPA" ontology that leverages several existing ontologies
** recommendations on the draft checklist, and input form
* optionally
** a screencast
** a draft GSOC proposal

== Background, Motivation, and Aims ==

Metadata annotations represent an essential part of the design of phylotastic systems, for two reasons. First, while we do not have a robust and detailed understand of how users will make use of phylotastic systems, we assume that they will wish to identify trees based on sources and methods. For instance, a user may restrict a phylotastic query so as to include only trees inferred by Maximum Likelihood, or to exclude grafted trees, or to implicate only the tree associated with the publication by Bininda-Emonds, 2007.

Second, one of the design criteria of phylotastic systems is to provide credible results, which in the scientific world means providing a description of provenance suitable for a scientific publication. To be credible, a tree generated by a phylotastic system must include a description of how it was derived, which includes information on source trees as well as a description of any subsequent manipulations. Yet, metadata play little or no role in current phylotastic component implementations.

Some guidance may be obtained from prior art relating to databases and to metadata. Two databases for trees exist already, TreeBASE and Dryad Dryad does not provide any explicit support for tree-specific annotations. The TreeBASE input interface allows citation data, creates links to species ident, and links a matrix to a tree with a "analysis" link that may implicate a particular software program. Though useful, the TreeBASE model falls far short of the recommendations for a "minimum information" standard for phylogeny metadata known as MIAPA, or "Minimum Information About a Phylogenetic Analysis". For instance, the draft MIAPA checklist from TDWG2011 calls for an explicit indication of whether a tree is a gene or species tree, whether it is rooted, what software (and version number) was used to derive it, and so on.

The TreeAnnotation team of hackathon 2 (Enrico, Hilmar, Joachim, Arlin, Ramona and 0.5 of Andrea) decided to conduct an annotation exercise that would cover the flow of information from initial annotation of trees, to querying of treestores (not including the annotation of subsequent phylotastic manipulations such as pruning or scaling).

The motivation for this exercise relates partly to the aspirational nature of MIAPA, which was proposed many years ago but has never evolved into a clear standard supported with convenient technology. Those of us who have been involved in MIAPA-related efforts sensed a need for practical experiences in real-world uses of annotations. While the scope of Phylotastic is narrower than that of MIAPA, in the sense of covering only species trees, and mainly large ones, the challenge of supporting useful metadata queries in a treestore represents a critical test of the relevance of the MIAPA checklist and the technology for encoding and managing semantic annotations.

We also hoped to enrich current phylotastic implementations by providing metadata for a specific set of useful trees. Hackathon participants have been using a handful of trees (APGIII, Bininda-Emonds, etc) without any metadata on citations or methods.

Thus, our approach has 3 inter-connected aims:
* to create a set of 10 usefully annotated source trees
* to demonstrate the feasibility of metadata-based querying in a treestore
* to leverage a practical annotation exercise to advance the MIAPA project

== Approach ==

Our approach consisted of the following steps
# identify 10 useful source trees with available publications
# generate free-text annotations
# encode citations and annotations in computable form
# load the citation, annotations, and trees into a treestore
# demonstrate querying based on metadata

In particular, we chose to gather metadata corresponding to the MIAPA draft checklist, to enode it as RDF using a new ontology that imports several other ontologies, and to load the results into Ben Morris's Virtuoso-based treestore implementation. On Day 3, we decided to begin by focusing on citations, which are not in the MIAPA checklist, with the plan to carry citation data through steps 3 to 5.

'''workflow, in more detail'''
# Identify 10 trees for use as phylotastic source trees. Most of the trees were identified pre-hackathon by Arlin. One of the trees was replaced on day 2 due to lack of metadata (the unpublished fish tree of Westneat & Lundberg).
# Annotate them in free-text form. This was done on day #2 as a team effort by Ramona, Enrico, Arlin and Andrea.
#* create [https://docs.google.com/spreadsheet/gform?key=0An7iC3iCSumjdDRob1NjUUdxVHFRSG9ZQWltZ0ExZ3c&hl=en#edit web form] in Google docs for input of annotations, based on [http://wiki.tdwg.org/twiki/bin/view/Phylogenetics/MIAPADraft#Reconciled_draft_checklist MIAPA draft checklist] from TDWG 2011 workshop
#* Spread sheet has pull down menus, plus options for free text entries under "other"
# Transform annotations into a formal language statements in RDF. This was done on Days 3 and 4 by Ramona, Arlin, Enrico and Hilmar.
#* Literature Citations
#** after some discussion, we decided to use BIBO (not dc or prism alone)
#** we spent 6 to 8 person hours trying to do this interactively in Protege before finding an automated pathway of discovery and conversion via PubMed--> EndNote --(bibtex export)--> Zotero --> bibo export (bibliontology RDF).
#** here is the [[file:10trees_bibliontology.rdf]]
#* Hilmar developed an annotation ontology that incorporates CDAO, OBI, PROV and other ontologies
# Load trees into TreeStore. On days 2 to 3, Joachim worked on the technology for getting our encodings into a triplestore. Part of the challenge was deciding on an URL scheme.
# Execute queries to demonstrate success. On Day 3 we had success in querying for citation metadata. On Day 5 ? ? ?

== Model for semantic encoding ==

==additional suggestions for MIAPA ontology==
From annotation session on afternoon of 1/31.

* Add a class for parsimony under algorithm. [http://github.com/miapa/miapa/issues/7 Filed as Issue #7]

* It would be good to generate an instance of useMaximumLikelihood ("Maximum Likelihood algorithm") in MIAPA, so we don't have to create one for each annotation. [http://github.com/miapa/miapa/issues/8 Filed as Issue #8]

* Alternatively, maybe make classes of software (like PhyML or RAxML) implement ML algorithm, rather than having to assert it for each instance we create. Some software can use multiple algorithms, so we can't do this for each case.
** ''Note that in OWL classes cannot be asserted to have property values, only instances can. We can put property restrictions with existential quantification on classes, and a OWL reasoner could then infer that an instance must have at least one such property association (and thus a DL query should in principle return the instance), but this wouldn't work in an RDF triple store so that we could then actually query for these things in SPARQL.''
** ''Note also that there can be multiple swo:implements assertions for a software instance, so multiple algorithms can be easily asserted. However, this wouldn't the also say which of those implemented algorithms was the one utilized for the generation of the tree of alignment. The idea is that this would be evident from the miapa:'Parameter specification'.''

* Remove class for SILVA. [http://github.com/miapa/miapa/issues/9 Filed as Issue #9]

* Add new class for set of trees. [http://github.com/miapa/miapa/issues/10 Filed as Issue #10]

== more annotations ==

miapa ontology
* topology
** gene tree vs species tree: Network:Tree:'Gene tree' or SpeciesTree
** rooted: Network:Tree:RootedTree or UnrootedTree
** 'Consensus tree'
* otus
** toTaxon, object property, points to taxon concept, can be URI from NCBI or other authority
** derived_from specimen
** location imported from geo
* branch properties
** branch lengths:
*** data property edge length
*** object property has_Annotation edge_length
** branch support: data property has support value either bootstrap or posterior prob
* character matrix
* alignment method
** name of software, version
** parameters
** manual correction
* tree inference method
** name of software, version: tree wasGeneratedBy (activity=) software procedure; software procedure wasAssociatedWith instance of software agent named "RaXML"
** parameters: (activity) used instance of a parameter specification (which is a kind of plan)
** character weights

Hilmar: /* additional suggestions for MIAPA ontology */

2013-02-01T16:51:51Z

additional suggestions for MIAPA ontology

← Older revision		Revision as of 16:51, 1 February 2013
Line 92:		Line 92:
	From annotation session on afternoon of 1/31.		From annotation session on afternoon of 1/31.

	Add a class for parsimony under algorithm.		* Add a class for parsimony under algorithm. [http://github.com/miapa/miapa/issues/7 Filed as Issue #7]

	It would be good to generate an instance of useMaximumLikelihood ("Maximum Likelihood algorithm") in MIAPA, so we don't have to create one for each annotation.		* It would be good to generate an instance of useMaximumLikelihood ("Maximum Likelihood algorithm") in MIAPA, so we don't have to create one for each annotation. [http://github.com/miapa/miapa/issues/8 Filed as Issue #8]

	Alternatively, maybe make classes of software (like PhyML or RAxML) implement ML algorithm, rather than having to assert it for each instance we create. Some software can use multiple algorithms, so we can't do this for each case.		* Alternatively, maybe make classes of software (like PhyML or RAxML) implement ML algorithm, rather than having to assert it for each instance we create. Some software can use multiple algorithms, so we can't do this for each case.
	* Note that in OWL classes cannot be asserted to have property values, only instances can. We can put property restrictions with existential quantification on classes, and a OWL reasoner could then infer that an instance must have at least one such property association (and thus a DL query should in principle return the instance), but this wouldn't work in an RDF triple store so that we could then actually query for these things in SPARQL.		** ''Note that in OWL classes cannot be asserted to have property values, only instances can. We can put property restrictions with existential quantification on classes, and a OWL reasoner could then infer that an instance must have at least one such property association (and thus a DL query should in principle return the instance), but this wouldn't work in an RDF triple store so that we could then actually query for these things in SPARQL.''
	* Note also that there can be multiple swo:implements assertions for a software instance, so multiple algorithms can be easily asserted. However, this wouldn't the also say which of those implemented algorithms was the one utilized for the generation of the tree of alignment. The idea is that this would be evident from the miapa:'Parameter specification'.		** ''Note also that there can be multiple swo:implements assertions for a software instance, so multiple algorithms can be easily asserted. However, this wouldn't the also say which of those implemented algorithms was the one utilized for the generation of the tree of alignment. The idea is that this would be evident from the miapa:'Parameter specification'.''

	Remove class for SILVA.		* Remove class for SILVA. [http://github.com/miapa/miapa/issues/9 Filed as Issue #9]

	Add new class for set of trees.		* Add new class for set of trees. [http://github.com/miapa/miapa/issues/10 Filed as Issue #10]

	=== more annotations ===		=== more annotations ===

Hilmar: /* additional suggestions for MIAPA ontology */

2013-02-01T16:43:03Z

additional suggestions for MIAPA ontology

New page

'''Synopsis''' Annotate a small set of large trees used as sources of phylogenetic knowledge in an automated delivery system for tree-o-life knowledge called "Phylotastic".

== Quick links ==

'''Reports'''
* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees, focusing on the types of metadata available, and how they might be used in phylotastic systems
** [https://docs.google.com/spreadsheet/ccc?key=0An7iC3iCSumjdDRob1NjUUdxVHFRSG9ZQWltZ0ExZ3c#gid=0 spreadsheet]
* [[TreestoreMetadataQueryDemonstration]] - report on the model of semantic encoding, the technology for translation, the treestore technology, and the implications of this for supporting phylotastic querying
* [[AdvancingMIAPA]] report page - report on the adequacy of the MIAPA checklist, recommendations for revisions, ontology development, challenges of semantic encoding, and also (redundant to above report) the model of semantic encoding.

'''Other tangible outcomes'''
* new MIAPA ontology
* GSOC project proposal

'''Key resources'''
* [http://wiki.tdwg.org/twiki/bin/view/Phylogenetics/MIAPADraft#Reconciled_draft_checklist checklist] from TDWG 2011
*

==Overview==

Metadata annotations represent an essential part of the design of phylotastic systems, enabling users to find trees based on sources and methods, and to generate a credible report of provenance for phylotastically generated trees. Yet, metadata play no role in current phylotastic component implementations. The TreeAnnotation team of hackathon 2 (Enrico, Hilmar, Joachim, Arlin, Ramona and 0.5 of Andrea) set out to address this deficiency. We developed an approach with 3 inter-connected goals:

* create a set of 10 usefully annotated source trees
* demonstrate metadata-based querying in a treestore
* leverage this exercise to advance the MIAPA project

Our approach consisted of the following steps
# identify 10 useful source trees with available publications
# generate free-text annotations
# encode citations and annotations in computable form
# load the citation, annotations, and trees into a treestore
# demonstrate querying based on metadata

In particular, we chose to gather metadata corresponding to the MIAPA draft checklist, to enode it as RDF using a new ontology that imports several other ontologies, and to load the results into Ben Morris's Virtuoso-based treestore implementation.

During the hackathon, group members spent their time developing and revising a strategy, interpreting source materials, developing language support, encoding annotations, implementing tools, and addressing emerging challenges.

The tangible outcomes of the group relate to phylotastic source trees (a set of trees with metadata); software tools for processing, storage and querying; an ontology to support MIAPA annotations, along with a revised MIAPA checklist and form; and written reports on these 3 types of outputs, available on this wiki.

== Detailed approach ==

* develop plan (day 1)
** revise as needed
** some work is done in parallel

* main workflow
# identify 10 trees for use as phylotastic source trees
# annotate them in free-text form
#* create [https://docs.google.com/spreadsheet/gform?key=0An7iC3iCSumjdDRob1NjUUdxVHFRSG9ZQWltZ0ExZ3c&hl=en#edit web form] in Google docs for input of annotations, based on [http://wiki.tdwg.org/twiki/bin/view/Phylogenetics/MIAPADraft#Reconciled_draft_checklist MIAPA draft checklist] from TDWG 2011 workshop
#* Spread sheet has pull down menus, plus options for free text entries under "other"
# transform annotations into a formal language statements in RDF
#* encoding process is iterative with ontology editing
#* Hilmar is working on language support
#* Joachim is working on the technology for getting this into a triplestore
#* Get URI for tree from TreeStore, add annotations to that URI in Protege
# Load trees into TreeStore
#* Will need to have trees in the correct format
# execute queries to demonstrate success

== Log and accomplishments ==
* initial plan (day 1)
* initial MIAPA checklist-based input form (day 1)
* revised input form
* plan for (temporarily) storing trees and matrices (data) separate from metadata
* annotations of 10 trees
* translation technology
** NEXUS issues, dendropy,
** protege deals poorly with unnamed individuals
* ontology for annotation

From day 4, [[Media:followup_goals.jpg]] from white board.

=== citation exercise ===

'''goal:''' annotate trees with citation data, encode, import into treestore, demonstrate querying based on citation metadata

'''notes on encoding'''
* after some discussion, we decided to use BIBO (not dc or prism alone)
* we failed to find any pre-existing method to auto-convert EndNote (or BibTex or Zotero) into BIBO
* so we started hand-encoding them using Protege instances
** authors
** articles
*** used Data property "short title" instead of object property title
*** used date of issue for publication year
** author-lists (RDF:list?)
* ultimately we ended up getting the encoded citations via PubMed--> EndNote --> bibtex export --> Zotero --> bibo export (bibliontology RDF).
** here is the [[file:10trees_bibliontology.rdf]]

===additional suggestions for MIAPA ontology===
From annotation session on afternoon of 1/31.

Add a class for parsimony under algorithm.

It would be good to generate an instance of useMaximumLikelihood ("Maximum Likelihood algorithm") in MIAPA, so we don't have to create one for each annotation.

Alternatively, maybe make classes of software (like PhyML or RAxML) implement ML algorithm, rather than having to assert it for each instance we create. Some software can use multiple algorithms, so we can't do this for each case.
* Note that in OWL classes cannot be asserted to have property values, only instances can. We can put property restrictions with existential quantification on classes, and a OWL reasoner could then infer that an instance must have at least one such property association (and thus a DL query should in principle return the instance), but this wouldn't work in an RDF triple store so that we could then actually query for these things in SPARQL.
* Note also that there can be multiple swo:implements assertions for a software instance, so multiple algorithms can be easily asserted. However, this wouldn't the also say which of those implemented algorithms was the one utilized for the generation of the tree of alignment. The idea is that this would be evident from the miapa:'Parameter specification'.

Remove class for SILVA.

Add new class for set of trees.

=== more annotations ===

miapa ontology
* topology
** gene tree vs species tree: Network:Tree:'Gene tree' or SpeciesTree
** rooted: Network:Tree:RootedTree or UnrootedTree
** 'Consensus tree'
* otus
** toTaxon, object property, points to taxon concept, can be URI from NCBI or other authority
** derived_from specimen
** location imported from geo
* branch properties
** branch lengths:
*** data property edge length
*** object property has_Annotation edge_length
** branch support: data property has support value either bootstrap or posterior prob
* character matrix
* alignment method
** name of software, version
** parameters
** manual correction
* tree inference method
** name of software, version: tree wasGeneratedBy (activity=) software procedure; software procedure wasAssociatedWith instance of software agent named "RaXML"
** parameters: (activity) used instance of a parameter specification (which is a kind of plan)
** character weights

=== semantic links for tree, citation, methods, etc ===

* tree has unique URI produced during loading: http://phylotastic.org/hack2/...number.../...treename...#tree1

* how rooted tree connects together
:tree1 has_root node0 ;

* how unrooted tree connects together, using the belongs_to_tree relation
:node9> obo:CDAO_0000200 :tree1 ;
* and the same for all the other nodes and edges.

* how tree connects with citation (assume that pub1 is the root of the <bibo:AcademicArticle> individual )
:tree1 dcterms:isReferencedBy :pub1 ;

* some other ideas
** :pub1 IAO:is_about :tree1
** :pub1 documents :tree1
** cito:provides_methods_for :tree1
** :pub1 cito:provides_data_for :tree1

* how tree connects with methods annotation
:tree1 prov:wasGeneratedBy :tree_activity1 ;

* how char matrix connects with methods annotation
:align1 prov:wasGeneratedBy :align_activity1 ;

* how tree connects with char matrix
:tree1 prov:wasDerivedFrom :align1 ;

==Annotation Workflow==

'''Example file:''' Tree_2_Peters_et_al.newick

1. `python treestore.py add Tree_2_Peters_et_al.newick newick Peters2011hymenoptera`
* reads Newick file `Tree_2_Peters_et_al.newick`
* stores the tree in the named graph `http://prefix/Peters2011hymenoptera`
* the URI prefix is automatically generated; it is a hash that (more or less) uniquely identifies the data loaded
2. `python treestore.py uri`
* lists tree URIs in the triple store
* will show something along the line: "Peters2011hymenoptera http://phylotastic.org/hack2/bd414f8f72a8fabb9454b4ea72cf0e8a760171ba/Peters2011hymenoptera#tree0000001"
3. `rdfcat -out N-TRIPLE annotations.rdf > annotations.ntriples`
* takes annotations (saved with Protege as RDF/XML, Turtle, or other format)
* outputs N-Triples
4. `python treestore.py add annotations.ntriples ntriples http://phylotastic.org/hack2/bd414f8f72a8fabb9454b4ea72cf0e8a760171ba/Peters2011hymenoptera`
* adds the annotations to the named graph `http://phylotastic.org/hack2/bd414f8f72a8fabb9454b4ea72cf0e8a760171ba/Peters2011hymenoptera`
* the URI for the named graph is the URI returned by `python treestore.py uri` up to the `#` character

Hilmar: /* semantic links for tree, citation, methods, etc */

2013-01-31T23:02:36Z

semantic links for tree, citation, methods, etc

← Older revision		Revision as of 23:02, 31 January 2013
Line 113:		Line 113:

	* how rooted tree connects together		* how rooted tree connects together
	tree1 has_root node0		:tree1 has_root node0 ;

	* how unrooted tree connects together, using the belongs_to_tree relation		* how unrooted tree connects together, using the belongs_to_tree relation
Line 129:		Line 129:

	* how tree connects with methods annotation		* how tree connects with methods annotation
	:tree1 prov:wasGeneratedBy :tree_activity1		:tree1 prov:wasGeneratedBy :tree_activity1 ;

	* how char matrix connects with methods annotation		* how char matrix connects with methods annotation
	:align1 prov:wasGeneratedBy :align_activity1		:align1 prov:wasGeneratedBy :align_activity1 ;

	* how tree connects with char matrix		* how tree connects with char matrix
	:tree1 prov:wasDerivedFrom :align1		:tree1 prov:wasDerivedFrom :align1 ;

	==Annotation Workflow==		==Annotation Workflow==

Hilmar: /* semantic links for tree, citation, methods, etc */

2013-01-31T23:02:06Z

semantic links for tree, citation, methods, etc

New page

'''Synopsis''' Annotate a small set of large trees used as sources of phylogenetic knowledge in an automated delivery system for tree-o-life knowledge called "Phylotastic".

== Quick links ==

* [[AnnotatedPhylotasticSourceTrees]] - report on the set of source trees, focusing on the types of metadata available, and how they might be used in phylotastic systems
** [https://docs.google.com/spreadsheet/ccc?key=0An7iC3iCSumjdDRob1NjUUdxVHFRSG9ZQWltZ0ExZ3c#gid=0 spreadsheet]
* [[TreestoreMetadataQueryDemonstration]] - report on the model of semantic encoding, the technology for translation, the treestore technology, and the implications of this for supporting phylotastic querying
* [[AdvancingMIAPA]] report page - report on the adequacy of the MIAPA checklist, recommendations for revisions, ontology development, challenges of semantic encoding, and also (redundant to above report) the model of semantic encoding.
** [http://wiki.tdwg.org/twiki/bin/view/Phylogenetics/MIAPADraft#Reconciled_draft_checklist checklist] from TDWG 2011
** new MIAPA ontology
**

==Overview==

Metadata annotations represent an essential part of the design of phylotastic systems, enabling users to find trees based on sources and methods, and to generate a credible report of provenance for phylotastically generated trees. Yet, metadata play no role in current phylotastic component implementations. The TreeAnnotation team of hackathon 2 (Enrico, Hilmar, Joachim, Arlin, Ramona and 0.5 of Andrea) set out to address this deficiency. We developed an approach with 3 inter-connected goals:

* create a set of 10 usefully annotated source trees
* demonstrate metadata-based querying in a treestore
* leverage this exercise to advance the MIAPA project

Our approach consisted of the following steps
# identify 10 useful source trees with available publications
# generate free-text annotations
# encode citations and annotations in computable form
# load the citation, annotations, and trees into a treestore
# demonstrate querying based on metadata

In particular, we chose to gather metadata corresponding to the MIAPA draft checklist, to enode it as RDF using a new ontology that imports several other ontologies, and to load the results into Ben Morris's Virtuoso-based treestore implementation.

During the hackathon, group members spent their time developing and revising a strategy, interpreting source materials, developing language support, encoding annotations, implementing tools, and addressing emerging challenges.

The tangible outcomes of the group relate to phylotastic source trees (a set of trees with metadata); software tools for processing, storage and querying; an ontology to support MIAPA annotations, along with a revised MIAPA checklist and form; and written reports on these 3 types of outputs, available on this wiki.

== Detailed approach ==

* develop plan (day 1)
** revise as needed
** some work is done in parallel

* main workflow
# identify 10 trees for use as phylotastic source trees
# annotate them in free-text form
#* create [https://docs.google.com/spreadsheet/gform?key=0An7iC3iCSumjdDRob1NjUUdxVHFRSG9ZQWltZ0ExZ3c&hl=en#edit web form] in Google docs for input of annotations, based on [http://wiki.tdwg.org/twiki/bin/view/Phylogenetics/MIAPADraft#Reconciled_draft_checklist MIAPA draft checklist] from TDWG 2011 workshop
#* Spread sheet has pull down menus, plus options for free text entries under "other"
# transform annotations into a formal language statements in RDF
#* encoding process is iterative with ontology editing
#* Hilmar is working on language support
#* Joachim is working on the technology for getting this into a triplestore
#* Get URI for tree from TreeStore, add annotations to that URI in Protege
# Load trees into TreeStore
#* Will need to have trees in the correct format
# execute queries to demonstrate success

== Log and accomplishments ==
* initial plan (day 1)
* initial MIAPA checklist-based input form (day 1)
* revised input form
* plan for (temporarily) storing trees and matrices (data) separate from metadata
* annotations of 10 trees
* translation technology
** NEXUS issues, dendropy,
** protege deals poorly with unnamed individuals
* ontology for annotation
**

=== citation exercise ===

'''goal:''' annotate trees with citation data, encode, import into treestore, demonstrate querying based on citation metadata

'''notes on encoding'''
* after some discussion, we decided to use BIBO (not dc or prism alone)
* we failed to find any pre-existing method to auto-convert EndNote (or BibTex or Zotero) into BIBO
* so we started hand-encoding them using Protege instances
** authors
** articles
*** used Data property "short title" instead of object property title
*** used date of issue for publication year
** author-lists (RDF:list?)
* ultimately we ended up getting the encoded citations via PubMed--> EndNote --> bibtex export --> Zotero --> bibo export (bibliontology RDF).
** here is the [[file:10trees_bibliontology.rdf]]

=== more annotations ===

miapa ontology
* topology
** gene tree vs species tree: Network:Tree:'Gene tree' or SpeciesTree
** rooted: Network:Tree:RootedTree or UnrootedTree
** 'Consensus tree'
* otus
** toTaxon, object property, points to taxon concept, can be URI from NCBI or other authority
** derived_from specimen
** location imported from geo
* branch properties
** branch lengths:
*** data property edge length
*** object property has_Annotation edge_length
** branch support: data property has support value either bootstrap or posterior prob
* character matrix
* alignment method
** name of software, version
** parameters
** manual correction
* tree inference method
** name of software, version: tree wasGeneratedBy (activity=) software procedure; software procedure wasAssociatedWith instance of software agent named "RaXML"
** parameters: (activity) used instance of a parameter specification (which is a kind of plan)
** character weights

=== semantic links for tree, citation, methods, etc ===

* tree has unique URI produced during loading: http://phylotastic.org/hack2/...number.../...treename...#tree1

* how rooted tree connects together
tree1 has_root node0

* how unrooted tree connects together, using the belongs_to_tree relation
:node9> obo:CDAO_0000200 :tree1 ;
* and the same for all the other nodes and edges.

* how tree connects with citation (assume that pub1 is the root of the <bibo:AcademicArticle> individual )
:tree1 dcterms:isReferencedBy :pub1 ;

* some other ideas
** :pub1 IAO:is_about :tree1
** :pub1 documents :tree1
** cito:provides_methods_for :tree1
** :pub1 cito:provides_data_for :tree1

* how tree connects with methods annotation
:tree1 prov:wasGeneratedBy :tree_activity1

* how char matrix connects with methods annotation
:align1 prov:wasGeneratedBy :align_activity1

* how tree connects with char matrix
:tree1 prov:wasDerivedFrom :align1

==Annotation Workflow==

'''Example file:''' Phylomatictree.nex

1. `python treestore.py add Phylomatictree.nex nexus phylomatictree`
* reads NEXUS file `Phylomatictree.nex`
* stores the tree in the named graph `phylomatictree`
2. `rdfcat -out N-TRIPLE annotations.rdf > annotations.ntriples`
* takes annotations (saved with Protege as RDF/XML, Turtle, or other format)
* outputs N-Triples
3. `python treestore.py add annotations.ntriples ntriples phylomatictree`
* adds the annotations to the named graph `phylomatictree`

'''Example file:''' Tree_2_Peters_et_al.newick

1. `python treestore.py add Tree_2_Peters_et_al.newick newick peters2`
* reads Newick file `Tree_2_Peters_et_al.newick`
* stores the tree in the named graph `peters2`
2. `rdfcat -out N-TRIPLE annotations.rdf > annotations.ntriples`
* takes annotations (saved with Protege as RDF/XML, Turtle, or other format)
* outputs N-Triples
3. `python treestore.py add annotations.ntriples ntriples peters2`
* adds the annotations to the named graph `peters2`

Hilmar: /* Annotated */

2013-01-30T04:12:45Z

Annotated

← Older revision		Revision as of 04:12, 30 January 2013
Line 138:		Line 138:
	** available as an SQL dump from ftp://ftp.ncbi.nih.gov/pub/taxonomy/ (see the README file)		** available as an SQL dump from ftp://ftp.ncbi.nih.gov/pub/taxonomy/ (see the README file)
	** the converted tree in Newick format is available compressed from http://itol.embl.de/other_trees.shtml, as follows		** the converted tree in Newick format is available compressed from http://itol.embl.de/other_trees.shtml, as follows
	* [http://itol.embl.de/ncbi_tree/ncbi_complete_collapsed_with_names.newick.gz complete tree, using scientific names, internal nodes with only one child are removed]. * also [[File:Ncbi complete collapsed with names.tre\| uploaded here as the uncompressed newick tree]]		*** [http://itol.embl.de/ncbi_tree/ncbi_complete_collapsed_with_names.newick.gz complete tree, using scientific names, internal nodes with only one child are removed].
	*** '''Note:''' This tree has ''every'' ~~taxa~~ in NCBI, including ones like "Insertion_sequence_IS2" and "Plasmid_pHV2". For Phylotastic, we would probably want to generate a tree that is some subset of this.		*** also [[File:Ncbi complete collapsed with names.tre]] uploaded here as the uncompressed newick tree
			*** '''Note:''' This tree has ''every'' taxon in NCBI, including ones like "Insertion_sequence_IS2" and "Plasmid_pHV2" (though in this version not those that have only a single child). For Phylotastic, we would probably want to generate a tree that is some subset of this.
	** manually curated		** manually curated
	** FYI, NCBI provides an interactive way to get a tree phylotastically (http://www.ncbi.nlm.nih.gov/guide/howto/gen-com-tree/)		** FYI, NCBI provides an interactive way to get a tree phylotastically (http://www.ncbi.nlm.nih.gov/guide/howto/gen-com-tree/)

Hilmar: /* Annotated */

2013-01-30T04:08:52Z

Annotated

← Older revision		Revision as of 04:08, 30 January 2013
Line 93:		Line 93:
	** needs conversion to Newick or NEXUS format		** needs conversion to Newick or NEXUS format
	** (note: this tree structure in TOL.xml.zip is old, it is from October 22, 2006)		** (note: this tree structure in TOL.xml.zip is old, it is from October 22, 2006)
			** URI: http://tolweb.org/tree/

	* '''angiosperm''' phylogeny from Smith et al. 2011, [http://www.amjbot.org/content/98/3/404.short]		* '''angiosperm''' phylogeny from Smith et al. 2011, [http://www.amjbot.org/content/98/3/404.short]
Line 136:		Line 137:
	** 250000 species		** 250000 species
	** available as an SQL dump from ftp://ftp.ncbi.nih.gov/pub/taxonomy/ (see the README file)		** available as an SQL dump from ftp://ftp.ncbi.nih.gov/pub/taxonomy/ (see the README file)
	** the converted tree in Newick format is available from http://itol.embl.de/other_trees.shtml, as follows		** the converted tree in Newick format is available compressed from http://itol.embl.de/other_trees.shtml, as follows
	*** [~~[file~~:~~Ncbi complete collapsed with names~~.~~tre]]:~~ complete tree, using scientific names; internal nodes with only one child are removed. ~~This is a newick tree, but can only upload to wiki as~~ .tre~~. Switch back to .~~newick ~~for github.~~		* [http://itol.embl.de/ncbi_tree/ncbi_complete_collapsed_with_names.newick.gz complete tree, using scientific names, internal nodes with only one child are removed]. * also [[File:Ncbi complete collapsed with names.tre\| uploaded here as the uncompressed newick tree]]
	*** '''Note:''' This tree has ''every'' taxa in NCBI, including ones like "Insertion_sequence_IS2" and "Plasmid_pHV2". For Phylotastic, we would probably want to generate a tree that is some subset of this.		*** '''Note:''' This tree has ''every'' taxa in NCBI, including ones like "Insertion_sequence_IS2" and "Plasmid_pHV2". For Phylotastic, we would probably want to generate a tree that is some subset of this.
	** manually curated		** manually curated
	** FYI, NCBI provides an interactive way to get a tree phylotastically (http://www.ncbi.nlm.nih.gov/guide/howto/gen-com-tree/)		** FYI, NCBI provides an interactive way to get a tree phylotastically (http://www.ncbi.nlm.nih.gov/guide/howto/gen-com-tree/)
			** URI: http://www.ncbi.nlm.nih.gov/Taxonomy/CommonTree/normalized

	==== Not annotated yet ====		==== Not annotated yet ====