====== Using the API to pull/push data from/to CKAN ======

===== Who is this guide for =====

  * Developers willing to access data in LOD formats
  * Partners willing to integrate ODM data into their platforms

===== What this guide teaches =====

  * How to call the catalog endpoint to obtain a list of datasets
  * How to get a particular dataset in various LOD formats 
===== Things to know beforehand =====

==== What is Linked Open Data (LOD) and what is the difference from Open Data ====

Quoted from [[http://linkeddata.org|linkeddata.org]]:

//Linked Data is about using the Web to connect related data that wasn't previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as "a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF."//

==== RDF is not a format, it is a specification ====

The Resource Description Framework (RDF) is a family of World Wide Web Consortium (W3C) specifications originally designed as a metadata data model. It is used for describing graph-like ( nodes and vertices ) relationships between pieces of information by using RDF Triples.

An RDF triple contains three components:

- the **subject**, which is an RDF URI reference or a blank node.
- the **predicate**, which is an RDF URI reference.
- the **object**, which is an RDF URI reference, a literal or a blank node.

It is important to distinguish the RDF/XML format from the abstract RDF model itself. Although the RDF/XML format is still in use, other RDF serializations are now preferred by many RDF users, both because they are more human-friendly,[32] and because some RDF graphs are not representable in RDF/XML due to restrictions on the syntax of XML QNames.

==== The most common RDF serialization formats ====

From [[https://en.wikipedia.org/wiki/Resource_Description_Framework#Serialization_formats|RDF definition on wikipedia]]

  * **Turtle:** a compact, human-friendly format.
  * **N-Triples:** a very simple, easy-to-parse, line-based format that is not as compact as Turtle.
  * **N-Quads:** a superset of N-Triples, for serializing multiple RDF graphs.
  * **JSON-LD:** a JSON-based serialization.
  * **N3 or Notation3:** a non-standard serialization that is very similar to Turtle, but has some additional features, such as the ability to define inference rules.
  * **RDF/XML:** an XML-based syntax that was the first standard format for serializing RDF.

==== This feature is based on CKAN DCAT extension ====

ODM's CKAN instance exposes this functionality thanks to the great job done by the contributors of https://github.com/ckan/ckanext-dcat, a CKAN extension for exposing data in RDF, and many more things.

===== RDF DCAT endpoints =====

RDF representations of a particular dataset can accessed using the following endpoint:

<code>https://data.opendevelopmentmekong.net/dataset/{dataset-id}.{format}</code>

The extension will determine the RDF serialization format returned. The currently supported values are:

^ Extension      ^ Format ^  Media Type      
| xml   | RDF/XML | application/rdf+xml  
| ttl   | Turtle | text/turtle
| n3   | Notation3 | text/n3 
| jsonld   | JSON-LD | application/ld+json 


The fallback rdf format defaults to RDF/XML.

Here's an example of the different formats available:

  * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.rdf
  * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.ttl
  * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.n3
  * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.jsonld

===== The catalog endpoint =====

Additionally to the individual dataset representations, there is a catalog-wide endpoint for retrieving multiple datasets at the same time (the datasets are paginated, see below for details):

<code>https://data.opendevelopmentmekong.net/catalog.{format}?[page={page}]&[modified_date={date}]</code>

As described previously, the extension will determine the RDF serialization format returned.

    http://data.opendevelopmentmekong.net/catalog.rdf
    http://data.opendevelopmentmekong.net/catalog.xml
    http://data.opendevelopmentmekong.net/catalog.ttl

The number of datasets returned is limited. The response will include paging info, serialized using the Hydra vocabulary. The different terms are self-explanatory, and can be used by clients to iterate the catalog:

<code>
@prefix hydra: <http://www.w3.org/ns/hydra/core#> .

<http://example.com/catalog.ttl?page=1> a hydra:PagedCollection ;
    hydra:firstPage "http://example.com/catalog.ttl?page=1" ;
    hydra:itemsPerPage 100 ;
    hydra:lastPage "http://example.com/catalog.ttl?page=3" ;
    hydra:nextPage "http://example.com/catalog.ttl?page=2" ;
    hydra:totalItems 283 .
</code>

The default number of datasets returned is 100.

The catalog endpoint also supports a modified_date parameter to restrict datasets to those modified from a certain date. The parameter value should be a valid ISO-8601 date:

http://data.opendevelopmentmekong.net/catalog.xml?modified_since=2015-07-24

===== Metadata terms being used =====

==== Datasets ====

^ Metadata field name ^ ODM Ckan internal id    ^  Standard term   ^ Standard term uri ^
| Title               | title_translated        | dcterms:title    | http://purl.org/dc/terms/title |
| Description               | notes_translated        | dcterms:description    | http://purl.org/dc/terms/description |
| Topics               | taxonomy        | foaf:topic    | http://xmlns.com/foaf/0.1/topic |
| License               | odm_license        | dcterms:licence    | http://purl.org/dc/terms/licence |
| Copyright               | odm_copyright        | cro:Copyright    | http://rhizomik.net/ontologies/copyrightonto.owl#copyright |
| Access and use constraints               | odm_access_and_use_constraints        | md:useconstraints    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints |
| Organization               | owner_org        | foaf:Organization    | http://xmlns.com/foaf/0.1/organization|
| Version               | version        | doap:version    | http://usefulinc.com/ns/doap#version |
| Contact               | odm_contact        | ebucore:Contact    | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact |
| Language               | odm_language        | dcterms:language    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language |
| Date created               | odm_date_created        | dcterms:created   | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created |
| Date uploaded               | odm_date_uploaded        | schema:uploadDate    | http://schema.org/uploadDate |
| Date modified               | odm_date_modified        | dcterms:modified    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified |
| Geographical area (Spatial range)               | odm_spatial_range        | gn:countryCode    | http://www.geonames.org/ontology#countryCode |
| Accuracy               | odm_accuracy        | dqm:accuracy    | http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy |
| Logical consistency               | odm_logical_consistency        | dq:LogicalConsistency    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency |
| Completeness               | odm_completeness        | dq:Completeness    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness |
| Source(s)               | odm_source        | dcterms:source    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source |
| Attributes               | odm_attributes        | omn:Attribute    | http://open-multinet.info/ontology/omn#isAttributeOf |
==== Library records ====

^ Metadata field name ^ ODM Ckan internal id    ^  Standard term   ^ Standard term uri ^
| Document type               | document_type        | agls:documentType   | http://www.agls.gov.au/agls/terms/documentType |
| Language of document               | odm_language        | dcterms:language    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language |
| Formal full title               | title_translated        | dcterms:title    | http://purl.org/dc/terms/title |
| Short title (alternative/varying form of title)               | marc21_246        | gc:shortTitle    | http://www.oegov.org/core/owl/gc#shortTitle |
| Topics               | taxonomy        | foaf:topic    | http://xmlns.com/foaf/0.1/topic |
| Short summary (contents)               | notes_translated        | dcterms:description    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#description |
| Geographic area (spatial range)               | odm_spatial_range        | gn:countryCode    | http://www.geonames.org/ontology#countryCode |
| Copyright               | odm_copyright        | cro:Copyright    | http://rhizomik.net/ontologies/copyrightonto.owl#copyright |
| Access and use constraints               | odm_access_and_use_constraints        | md:useconstraints    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints |
| Version / Edition | version        | doap:version    | http://usefulinc.com/ns/doap#version |
| Organization               | owner_org        | foaf:Organization    | http://xmlns.com/foaf/0.1/organization|
| Date uploaded               | odm_date_uploaded        | schema:uploadDate    | http://schema.org/uploadDate |
| License               | license_id        | dcterms:licence    | http://purl.org/dc/terms/licence |
| Contact               | odm_contact        | ebucore:Contact    | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact |
| Author (individual)               | marc21_100        | opus:author    | http://lsdis.cs.uga.edu/projects/semdis/opus#author |
| Author (corporate)               | marc21_110        | opus:author    | http://lsdis.cs.uga.edu/projects/semdis/opus#author |
| Co-author (individual)               | marc21_700        | opus:coauthor    | http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor |
| Co-Author (corporate)               | marc21_710        | opus:coauthor    | http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor |
| ISBN number               | marc21_020        | opus:isbn    | http://lsdis.cs.uga.edu/projects/semdis/opus#isbn
| ISSN number               | marc21_022        | dbpedia-owl:issn    | http://dbpedia.org/ontology/issn |
| Publication place               | marc21_260a        | mrel:pup    | http://id.loc.gov/vocabulary/relators/pup |
| Publisher               | marc21_260b        | dcterms:publisher    | http://purl.org/dc/terms/publisher |
| Publication date               | marc21_260c        |  dcterms:issued    | http://purl.org/dc/terms/issued |
| Pagination               | marc21_300        | bibo:numPages    | http://bibliontology.com/bibo/bibo.php#numPages|
| General note               | marc21_500        | skos:note |  https://www.w3.org/2009/08/skos-reference/skos.html#note |
| Legacy reference document | odm_reference_document        | pproc:documentReference    | http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference |
| Province(s) | odm_province  |     |  |
| Date modified | odm_date_modified  |     |  |
| Keywords | odm_keywords  |     |  |
==== Laws records ====

^ Metadata field name ^ ODM Ckan internal id    ^  Standard term   ^ Standard term uri ^
| Document type               | odm_document_type        | agls:documentType   | http://www.agls.gov.au/agls/terms/documentType |
| Document reference number               | odm_document_number        | dbpedia-owl:documentNumber    | http://dbpedia.org/ontology/documentNumber |
| Language               | odm_language        | dcterms:language    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language |
| Formal full title               | title_translated        | dcterms:title    | http://purl.org/dc/terms/title |
| Alternative/short title               | odm_short_title       | gc:shortTitle   | http://www.oegov.org/core/owl/gc#shortTitle |
| Topics               | taxonomy        | foaf:topic    | http://xmlns.com/foaf/0.1/topic |
| Short summary               | notes_translated       | dcterms:description    | http://purl.org/dc/terms/description |
| Geographic area (spatial range)               | odm_spatial_range        | gn:countryCode    | http://www.geonames.org/ontology#countryCode |
| Organization               | owner_org        | foaf:Organization    | http://xmlns.com/foaf/0.1/organization|
| License               | license_id        | dcterms:licence    | http://purl.org/dc/terms/licence |
| Copyright               | odm_copyright        | cro:Copyright    | http://rhizomik.net/ontologies/copyrightonto.owl#copyright |
| Adoption date/Promulgation date/Signing               | odm_promulgation_date        | bibframe:legalDate    | http://id.loc.gov/ontologies/bibframe.html#p_legalDate |
| Short notes of change               | odm_laws_previous_changes_notes        | skos:changeNote    | https://www.w3.org/2009/08/skos-reference/skos.html#changeNote |
| Contact               | odm_contact        | ebucore:Contact    | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact |
| Notes               | odm_laws_notes        | skos:note    | https://www.w3.org/2009/08/skos-reference/skos.html#note |
| Legacy reference document | odm_reference_document        | pproc:documentReference    | http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference |
| Province(s) | odm_province  |     |  |
| Date modified | odm_date_modified  |     |  |
| Date updated | odm_date_updated  |     |  |
| Keywords | odm_keywords  |     |  |
| Issuing agency/parties | odm_laws_issuing_agency_parties  |     |  |
| Implementing agencies	| odm_laws_implementing_agencies  |     |  |
|  |  odm_laws_primary_policy_reference_point  |     |  |
|  | odm_access_and_use_constraints  |     |  |
| Status  | odm_laws_status  |     |  |
| Version date (of draft)  | odm_laws_version_date  |     |  |
| Effective/Enforced Date	  | odm_effective_date  |     |  |
| Previous legal document	  | odm_laws_previous_legal_document  |     |  |
|	  | odm_laws_previous_changes_notes  |     |  |
| Parent document	  | odm_laws_parent_document  |     |  |
| Child Document	  | odm_laws_child_document  |     |  |
| Other reference or supporting documents	  | odm_laws_other_references  |     |  |
| Publication reference	  | odm_laws_official_publication_reference  |     |  |
| Links to source	  | odm_laws_source  |     |  |

==== Agreements records ====

^ Metadata field name ^ ODM Ckan internal id  ^  Standard term  ^ Standard term uri ^
| Title               | title_translated        | dcterms:title    | http://purl.org/dc/terms/title |
| Description               | notes_translated        | dcterms:description    | http://purl.org/dc/terms/description |
| Topics               | taxonomy        | foaf:topic    | http://xmlns.com/foaf/0.1/topic |
| License               | odm_license        | dcterms:licence    | http://purl.org/dc/terms/licence |
| Copyright               | odm_copyright        | cro:Copyright    | http://rhizomik.net/ontologies/copyrightonto.owl#copyright |
| Access and use constraints               | odm_access_and_use_constraints        | md:useconstraints    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints |
| Organization               | owner_org        | foaf:Organization    | http://xmlns.com/foaf/0.1/organization|
| Version               | version        | doap:version    | http://usefulinc.com/ns/doap#version |
| Contact               | odm_contact        | ebucore:Contact    | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact |
| Language               | odm_language        | dcterms:language    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language |
| Date created               | odm_date_created        | dcterms:created   | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created |
| Date uploaded               | odm_date_uploaded        | schema:uploadDate    | http://schema.org/uploadDate |
| Date modified               | odm_date_modified        | dcterms:modified    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified |
| Geographical area (Spatial range)               | odm_spatial_range        | gn:countryCode    | http://www.geonames.org/ontology#countryCode |
| Accuracy               | odm_accuracy        | dqm:accuracy    | http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy |
| Logical consistency               | odm_logical_consistency        | dq:LogicalConsistency    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency |
| Completeness               | odm_completeness        | dq:Completeness    | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness |
| Source(s)               | odm_source        | dcterms:source    | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source |
| Attributes               | odm_attributes        | omn:Attribute    | http://open-multinet.info/ontology/omn#isAttributeOf |
| Document type	| odm_agreement_document_type  | | |
| Participating share	| odm_agreement_participating_share  | | |
| Contracting Parties	| odm_agreement_contracting_parties  | | |
| Government Entity (multiple)	| odm_agreement_government_entity | | |
| Concession/License name	| odm_agreement_concession_name   | | |
| Disclosure Mode	| odm_agreement_disclosure_mode  | | |
| Document reference no.	| odm_agreement_document_reference_number  | | |
| LandMatrix deal No.	| odm_agreement_landmatrix_no          | | |
| Copyright	| odm_copyright  | | |
| | odm_access_and_use_constraints  | | |
| Organization	| owner_org  | | |
| Version / Edition	| version  | | |
| Language of document	| odm_language  | | |
| Date created	| odm_date_created  | | |
| Date uploaded	| odm_date_uploaded  | | |
| Date modified	| odm_date_modified  | | |
| Province(s)	| odm_province  | | |
|  | odm_metadata_reference_information  | | |
| Amendment to agreement	| odm_agreement_amendment_to_contract  | | |
| Signature date	| odm_agreement_signature_date  | | |
|  | odm_agreement_short_notes_of_change  | | |
| Links to source	| odm_agreement_source  | | |
|  | odm_contact  | | |
|  | odm_agreement_notes  | | |
| Granted area	| odm_agreement_granted_area  | | |
| Contract term	| odm_agreement_contract_term  | | |
| Contract term unit	| odm_agreement_contract_term_unit  | | |
| Payment for concession fees	| odm_agreement_payment_for_concession_fees  | | |
| Payment for concession fees unit	| odm_agreement_payment_for_concession_fees_unit  | | |
| Guaranty deposit for contract implementation ($)	| odm_agreement_guaranty_deposit_  for_contract_implementation  | | |
| Land use planning	| odm_agreement_land_use_planning  | | |
| Land use planning unit	| odm_agreement_land_use_planning_unit  | | |
| 	| odm_agreement_parties_obligations  | | |
|  | odm_agreement_job_creation_summary  | | |
| Number of created jobs	| odm_agreement_job_creation_number  | | |
|  | odm_agreement_training_summary  | | |
| Fund allocation for training	| odm_agreement_training_number  | | |
|  | odm_agreement_environmental_protection  | | |
|  | odm_agreement_sociocultural_protection  | | |
|  | odm_agreement_fiscal_duties_summary  | | |
| Total amount of fiscal duties	| odm_agreement_fiscal_duties_number  | | |
| Environmental impact assessment (EIA)	| odm_agreement_eia  | | |
| Total amount of social fund	| odm_agreement_social_fund  | | |
|  | odm_agreement_environmental_fund_summary  | | |
| Total amount of environmental funds	| odm_agreement_environmental_funds  | | |
|  | odm_agreement_suspension_revocation_termination  | | |
| Related documents	| odm_agreement_related_documents  | | |
| Related Project	| odm_agreement_suspension_related_project  | | |
| Legacy reference document	| odm_reference_document  | | |
| Keywords	| odm_keywords  | | |
| Open Contracting Identifier 	| open_contracting_id  |   | |

===== Visualizing ODM's datasets =====

TBD

===== To improve =====

==== Split multilingual versions using lang="CODE" ====

Currently, multilingual values are not exposed following RDF standards but as json object.

==== Use URIs for standard values such as odm_spatial_range ( gn:countryCode )  or taxonomy ( foaf:topics ) ====

Instead of using just literals for the object of the different predicates, as is the case now, standard values (such as a country code) should be exposed via an URI.

==== Do not generate Empty/None triples ====

In case a triple has the value **None**, it should not be generated.

==== Implement mapping between ODM taxonomy terms and terms in standard vocabularies (LandVoc, etc...) ====

In order to expose topics using standard terms