User Tools

Site Tools


Menu

 ?

About

Basic concepts and guidelines

User Guides

For users

For mappers

Technical

Learn about Dokuwiki

public:using_the_rdf_endpoints_to_pull_data_from_ckan

Using the API to pull/push data from/to CKAN

Who is this guide for

  • Developers willing to access data in LOD formats
  • Partners willing to integrate ODM data into their platforms

What this guide teaches

  • How to call the catalog endpoint to obtain a list of datasets
  • How to get a particular dataset in various LOD formats

Things to know beforehand

What is Linked Open Data (LOD) and what is the difference from Open Data

Quoted from linkeddata.org:

Linked Data is about using the Web to connect related data that wasn't previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as “a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF.”

RDF is not a format, it is a specification

The Resource Description Framework (RDF) is a family of World Wide Web Consortium (W3C) specifications originally designed as a metadata data model. It is used for describing graph-like ( nodes and vertices ) relationships between pieces of information by using RDF Triples.

An RDF triple contains three components:

- the subject, which is an RDF URI reference or a blank node. - the predicate, which is an RDF URI reference. - the object, which is an RDF URI reference, a literal or a blank node.

It is important to distinguish the RDF/XML format from the abstract RDF model itself. Although the RDF/XML format is still in use, other RDF serializations are now preferred by many RDF users, both because they are more human-friendly,[32] and because some RDF graphs are not representable in RDF/XML due to restrictions on the syntax of XML QNames.

The most common RDF serialization formats

From RDF definition on wikipedia

  • Turtle: a compact, human-friendly format.
  • N-Triples: a very simple, easy-to-parse, line-based format that is not as compact as Turtle.
  • N-Quads: a superset of N-Triples, for serializing multiple RDF graphs.
  • JSON-LD: a JSON-based serialization.
  • N3 or Notation3: a non-standard serialization that is very similar to Turtle, but has some additional features, such as the ability to define inference rules.
  • RDF/XML: an XML-based syntax that was the first standard format for serializing RDF.

This feature is based on CKAN DCAT extension

ODM's CKAN instance exposes this functionality thanks to the great job done by the contributors of https://github.com/ckan/ckanext-dcat, a CKAN extension for exposing data in RDF, and many more things.

RDF DCAT endpoints

RDF representations of a particular dataset can accessed using the following endpoint:

https://data.opendevelopmentmekong.net/dataset/{dataset-id}.{format}

The extension will determine the RDF serialization format returned. The currently supported values are:

Extension Format
xml RDF/XML
ttl Turtle
n3 Notation3
jsonld JSON-LD

The fallback rdf format defaults to RDF/XML.

Here's an example of the different formats available:

The catalog endpoint

Additionally to the individual dataset representations, there is a catalog-wide endpoint for retrieving multiple datasets at the same time (the datasets are paginated, see below for details):

https://data.opendevelopmentmekong.net/catalog.{format}?[page={page}]&[modified_date={date}]

As described previously, the extension will determine the RDF serialization format returned.

  http://data.opendevelopmentmekong.net/catalog.rdf
  http://data.opendevelopmentmekong.net/catalog.xml
  http://data.opendevelopmentmekong.net/catalog.ttl

The number of datasets returned is limited. The response will include paging info, serialized using the Hydra vocabulary. The different terms are self-explanatory, and can be used by clients to iterate the catalog:

@prefix hydra: <http://www.w3.org/ns/hydra/core#> .

<http://example.com/catalog.ttl?page=1> a hydra:PagedCollection ;
    hydra:firstPage "http://example.com/catalog.ttl?page=1" ;
    hydra:itemsPerPage 100 ;
    hydra:lastPage "http://example.com/catalog.ttl?page=3" ;
    hydra:nextPage "http://example.com/catalog.ttl?page=2" ;
    hydra:totalItems 283 .

The default number of datasets returned is 100.

The catalog endpoint also supports a modified_date parameter to restrict datasets to those modified from a certain date. The parameter value should be a valid ISO-8601 date:

http://data.opendevelopmentmekong.net/catalog.xml?modified_since=2015-07-24

Metadata terms being used

Datasets

Metadata field name ODM Ckan internal id Standard term Standard term uri
Title title_translated dcterms:title http://purl.org/dc/terms/title
Description notes_translated dcterms:description http://purl.org/dc/terms/description
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
License odm_license dcterms:licence http://purl.org/dc/terms/licence
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Access and use constraints odm_access_and_use_constraints md:useconstraints http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
Version version doap:version http://usefulinc.com/ns/doap#version
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Language odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Date created odm_date_created dcterms:created http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created
Date uploaded odm_date_uploaded schema:uploadDate http://schema.org/uploadDate
Date modified odm_date_modified dcterms:modified http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified
Geographical area (Spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Accuracy odm_accuracy dqm:accuracy http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy
Logical consistency odm_logical_consistency dq:LogicalConsistency http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency
Completeness odm_completeness dq:Completeness http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness
Source(s) odm_source dcterms:source http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source
Attributes odm_attributes omn:Attribute http://open-multinet.info/ontology/omn#isAttributeOf

Library records

Metadata field name ODM Ckan internal id Standard term Standard term uri
Document type document_type agls:documentType http://www.agls.gov.au/agls/terms/documentType
Language of document odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Formal full title title_translated dcterms:title http://purl.org/dc/terms/title
Short title (alternative/varying form of title) marc21_246 gc:shortTitle http://www.oegov.org/core/owl/gc#shortTitle
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
Short summary (contents) notes_translated dcterms:description http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#description
Geographic area (spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Access and use constraints odm_access_and_use_constraints md:useconstraints http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints
Version / Edition version doap:version http://usefulinc.com/ns/doap#version
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
Date uploaded odm_date_uploaded schema:uploadDate http://schema.org/uploadDate
License license_id dcterms:licence http://purl.org/dc/terms/licence
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Author (individual) marc21_100 opus:author http://lsdis.cs.uga.edu/projects/semdis/opus#author
Author (corporate) marc21_110 opus:author http://lsdis.cs.uga.edu/projects/semdis/opus#author
Co-author (individual) marc21_700 opus:coauthor http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor
Co-Author (corporate) marc21_710 opus:coauthor http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor
ISBN number marc21_020 opus:isbn
ISSN number marc21_022 dbpedia-owl:issn http://dbpedia.org/ontology/issn
Publication place marc21_260a mrel:pup http://id.loc.gov/vocabulary/relators/pup
Publisher marc21_260b dcterms:publisher http://purl.org/dc/terms/publisher
Publication date marc21_260c dcterms:issued http://purl.org/dc/terms/issued
Pagination marc21_300 bibo:numPages http://bibliontology.com/bibo/bibo.php#numPages
General note marc21_500 skos:note https://www.w3.org/2009/08/skos-reference/skos.html#note
Legacy reference document odm_reference_document pproc:documentReference http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference
Province(s) odm_province
Date modified odm_date_modified
Keywords odm_keywords

Laws records

Metadata field name ODM Ckan internal id Standard term Standard term uri
Document type odm_document_type agls:documentType http://www.agls.gov.au/agls/terms/documentType
Document reference number odm_document_number dbpedia-owl:documentNumber http://dbpedia.org/ontology/documentNumber
Language odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Formal full title title_translated dcterms:title http://purl.org/dc/terms/title
Alternative/short title odm_short_title gc:shortTitle http://www.oegov.org/core/owl/gc#shortTitle
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
Short summary notes_translated dcterms:description http://purl.org/dc/terms/description
Geographic area (spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
License license_id dcterms:licence http://purl.org/dc/terms/licence
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Adoption date/Promulgation date/Signing odm_promulgation_date bibframe:legalDate http://id.loc.gov/ontologies/bibframe.html#p_legalDate
Short notes of change odm_laws_previous_changes_notes skos:changeNote https://www.w3.org/2009/08/skos-reference/skos.html#changeNote
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Notes odm_laws_notes skos:note https://www.w3.org/2009/08/skos-reference/skos.html#note
Legacy reference document odm_reference_document pproc:documentReference http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference
Province(s) odm_province
Date modified odm_date_modified
Date updated odm_date_updated
Keywords odm_keywords
Issuing agency/parties odm_laws_issuing_agency_parties
Implementing agencies odm_laws_implementing_agencies
odm_laws_primary_policy_reference_point
odm_access_and_use_constraints
Status odm_laws_status
Version date (of draft) odm_laws_version_date
Effective/Enforced Date odm_effective_date
Previous legal document odm_laws_previous_legal_document
odm_laws_previous_changes_notes
Parent document odm_laws_parent_document
Child Document odm_laws_child_document
Other reference or supporting documents odm_laws_other_references
Publication reference odm_laws_official_publication_reference
Links to source odm_laws_source

Agreements records

Metadata field name ODM Ckan internal id Standard term Standard term uri
Title title_translated dcterms:title http://purl.org/dc/terms/title
Description notes_translated dcterms:description http://purl.org/dc/terms/description
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
License odm_license dcterms:licence http://purl.org/dc/terms/licence
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Access and use constraints odm_access_and_use_constraints md:useconstraints http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
Version version doap:version http://usefulinc.com/ns/doap#version
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Language odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Date created odm_date_created dcterms:created http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created
Date uploaded odm_date_uploaded schema:uploadDate http://schema.org/uploadDate
Date modified odm_date_modified dcterms:modified http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified
Geographical area (Spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Accuracy odm_accuracy dqm:accuracy http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy
Logical consistency odm_logical_consistency dq:LogicalConsistency http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency
Completeness odm_completeness dq:Completeness http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness
Source(s) odm_source dcterms:source http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source
Attributes odm_attributes omn:Attribute http://open-multinet.info/ontology/omn#isAttributeOf
Document type odm_agreement_document_type
Participating share odm_agreement_participating_share
Contracting Parties odm_agreement_contracting_parties
Government Entity (multiple) odm_agreement_government_entity
Concession/License name odm_agreement_concession_name
Disclosure Mode odm_agreement_disclosure_mode
Document reference no. odm_agreement_document_reference_number
LandMatrix deal No. odm_agreement_landmatrix_no
Copyright odm_copyright
odm_access_and_use_constraints
Organization owner_org
Version / Edition version
Language of document odm_language
Date created odm_date_created
Date uploaded odm_date_uploaded
Date modified odm_date_modified
Province(s) odm_province
odm_metadata_reference_information
Amendment to agreement odm_agreement_amendment_to_contract
Signature date odm_agreement_signature_date
odm_agreement_short_notes_of_change
Links to source odm_agreement_source
odm_contact
odm_agreement_notes
Granted area odm_agreement_granted_area
Contract term odm_agreement_contract_term
Contract term unit odm_agreement_contract_term_unit
Payment for concession fees odm_agreement_payment_for_concession_fees
Payment for concession fees unit odm_agreement_payment_for_concession_fees_unit
Guaranty deposit for contract implementation ($) odm_agreement_guaranty_deposit_ for_contract_implementation
Land use planning odm_agreement_land_use_planning
Land use planning unit odm_agreement_land_use_planning_unit
odm_agreement_parties_obligations
odm_agreement_job_creation_summary
Number of created jobs odm_agreement_job_creation_number
odm_agreement_training_summary
Fund allocation for training odm_agreement_training_number
odm_agreement_environmental_protection
odm_agreement_sociocultural_protection
odm_agreement_fiscal_duties_summary
Total amount of fiscal duties odm_agreement_fiscal_duties_number
Environmental impact assessment (EIA) odm_agreement_eia
Total amount of social fund odm_agreement_social_fund
odm_agreement_environmental_fund_summary
Total amount of environmental funds odm_agreement_environmental_funds
odm_agreement_suspension_revocation_termination
Related documents odm_agreement_related_documents
Related Project odm_agreement_suspension_related_project
Legacy reference document odm_reference_document
Keywords odm_keywords
Open Contracting Identifier open_contracting_id

Visualizing ODM's datasets

TBD

To improve

Split multilingual versions using lang="CODE"

Currently, multilingual values are not exposed following RDF standards but as json object.

Use URIs for standard values such as odm_spatial_range ( gn:countryCode ) or taxonomy ( foaf:topics )

Instead of using just literals for the object of the different predicates, as is the case now, standard values (such as a country code) should be exposed via an URI.

Do not generate Empty/None triples

In case a triple has the value None, it should not be generated.

Implement mapping between ODM taxonomy terms and terms in standard vocabularies (LandVoc, etc...)

In order to expose topics using standard terms

public/using_the_rdf_endpoints_to_pull_data_from_ckan.txt · Last modified: 2020/06/23 15:05 (external edit)