User Tools

Site Tools


public:using_the_rdf_endpoints_to_pull_data_from_ckan

Using the API to pull/push data from/to CKAN

Who is this guide for

  • Developers willing to access data in LOD formats
  • Partners willing to integrate ODM data into their platforms

What this guide teaches

  • How to call the catalog endpoint to obtain a list of datasets
  • How to get a particular dataset in various LOD formats

Things to know in forehand

What is Linked Open Data (LOD) and what is the difference from Open Data

Quoted from linkeddata.org:

Linked Data is about using the Web to connect related data that wasn't previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as “a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF.”

RDF is not a format, it is a specification

The Resource Description Framework (RDF) is a family of World Wide Web Consortium (W3C) specifications originally designed as a metadata data model. It is used for describing graph-like ( nodes and vertices ) relationships between pieces of information by using RDF Triples.

An RDF triple contains three components:

- the subject, which is an RDF URI reference or a blank node. - the predicate, which is an RDF URI reference. - the object, which is an RDF URI reference, a literal or a blank node.

It is important to distinguish the RDF/XML format from the abstract RDF model itself. Although the RDF/XML format is still in use, other RDF serializations are now preferred by many RDF users, both because they are more human-friendly,[32] and because some RDF graphs are not representable in RDF/XML due to restrictions on the syntax of XML QNames.

The most common RDF serialization formats

From RDF definition on wikipedia

  • Turtle: a compact, human-friendly format.
  • N-Triples: a very simple, easy-to-parse, line-based format that is not as compact as Turtle.
  • N-Quads: a superset of N-Triples, for serializing multiple RDF graphs.
  • JSON-LD: a JSON-based serialization.
  • N3 or Notation3: a non-standard serialization that is very similar to Turtle, but has some additional features, such as the ability to define inference rules.
  • RDF/XML: an XML-based syntax that was the first standard format for serializing RDF.

This feature is based on CKAN DCAT extension

ODM's CKAN instance exposes this functionality thanks to the great job done by the contributors of https://github.com/ckan/ckanext-dcat, a CKAN extension for exposing data in RDF, and many more things.

RDF DCAT endpoints

RDF representations of a particular dataset can accessed using the following endpoint:

https://data.opendevelopmentmekong.net/dataset/{dataset-id}.{format}

The extension will determine the RDF serialization format returned. The currently supported values are:

Extension Format
xml RDF/XML
ttl Turtle
n3 Notation3
jsonld JSON-LD

The fallback rdf format defaults to RDF/XML.

Here's an example of the different formats available:

The catalog endpoint

Additionally to the individual dataset representations, there is a catalog-wide endpoint for retrieving multiple datasets at the same time (the datasets are paginated, see below for details):

https://data.opendevelopmentmekong.net/catalog.{format}?[page={page}]&[modified_date={date}]

As described previously, the extension will determine the RDF serialization format returned.

  http://data.opendevelopmentmekong.net/catalog.rdf
  http://data.opendevelopmentmekong.net/catalog.xml
  http://data.opendevelopmentmekong.net/catalog.ttl

The number of datasets returned is limited. The response will include paging info, serialized using the Hydra vocabulary. The different terms are self-explanatory, and can be used by clients to iterate the catalog:

@prefix hydra: <http://www.w3.org/ns/hydra/core#> .

<http://example.com/catalog.ttl?page=1> a hydra:PagedCollection ;
    hydra:firstPage "http://example.com/catalog.ttl?page=1" ;
    hydra:itemsPerPage 100 ;
    hydra:lastPage "http://example.com/catalog.ttl?page=3" ;
    hydra:nextPage "http://example.com/catalog.ttl?page=2" ;
    hydra:totalItems 283 .

The default number of datasets returned is 100.

The catalog endpoint also supports a modified_date parameter to restrict datasets to those modified from a certain date. The parameter value should be a valid ISO-8601 date:

http://data.opendevelopmentmekong.net/catalog.xml?modified_since=2015-07-24

Metadata terms being used

Datasets

Metadata field name ODM Ckan internal id Standard term Standard term uri
Title title_translated dcterms:title http://purl.org/dc/terms/title
Description notes_translated dcterms:description http://purl.org/dc/terms/description
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
License license_id dcterms:licence http://purl.org/dc/terms/licence
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Access and use constraints odm_access_and_use_constraints md:useconstraints http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
Version version doap:version http://usefulinc.com/ns/doap#version
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Language odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Date created odm_date_created dcterms:created http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created
Date uploaded odm_date_uploaded schema:uploadDate http://schema.org/uploadDate
Date modified odm_date_modified dcterms:modified http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified
Geographical area (Spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Accuracy odm_accuracy dqm:accuracy http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy
Logical consistency odm_logical_consistency dq:LogicalConsistency http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency
Completeness odm_completeness dq:Completeness http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness
Source(s) odm_source dcterms:source http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source
Attributes odm_attributes omn:Attribute http://open-multinet.info/ontology/omn#isAttributeOf

Library records

Metadata field name ODM Ckan internal id Standard term Standard term uri
Document type document_type agls:documentType http://www.agls.gov.au/agls/terms/documentType
Language of document odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Formal full title title_translated dcterms:title http://purl.org/dc/terms/title
Short title (alternative/varying form of title) marc21_246 gc:shortTitle http://www.oegov.org/core/owl/gc#shortTitle
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
Short summary (contents) notes_translated dcterms:description http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#description
Geographic area (spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Access and use constraints odm_access_and_use_constraints md:useconstraints http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints
Version / Edition version doap:version http://usefulinc.com/ns/doap#version
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
Date uploaded odm_date_uploaded schema:uploadDate http://schema.org/uploadDate
License license_id dcterms:licence http://purl.org/dc/terms/licence
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Author (individual) marc21_100 opus:author http://lsdis.cs.uga.edu/projects/semdis/opus#author
Author (corporate) marc21_110 opus:author http://lsdis.cs.uga.edu/projects/semdis/opus#author
Co-author (individual) marc21_700 opus:coauthor http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor
Co-Author (corporate) marc21_710 opus:coauthor http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor
ISBN number marc21_020 opus:isbn
ISSN number marc21_022 dbpedia-owl:issn http://dbpedia.org/ontology/issn
Publication place marc21_260a mrel:pup http://id.loc.gov/vocabulary/relators/pup
Publisher marc21_260b dcterms:publisher http://purl.org/dc/terms/publisher
Publication date marc21_260c dcterms:issued http://purl.org/dc/terms/issued
Pagination marc21_300 bibo:numPages http://bibliontology.com/bibo/bibo.php#numPages
General note marc21_500 skos:note https://www.w3.org/2009/08/skos-reference/skos.html#note
Legacy reference document odm_reference_document pproc:documentReference http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference

Laws records

Metadata field name ODM Ckan internal id Standard term Standard term uri
Document type document_type agls:documentType http://www.agls.gov.au/agls/terms/documentType
Document reference number odm_document_number dbpedia-owl:documentNumber http://dbpedia.org/ontology/documentNumber
Language odm_language dcterms:language http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language
Formal full title title_translated dcterms:title http://purl.org/dc/terms/title
Alternative/short title odm_short_title gc:shortTitle http://www.oegov.org/core/owl/gc#shortTitle
Topics taxonomy foaf:topic http://xmlns.com/foaf/0.1/topic
Short summary notes_translated dcterms:description http://purl.org/dc/terms/description
Geographic area (spatial range) odm_spatial_range gn:countryCode http://www.geonames.org/ontology#countryCode
Organization owner_org foaf:Organization http://xmlns.com/foaf/0.1/organization
License license_id dcterms:licence http://purl.org/dc/terms/licence
Copyright odm_copyright cro:Copyright http://rhizomik.net/ontologies/copyrightonto.owl#copyright
Adoption date/Promulgation date/Signing odm_promulgation_date bibframe:legalDate http://id.loc.gov/ontologies/bibframe.html#p_legalDate
Short notes of change odm_laws_previous_changes_notes skos:changeNote https://www.w3.org/2009/08/skos-reference/skos.html#changeNote
Contact odm_contact ebucore:Contact https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact
Notes odm_laws_notes skos:note https://www.w3.org/2009/08/skos-reference/skos.html#note
Legacy reference document odm_reference_document pproc:documentReference http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference

Visualizing ODM's datasets

TBD

To improve

Split multilingual versions using lang="CODE"

Currently, multilingual values are not exposed following RDF standards but as json object.

Use URIs for standard values such as odm_spatial_range ( gn:countryCode ) or taxonomy ( foaf:topics )

Instead of using just literals for the object of the different predicates, as is the case now, standard values (such as a country code) should be exposed via an URI.

Do not generate Empty/None triples

In case a triple has the value None, it should not be generated.

Implement mapping between ODM taxonomy terms and terms in standard vocabularies (LandVoc, etc...)

In order to expose topics using standard terms

public/using_the_rdf_endpoints_to_pull_data_from_ckan.txt · Last modified: 2016/05/18 15:49 by acorbi