public:using_the_rdf_endpoints_to_pull_data_from_ckan
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
public:using_the_rdf_endpoints_to_pull_data_from_ckan [2016/05/04 14:26] acorbi |
public:using_the_rdf_endpoints_to_pull_data_from_ckan [2020/06/23 15:05] (current) |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| ===== Who is this guide for ===== | ===== Who is this guide for ===== | ||
| - | * Developers willing to access data in OLD formats | + | * Developers willing to access data in LOD formats |
| * Partners willing to integrate ODM data into their platforms | * Partners willing to integrate ODM data into their platforms | ||
| Line 9: | Line 9: | ||
| * How to call the catalog endpoint to obtain a list of datasets | * How to call the catalog endpoint to obtain a list of datasets | ||
| - | * How to get a particular dataset in various OLD formats | + | * How to get a particular dataset in various LOD formats |
| - | ===== Things to know in forehand ===== | + | ===== Things to know beforehand ===== |
| - | ==== What is Open Linked Data (OLD) and what is the difference from Open Data ==== | + | ==== What is Linked Open Data (LOD) and what is the difference from Open Data ==== |
| Quoted from [[http://linkeddata.org|linkeddata.org]]: | Quoted from [[http://linkeddata.org|linkeddata.org]]: | ||
| Line 24: | Line 24: | ||
| An RDF triple contains three components: | An RDF triple contains three components: | ||
| - | - the subject, which is an RDF URI reference or a blank node. | + | - the **subject**, which is an RDF URI reference or a blank node. |
| - | - the predicate, which is an RDF URI reference. | + | - the **predicate**, which is an RDF URI reference. |
| - | - the object, which is an RDF URI reference, a literal or a blank node. | + | - the **object**, which is an RDF URI reference, a literal or a blank node. |
| It is important to distinguish the RDF/XML format from the abstract RDF model itself. Although the RDF/XML format is still in use, other RDF serializations are now preferred by many RDF users, both because they are more human-friendly,[32] and because some RDF graphs are not representable in RDF/XML due to restrictions on the syntax of XML QNames. | It is important to distinguish the RDF/XML format from the abstract RDF model itself. Although the RDF/XML format is still in use, other RDF serializations are now preferred by many RDF users, both because they are more human-friendly,[32] and because some RDF graphs are not representable in RDF/XML due to restrictions on the syntax of XML QNames. | ||
| Line 44: | Line 44: | ||
| ODM's CKAN instance exposes this functionality thanks to the great job done by the contributors of https://github.com/ckan/ckanext-dcat, a CKAN extension for exposing data in RDF, and many more things. | ODM's CKAN instance exposes this functionality thanks to the great job done by the contributors of https://github.com/ckan/ckanext-dcat, a CKAN extension for exposing data in RDF, and many more things. | ||
| + | |||
| + | ===== RDF DCAT endpoints ===== | ||
| + | |||
| + | RDF representations of a particular dataset can accessed using the following endpoint: | ||
| + | |||
| + | <code>https://data.opendevelopmentmekong.net/dataset/{dataset-id}.{format}</code> | ||
| + | |||
| + | The extension will determine the RDF serialization format returned. The currently supported values are: | ||
| + | |||
| + | ^ Extension ^ Format ^ Media Type | ||
| + | | xml | RDF/XML | application/rdf+xml | ||
| + | | ttl | Turtle | text/turtle | ||
| + | | n3 | Notation3 | text/n3 | ||
| + | | jsonld | JSON-LD | application/ld+json | ||
| + | |||
| + | |||
| + | The fallback rdf format defaults to RDF/XML. | ||
| + | |||
| + | Here's an example of the different formats available: | ||
| + | |||
| + | * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.rdf | ||
| + | * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.ttl | ||
| + | * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.n3 | ||
| + | * https://data.opendevelopmentmekong.net/dataset/economic-land-concessions.jsonld | ||
| ===== The catalog endpoint ===== | ===== The catalog endpoint ===== | ||
| - | ===== Endpoint for a particular dataset ===== | + | Additionally to the individual dataset representations, there is a catalog-wide endpoint for retrieving multiple datasets at the same time (the datasets are paginated, see below for details): |
| + | |||
| + | <code>https://data.opendevelopmentmekong.net/catalog.{format}?[page={page}]&[modified_date={date}]</code> | ||
| + | |||
| + | As described previously, the extension will determine the RDF serialization format returned. | ||
| + | |||
| + | http://data.opendevelopmentmekong.net/catalog.rdf | ||
| + | http://data.opendevelopmentmekong.net/catalog.xml | ||
| + | http://data.opendevelopmentmekong.net/catalog.ttl | ||
| + | |||
| + | The number of datasets returned is limited. The response will include paging info, serialized using the Hydra vocabulary. The different terms are self-explanatory, and can be used by clients to iterate the catalog: | ||
| + | |||
| + | <code> | ||
| + | @prefix hydra: <http://www.w3.org/ns/hydra/core#> . | ||
| + | |||
| + | <http://example.com/catalog.ttl?page=1> a hydra:PagedCollection ; | ||
| + | hydra:firstPage "http://example.com/catalog.ttl?page=1" ; | ||
| + | hydra:itemsPerPage 100 ; | ||
| + | hydra:lastPage "http://example.com/catalog.ttl?page=3" ; | ||
| + | hydra:nextPage "http://example.com/catalog.ttl?page=2" ; | ||
| + | hydra:totalItems 283 . | ||
| + | </code> | ||
| + | |||
| + | The default number of datasets returned is 100. | ||
| + | |||
| + | The catalog endpoint also supports a modified_date parameter to restrict datasets to those modified from a certain date. The parameter value should be a valid ISO-8601 date: | ||
| + | |||
| + | http://data.opendevelopmentmekong.net/catalog.xml?modified_since=2015-07-24 | ||
| + | |||
| + | ===== Metadata terms being used ===== | ||
| + | |||
| + | ==== Datasets ==== | ||
| + | |||
| + | ^ Metadata field name ^ ODM Ckan internal id ^ Standard term ^ Standard term uri ^ | ||
| + | | Title | title_translated | dcterms:title | http://purl.org/dc/terms/title | | ||
| + | | Description | notes_translated | dcterms:description | http://purl.org/dc/terms/description | | ||
| + | | Topics | taxonomy | foaf:topic | http://xmlns.com/foaf/0.1/topic | | ||
| + | | License | odm_license | dcterms:licence | http://purl.org/dc/terms/licence | | ||
| + | | Copyright | odm_copyright | cro:Copyright | http://rhizomik.net/ontologies/copyrightonto.owl#copyright | | ||
| + | | Access and use constraints | odm_access_and_use_constraints | md:useconstraints | http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints | | ||
| + | | Organization | owner_org | foaf:Organization | http://xmlns.com/foaf/0.1/organization| | ||
| + | | Version | version | doap:version | http://usefulinc.com/ns/doap#version | | ||
| + | | Contact | odm_contact | ebucore:Contact | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact | | ||
| + | | Language | odm_language | dcterms:language | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language | | ||
| + | | Date created | odm_date_created | dcterms:created | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created | | ||
| + | | Date uploaded | odm_date_uploaded | schema:uploadDate | http://schema.org/uploadDate | | ||
| + | | Date modified | odm_date_modified | dcterms:modified | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified | | ||
| + | | Geographical area (Spatial range) | odm_spatial_range | gn:countryCode | http://www.geonames.org/ontology#countryCode | | ||
| + | | Accuracy | odm_accuracy | dqm:accuracy | http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy | | ||
| + | | Logical consistency | odm_logical_consistency | dq:LogicalConsistency | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency | | ||
| + | | Completeness | odm_completeness | dq:Completeness | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness | | ||
| + | | Source(s) | odm_source | dcterms:source | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source | | ||
| + | | Attributes | odm_attributes | omn:Attribute | http://open-multinet.info/ontology/omn#isAttributeOf | | ||
| + | ==== Library records ==== | ||
| + | |||
| + | ^ Metadata field name ^ ODM Ckan internal id ^ Standard term ^ Standard term uri ^ | ||
| + | | Document type | document_type | agls:documentType | http://www.agls.gov.au/agls/terms/documentType | | ||
| + | | Language of document | odm_language | dcterms:language | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language | | ||
| + | | Formal full title | title_translated | dcterms:title | http://purl.org/dc/terms/title | | ||
| + | | Short title (alternative/varying form of title) | marc21_246 | gc:shortTitle | http://www.oegov.org/core/owl/gc#shortTitle | | ||
| + | | Topics | taxonomy | foaf:topic | http://xmlns.com/foaf/0.1/topic | | ||
| + | | Short summary (contents) | notes_translated | dcterms:description | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#description | | ||
| + | | Geographic area (spatial range) | odm_spatial_range | gn:countryCode | http://www.geonames.org/ontology#countryCode | | ||
| + | | Copyright | odm_copyright | cro:Copyright | http://rhizomik.net/ontologies/copyrightonto.owl#copyright | | ||
| + | | Access and use constraints | odm_access_and_use_constraints | md:useconstraints | http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints | | ||
| + | | Version / Edition | version | doap:version | http://usefulinc.com/ns/doap#version | | ||
| + | | Organization | owner_org | foaf:Organization | http://xmlns.com/foaf/0.1/organization| | ||
| + | | Date uploaded | odm_date_uploaded | schema:uploadDate | http://schema.org/uploadDate | | ||
| + | | License | license_id | dcterms:licence | http://purl.org/dc/terms/licence | | ||
| + | | Contact | odm_contact | ebucore:Contact | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact | | ||
| + | | Author (individual) | marc21_100 | opus:author | http://lsdis.cs.uga.edu/projects/semdis/opus#author | | ||
| + | | Author (corporate) | marc21_110 | opus:author | http://lsdis.cs.uga.edu/projects/semdis/opus#author | | ||
| + | | Co-author (individual) | marc21_700 | opus:coauthor | http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor | | ||
| + | | Co-Author (corporate) | marc21_710 | opus:coauthor | http://lsdis.cs.uga.edu/projects/semdis/opus#coauthor | | ||
| + | | ISBN number | marc21_020 | opus:isbn | http://lsdis.cs.uga.edu/projects/semdis/opus#isbn | ||
| + | | ISSN number | marc21_022 | dbpedia-owl:issn | http://dbpedia.org/ontology/issn | | ||
| + | | Publication place | marc21_260a | mrel:pup | http://id.loc.gov/vocabulary/relators/pup | | ||
| + | | Publisher | marc21_260b | dcterms:publisher | http://purl.org/dc/terms/publisher | | ||
| + | | Publication date | marc21_260c | dcterms:issued | http://purl.org/dc/terms/issued | | ||
| + | | Pagination | marc21_300 | bibo:numPages | http://bibliontology.com/bibo/bibo.php#numPages| | ||
| + | | General note | marc21_500 | skos:note | https://www.w3.org/2009/08/skos-reference/skos.html#note | | ||
| + | | Legacy reference document | odm_reference_document | pproc:documentReference | http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference | | ||
| + | | Province(s) | odm_province | | | | ||
| + | | Date modified | odm_date_modified | | | | ||
| + | | Keywords | odm_keywords | | | | ||
| + | ==== Laws records ==== | ||
| + | |||
| + | ^ Metadata field name ^ ODM Ckan internal id ^ Standard term ^ Standard term uri ^ | ||
| + | | Document type | odm_document_type | agls:documentType | http://www.agls.gov.au/agls/terms/documentType | | ||
| + | | Document reference number | odm_document_number | dbpedia-owl:documentNumber | http://dbpedia.org/ontology/documentNumber | | ||
| + | | Language | odm_language | dcterms:language | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language | | ||
| + | | Formal full title | title_translated | dcterms:title | http://purl.org/dc/terms/title | | ||
| + | | Alternative/short title | odm_short_title | gc:shortTitle | http://www.oegov.org/core/owl/gc#shortTitle | | ||
| + | | Topics | taxonomy | foaf:topic | http://xmlns.com/foaf/0.1/topic | | ||
| + | | Short summary | notes_translated | dcterms:description | http://purl.org/dc/terms/description | | ||
| + | | Geographic area (spatial range) | odm_spatial_range | gn:countryCode | http://www.geonames.org/ontology#countryCode | | ||
| + | | Organization | owner_org | foaf:Organization | http://xmlns.com/foaf/0.1/organization| | ||
| + | | License | license_id | dcterms:licence | http://purl.org/dc/terms/licence | | ||
| + | | Copyright | odm_copyright | cro:Copyright | http://rhizomik.net/ontologies/copyrightonto.owl#copyright | | ||
| + | | Adoption date/Promulgation date/Signing | odm_promulgation_date | bibframe:legalDate | http://id.loc.gov/ontologies/bibframe.html#p_legalDate | | ||
| + | | Short notes of change | odm_laws_previous_changes_notes | skos:changeNote | https://www.w3.org/2009/08/skos-reference/skos.html#changeNote | | ||
| + | | Contact | odm_contact | ebucore:Contact | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact | | ||
| + | | Notes | odm_laws_notes | skos:note | https://www.w3.org/2009/08/skos-reference/skos.html#note | | ||
| + | | Legacy reference document | odm_reference_document | pproc:documentReference | http://contsem.unizar.es/def/sector-publico/pproc.html#documentReference | | ||
| + | | Province(s) | odm_province | | | | ||
| + | | Date modified | odm_date_modified | | | | ||
| + | | Date updated | odm_date_updated | | | | ||
| + | | Keywords | odm_keywords | | | | ||
| + | | Issuing agency/parties | odm_laws_issuing_agency_parties | | | | ||
| + | | Implementing agencies | odm_laws_implementing_agencies | | | | ||
| + | | | odm_laws_primary_policy_reference_point | | | | ||
| + | | | odm_access_and_use_constraints | | | | ||
| + | | Status | odm_laws_status | | | | ||
| + | | Version date (of draft) | odm_laws_version_date | | | | ||
| + | | Effective/Enforced Date | odm_effective_date | | | | ||
| + | | Previous legal document | odm_laws_previous_legal_document | | | | ||
| + | | | odm_laws_previous_changes_notes | | | | ||
| + | | Parent document | odm_laws_parent_document | | | | ||
| + | | Child Document | odm_laws_child_document | | | | ||
| + | | Other reference or supporting documents | odm_laws_other_references | | | | ||
| + | | Publication reference | odm_laws_official_publication_reference | | | | ||
| + | | Links to source | odm_laws_source | | | | ||
| + | |||
| + | ==== Agreements records ==== | ||
| + | |||
| + | ^ Metadata field name ^ ODM Ckan internal id ^ Standard term ^ Standard term uri ^ | ||
| + | | Title | title_translated | dcterms:title | http://purl.org/dc/terms/title | | ||
| + | | Description | notes_translated | dcterms:description | http://purl.org/dc/terms/description | | ||
| + | | Topics | taxonomy | foaf:topic | http://xmlns.com/foaf/0.1/topic | | ||
| + | | License | odm_license | dcterms:licence | http://purl.org/dc/terms/licence | | ||
| + | | Copyright | odm_copyright | cro:Copyright | http://rhizomik.net/ontologies/copyrightonto.owl#copyright | | ||
| + | | Access and use constraints | odm_access_and_use_constraints | md:useconstraints | http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata#useConstraints | | ||
| + | | Organization | owner_org | foaf:Organization | http://xmlns.com/foaf/0.1/organization| | ||
| + | | Version | version | doap:version | http://usefulinc.com/ns/doap#version | | ||
| + | | Contact | odm_contact | ebucore:Contact | https://www.ebu.ch/metadata/ontologies/ebucore/index.html#Contact | | ||
| + | | Language | odm_language | dcterms:language | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#language | | ||
| + | | Date created | odm_date_created | dcterms:created | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#created | | ||
| + | | Date uploaded | odm_date_uploaded | schema:uploadDate | http://schema.org/uploadDate | | ||
| + | | Date modified | odm_date_modified | dcterms:modified | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#modified | | ||
| + | | Geographical area (Spatial range) | odm_spatial_range | gn:countryCode | http://www.geonames.org/ontology#countryCode | | ||
| + | | Accuracy | odm_accuracy | dqm:accuracy | http://semwebquality.org/dqm-vocabulary/v1/dqm#Accuracy | | ||
| + | | Logical consistency | odm_logical_consistency | dq:LogicalConsistency | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#LogicalConsistency | | ||
| + | | Completeness | odm_completeness | dq:Completeness | http://def.seegrid.csiro.au/isotc211/iso19115/2003/dataquality#Completeness | | ||
| + | | Source(s) | odm_source | dcterms:source | http://dublincore.org/documents/2012/06/14/dcmi-terms/?v=terms#source | | ||
| + | | Attributes | odm_attributes | omn:Attribute | http://open-multinet.info/ontology/omn#isAttributeOf | | ||
| + | | Document type | odm_agreement_document_type | | | | ||
| + | | Participating share | odm_agreement_participating_share | | | | ||
| + | | Contracting Parties | odm_agreement_contracting_parties | | | | ||
| + | | Government Entity (multiple) | odm_agreement_government_entity | | | | ||
| + | | Concession/License name | odm_agreement_concession_name | | | | ||
| + | | Disclosure Mode | odm_agreement_disclosure_mode | | | | ||
| + | | Document reference no. | odm_agreement_document_reference_number | | | | ||
| + | | LandMatrix deal No. | odm_agreement_landmatrix_no | | | | ||
| + | | Copyright | odm_copyright | | | | ||
| + | | | odm_access_and_use_constraints | | | | ||
| + | | Organization | owner_org | | | | ||
| + | | Version / Edition | version | | | | ||
| + | | Language of document | odm_language | | | | ||
| + | | Date created | odm_date_created | | | | ||
| + | | Date uploaded | odm_date_uploaded | | | | ||
| + | | Date modified | odm_date_modified | | | | ||
| + | | Province(s) | odm_province | | | | ||
| + | | | odm_metadata_reference_information | | | | ||
| + | | Amendment to agreement | odm_agreement_amendment_to_contract | | | | ||
| + | | Signature date | odm_agreement_signature_date | | | | ||
| + | | | odm_agreement_short_notes_of_change | | | | ||
| + | | Links to source | odm_agreement_source | | | | ||
| + | | | odm_contact | | | | ||
| + | | | odm_agreement_notes | | | | ||
| + | | Granted area | odm_agreement_granted_area | | | | ||
| + | | Contract term | odm_agreement_contract_term | | | | ||
| + | | Contract term unit | odm_agreement_contract_term_unit | | | | ||
| + | | Payment for concession fees | odm_agreement_payment_for_concession_fees | | | | ||
| + | | Payment for concession fees unit | odm_agreement_payment_for_concession_fees_unit | | | | ||
| + | | Guaranty deposit for contract implementation ($) | odm_agreement_guaranty_deposit_ for_contract_implementation | | | | ||
| + | | Land use planning | odm_agreement_land_use_planning | | | | ||
| + | | Land use planning unit | odm_agreement_land_use_planning_unit | | | | ||
| + | | | odm_agreement_parties_obligations | | | | ||
| + | | | odm_agreement_job_creation_summary | | | | ||
| + | | Number of created jobs | odm_agreement_job_creation_number | | | | ||
| + | | | odm_agreement_training_summary | | | | ||
| + | | Fund allocation for training | odm_agreement_training_number | | | | ||
| + | | | odm_agreement_environmental_protection | | | | ||
| + | | | odm_agreement_sociocultural_protection | | | | ||
| + | | | odm_agreement_fiscal_duties_summary | | | | ||
| + | | Total amount of fiscal duties | odm_agreement_fiscal_duties_number | | | | ||
| + | | Environmental impact assessment (EIA) | odm_agreement_eia | | | | ||
| + | | Total amount of social fund | odm_agreement_social_fund | | | | ||
| + | | | odm_agreement_environmental_fund_summary | | | | ||
| + | | Total amount of environmental funds | odm_agreement_environmental_funds | | | | ||
| + | | | odm_agreement_suspension_revocation_termination | | | | ||
| + | | Related documents | odm_agreement_related_documents | | | | ||
| + | | Related Project | odm_agreement_suspension_related_project | | | | ||
| + | | Legacy reference document | odm_reference_document | | | | ||
| + | | Keywords | odm_keywords | | | | ||
| + | | Open Contracting Identifier | open_contracting_id | | | | ||
| + | |||
| + | ===== Visualizing ODM's datasets ===== | ||
| + | |||
| + | TBD | ||
| + | |||
| + | ===== To improve ===== | ||
| + | |||
| + | ==== Split multilingual versions using lang="CODE" ==== | ||
| + | |||
| + | Currently, multilingual values are not exposed following RDF standards but as json object. | ||
| + | |||
| + | ==== Use URIs for standard values such as odm_spatial_range ( gn:countryCode ) or taxonomy ( foaf:topics ) ==== | ||
| + | |||
| + | Instead of using just literals for the object of the different predicates, as is the case now, standard values (such as a country code) should be exposed via an URI. | ||
| + | |||
| + | ==== Do not generate Empty/None triples ==== | ||
| + | |||
| + | In case a triple has the value **None**, it should not be generated. | ||
| + | ==== Implement mapping between ODM taxonomy terms and terms in standard vocabularies (LandVoc, etc...) ==== | ||
| + | In order to expose topics using standard terms | ||
public/using_the_rdf_endpoints_to_pull_data_from_ckan.1462371999.txt.gz · Last modified: 2020/06/23 15:03 (external edit)
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
