Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Round of introductions

Introduce yourself

• Name
• Background
• Expectations of the course
• Open Data?

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Course Outline

Course Overview

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Objectives

1. To gain an understanding of Open Data, its essential aspects, and the principles of opening data;
2. to learn how to find, analyse, and reuse open datasets;
3. to learn the processes involved in preparing and publishing open datasets.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Methods

• Ex cathedra presentations
• Discussions
• Theoretical and practical exercises
• Exploration and comparison of online services
• Individual assignment

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Assessment

Individual assignment

• Select one or more datasets from one or several platforms discussed during the course (if several datasets are selected, there must be a common thread)
• Analyse, describe and identify the potential uses of the dataset(s)
• Between 900 and 1,100 words (without references)
• Short paper to be uploaded onto Cyberlearn by Friday 22 March, 7pm

This assignment is weighted at 20% of the 7C2-CT module.

Further details can be discussed at any time during the course and during the workshop.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Assessment

Criteria

Characteristics of Open Data

• Definitions
• History
• The Impact of Open Data on Disciplines
• Typology
• Purposes
• Requirements
• Licences
• Technical means

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Definitions

Open(ness)

Open means anyone can freely access, use, modify, and share for any purpose (subject, at most, to requirements that preserve provenance and openness)

The Open Definition according to the Open Knowledge Network: https://opendefinition.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Definitions

Open(ness)

• No limitations on access of any kind
• No cost, no authentication, no national or institutional privileges

→ Knowledge funded by (i.e. produced under a mandate from) the public must benefit the public without any limitations

[Scholger 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Definitions

Data

[Chen & Floridi 2013]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Definitions

Data

Data at its most basic level as the absence of uniformity, whether in the real world or in some symbolic system. Only once such data have some recognisable structure and are given some meaning can they be considered information.

[Floridi 2010]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Definitions

Metadata

Data whose purpose is to describe and give information about other data.

[Oxford English Dictionary 2023b]

(Meta)data: semantic transposition

(...) there is no fixed boundary between “data” and “metadata”, and that information viewed as data in one discipline may be metadata in another.

[Alter et al. 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Definitions

Open Data

Open data and content can be freely used, modified, and shared by anyone for any purpose.

The Open Definition according to the Open Knowledge Network: https://opendefinition.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

History

Robert K. Merton (1910-2003)

American sociologist, considered a founding father of modern sociology, said in 1942:

Each researcher must contribute to the 'common pot' and give up intellectual property rights to allow knowledge to move forward.

https://en.wikipedia.org/wiki/Timeline_of_the_open-access_movement

While the term “open data” isn’t even 20 years old, the author puts the concept in a historical context; the idea that scientific research should be free to all was popularized by Robert King Merton in the early 1940s. Research (which produces data) should be shared freely for the common good.

https://data.gov/blog/open-data-history/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

History

Timeline

• 1942: The concept starts with Robert K. Merton
• 1995: The term 'Open Data' first appeared, related to the sharing of geophysical and environmental data.
• November 2005: Open Knowledge Foundation creates the Open Definition
• December 2007: The concept of open public data was discussed and defined in Sebastopol, CA, USA at a meeting of Internet activists. They identified 8 principles.
• February 2009: Tim Berners-Lee presents The next web at TED2009. He famously asked for "raw data now".

https://devopedia.org/open-data
https://www.opendatasoft.com/en/what-is-open-data-practical-guide/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

The Impact of Open Data on Disciplines

Catalysing interdisciplinary progress

• A foundation for collaboration and innovation: Open data drives interdisciplinary research by providing universally accessible datasets, fostering collaboration across diverse fields such as information science, digital humanities and data science.

• Enabling data-driven research: Facilitates advanced analysis and research in disciplines that rely heavily on data, increasing the accuracy and depth of insights and discoveries.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

The Impact of Open Data on Disciplines

Open Data in Information Science, Digital Humanities, and Data Science

• Information Science: Improves data management and accessibility, enabling more efficient data retrieval, archiving and dissemination practices.

• Digital Humanities: Enabling new digital approaches to humanities research, providing new insights into historical, cultural and linguistic studies through data analysis.

• Data Science: Leverages open data for predictive modelling, machine learning and big data analytics, enabling comprehensive analysis and informed decision-making in diverse fields such as health, finance and social sciences.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Typology

Main Sources

• Research: Open Research Data (ORD) / Open Scientific Data
• Government: Open Government Data (OGD)
• Non-profit organisations
• Private organisations

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Typology

Disciplines

• Cultural Heritage
• Healthcare
• Education
• Transportation
• Meteorology
• Geospatial Information
• Economic and Finance
• Legal and Criminal Justice
• Etc.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Research Data (ORD)

Research data are the evidence that underpins the answer to the research question, and can be used to validate findings regardless of its form (e.g. print, digital, or physical). These might be quantitative information or qualitative statements collected by researchers in the course of their work by experimentation, observation, modelling, interview or other methods, or information derived from existing evidence. Data may be raw or primary (e.g. direct from measurement or collection) or derived from primary data for subsequent analysis or interpretation (e.g. cleaned up or as an extract from a larger data set), or derived from existing sources where the rights may be held by others.

[Concordat Working Group 2016]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

ORD

For funding agencies (and institutions)

3. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

For organizations that supply metrics

11. Be open and transparent by providing data and methods used to calculate all metrics.
12. Provide the data under a licence that allows unrestricted reuse, and provide computational access to data, where possible.

San Francisco Declaration on Research Assessment (DORA): https://sfdora.org/read/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

ORD in Switzerland

SNSF Policy

The Swiss National Science Foundation (SNSF) expects all its funded researchers:

• to store the research data they have worked on and produced during the course of their research work,
• to share these data with other researchers, unless they are bound by legal, ethical, copyright, confidentiality or other clauses, and
• to deposit their data and metadata onto existing public repositories in formats that anyone can find, access and reuse without restriction.

https://www.snf.ch/en/dMILj9t4LNk8NwyR/topic/open-research-data

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

ORD in Switzerland

Vision

By facilitating access to and reuse of research data, ORD promotes better, more effective, and more impactful research for the benefit of society as a whole. Through the principles of open access and reusability of research data, ORD practices support transparent and reproducible research findings. Moreover, ORD fosters collaboration by promoting exchange among researchers across disciplines, legal systems and national borders, thus enabling creativity and innovation to thrive.

[Open Science Delegation 2021a]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

ORD in Switzerland

Action Areas

• Support researchers and research communities in imagining and adopting ORD practices
• Development, promotion, and maintenance of financially sustainable basic infrastructures and services for all researchers
• Equipping researchers for ORD skills development and exchange of best practices
• Building up systemic und supportive conditions for institutions and research communities

[Open Science Delegation 2021b]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Research Data Management (RDM)

Definition

RDM refers to the organisation, storage and preservation of data created during a research project.

Purpose

RDM ensures that research data are well-organised, maintained and accessible for current and future research, thereby improving their reliability, validity and reproducibility.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Data Management Plan (DMP)

Definition

A DMP is a formal document that outlines how data will be handled during and after a research project, covering aspects from collection to sharing and preservation.

Purpose

It serves as a guide for managing data efficiently and meets funding agency requirements for data stewardship. It is now mandatory for most funding agencies to require a DMP as part of the grant application process to secure funding. University libraries often have services and resources to assist researchers in creating these documents, providing expert guidance on best practices in data management.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

RDM and DMP

Interconnected Roles

RDM encompasses the day-to-day management of research data, while a DMP provides a structured plan for how to manage, share, and preserve data throughout the research project.

Planning and Execution

A DMP is essentially a blueprint for RDM. It outlines the policies and standards to be applied to the data, ensuring that data management practices are thought through from the outset of the project.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Government Data (OGD)

The work of government involves collecting huge amounts of data, much of which is not confidential (economic data, demographic data, spending data, crime data, transport data, etc). The value of much of this data can be greatly enhanced by releasing it as open data, freeing it for re-use by business, research, civil society, data journalists, etc.

Open Data Handbook [Open Knowledge 2016]:
https://opendatahandbook.org/glossary/en/terms/government-data/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

OGD in Switzerland

• Masterplan Open Government Data 2024-2027 [OFS 2023]: https://www.bfs.admin.ch/bfs/fr/home/services/ogd/masterplan.html

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

OGD in Switzerland

LMETA. Art. 10, al. 4

Les données sont mises en ligne gratuitement, en temps utile, sous une forme lisible par machine et dans un format ouvert. Elles peuvent être librement réutilisées, sous réserve d’obligations légales spéciales de mentionner la source des données.

[Loi fédérale sur l’utilisation des moyens électroniques pour l’exécution des tâches des autorités (LMETA) 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

OGD in Switzerland

Related efforts

• National data management (NaDB): https://www.bfs.admin.ch/bfs/en/home/nadb/nadb.html
• i14y Interoperability platform: https://www.i14y.admin.ch/
• Digital Switzerland Strategy: https://digital.swiss/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Purposes

• Transparency and democratic control
• Participation
• Self-empowerment
• Improved or new private products and services
• Innovation
• Improved efficiency and effectiveness of government services
• Impact measurement of policies
• New knowledge from combined data sources and patterns in large data volumes

Open Data Handbook [Open Knowledge 2016]
https://opendatahandbook.org/guide/en/why-open-data/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Purposes

Open Government Data Principles

1. Complete
2. Primary
3. Timely
4. Accessible
5. Machine processable
6. Non-discriminatory
7. Non-proprietary
8. License-free

https://public.resource.org/8_principles.html

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Requirements

Two main components

1. Legally open: available under an open (data) licence that permits anyone freely to access, reuse and redistribute
2. Technically open: data is available for no more than the cost of reproduction and in machine-readable and bulk form.

Open Data Handbook [Open Knowledge 2016]
https://opendatahandbook.org/glossary/en/terms/open-data/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Copyright and Copyleft

Copyright

• Grants creators exclusive rights to control use, reproduction, and distribution.
• Designed to protect creators' economic interests; allows monetization of work.

Copyleft

• Allows use, modification, and distribution with the condition of keeping works and derivatives open.
• Promotes freedom, sharing of knowledge, and collaborative improvement.

For further information about licences → Competence Center in Digital Law: https://www.ccdigitallaw.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Anglo-Saxon Copyright vs. European Author's Rights

Anglo-Saxon Copyright

• Focuses on economic rights; treats copyright as transferable property.
• Duration based on set years post-creation or author's life plus years.

European Author's Rights (droit d'auteur / Urheberrecht)

• Emphasises moral rights alongside economic rights.
• Grants inalienable moral rights to creators; duration includes lifetime plus post-death period (commonly 70 years).

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Creative Commons (CC)

• CC BY 4.0: Attribution
• CC BY-SA 4.0: Attribution, Share Alike
• CC BY-ND 4.0: Attribution, No Deritave Works
• CC BY-NC 4.0: Attribution, No Commercial Use
• CC BY-NC-SA 4.0: Attribution, No Commercial Use, Share Alike
• CC BY-NC-ND 4.0: Attribution, No Commercial Use, No Derivative Works
• Public Domain Dedication (CC0): No Rights Reserved
• Public Domain Mark: No Known Copyright

https://creativecommons.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Creative commons license spectrum [MJL 2020]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Rights Statements

• 12 different rights statements that can be used by cultural heritage institutions
• Three categories
  • In Copyright: statements for works that are in copyright
  • No Copyright: statements for works that are not in copyright
  • Other: stateements for works where the copyright status is unclear

https://rightsstatements.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Open Data Commons Open Database License (ODbL)

• Copyleft licence
• Attribution and Share-Alike for Data/Databases

https://opendatacommons.org/licenses/odbl/

ODbL is somewhat equivalent to CC BY-SA [Santos 2020].

Public Domain Dedication and License (PDDL)

• The PDDL places the data(base) in the public domain (waiving all rights)

https://opendatacommons.org/licenses/pddl/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Licences for software

• GNU General Public License (GPL)
• GNU Affero General Public License (AGPL)
• Mozilla Public License (MPL)
• MIT License
• Apache License

And others... → see for instance https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/licensing-a-repository

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Responsable AI Licenses (RAIL)

• Responsible AI Pubs Licenses
  • AIPubs Open RAIL-S
  • AIPubs Open RAIL-M
  • AIPubs Research-Use RAIL-S
  • AIPubs Research-Use RAIL-M
• Responsible AI End-User License (RAIL-A License)
• Responsible AI Source Code License (Open RAIL-S License)
• BigScience Open RAIL-M License (Open RAIL-M License)

https://www.licenses.ai/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Responsable AI Licenses (RAIL)

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

I Hate AI License (IHAIL)

• Based on CC BY 4.0
• It prohibits the use of the material with Artifical Intelligence (AI) technologies while allowing sharing, adaptation, and commercial use under certain terms.

https://ihateailicense.eu/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Licences

Recommendations for Open (Research) Data

1. CC0 (to the fullest extent allowed by law, as a complete waiver is not feasible under Swiss regulations)
2. CC BY 4.0
3. CC BY-SA 4.0

[Santos 2020]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Technical means

Important factors in providing structured data for machines

• Data(set) formats
  • Text-based formats
  • Binary-encoded formats
• Metadata standards / schemas (to describe the dataset)
• Documentation
• Protocols

And of course the underlying infrastructure...

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Infrastructure

Definitions

A collective term for the subordinate parts of an undertaking; substructure, foundation.

[Oxford English Dictionary 2023a]

People commonly envision infrastructure as a system of substrates – railroad, lines, pipes and plumbing, electrical power plants, and wires. It is by definition invisible, part of the background for other kinds of work. It is ready-to-hand. This image holds up well enough for many purposes – turn on the faucet for a drink of water and you use a vast infrastructure of plumbing and water regulation without usually thinking much about it.

[Star 1999]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Infrastructure

Dimensions

1. Embeddedness
2. Transparency
3. Reach or scope
4. Learned as part of membership
5. Links with conventions of practice

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Infrastructure

Dimensions

6. Embodiment of standards
7. Built on an installed base
8. Becomes visible upon breakdown
9. Is fixed in modular increments
10. Not all at once globally

[Star 1999]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

Plain Text (TXT)

WSe2			WS2			MoS2	

dk	Intensity	dk	Intensity	dk	Intensity	

855.87628	63	848.96433	-39	855.87628	372
855.25787	72.25	848.34546	2	855.25787	424
854.63942	64.25	847.72654	-39	854.63942	460
854.02093	58	847.10759	-37	854.02093	362
853.40239	66	846.4886	-28	853.40239	440

Sohier, T., Ponomarev, E., Gibertini, M., Berger, H., Marzari, N., Ubrig, N., & Morpurgo, A. F. (2019). Enhanced Electron-Phonon Interaction in Multivalley Materials [Data set]. Université de Genève, Yareta. https://doi.org/10.26037/yareta:jlzyhiwj6vfjrnbza4bkvobjai

File (extract): f1b.txt

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

Markdown (MD)

# e-periodica OAI-PMH - Ethnology and Folklore
This script was done to download the metadata of [e-periodica](https://www.e-periodica.ch/) through 
their OAI-PMH endpoint (`https://www.e-periodica.ch/oai`) that could be interesting to the PIA research 
project as we want to link our image-based collections to the e-periodica articles. 

There are more than 16k metadata articles which have 
the 390 `setSpec` (Ethnology, folklore) on e-periodica. Probably, the more relevant articles 
come from the `Korrespondenzblättern der SGV` (more than 2k articles), divided into these three sources: 

- https://www.e-periodica.ch/digbib/volumes?UID=sgv-001 
- https://www.e-periodica.ch/digbib/volumes?UID=sgv-002
- https://www.e-periodica.ch/digbib/volumes?UID=sgv-003 

## Records in CSV
- [All records](data/records.csv)
- [Extract (SGV)](data/sgv.csv)

Raemy, J. A. (2023). e-periodica OAI-PMH - Ethnology and Folklore (Version 1.0.0) [Computer software]. https://doi.org/10.5281/zenodo.7777797
File: README.MD

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

Comma-separated values (CSV)

"TRANSPORT_TYPE";"TRANSPORT_MEAN";"TRAVEL_REASON";"SOCIO_DEMO_VARIABLE_TYPE";"SOCIO_DEMO_VARIABLE";"PERIOD_REF";"UNIT_MEAS";"VALUE";"OBS_CONFIDENCE";"OBS_STATUS"
"TOT";"TOT";"ALL_REAS";"GEO";"CH";2015;"KM";36.8318;0.4602;"A"
"TOT";"TOT";"WORK";"GEO";"CH";2015;"KM";8.8512;0.1995;"A"
"TOT";"TOT";"SCHOOL";"GEO";"CH";2015;"KM";1.9104;0.1026;"A"
"TOT";"TOT";"SHOP";"GEO";"CH";2015;"KM";4.7651;0.1346;"A"
"TOT";"TOT";"LEISU";"GEO";"CH";2015;"KM";16.2548;0.3385;"A"
"TOT";"TOT";"SERV_ACC";"GEO";"CH";2015;"KM";1.8462;0.0982;"A"
"TOT";"TOT";"BUSIN";"GEO";"CH";2015;"KM";2.5514;0.1587;"A"
"TOT";"TOT";"OTH_REAS";"GEO";"CH";2015;"KM";0.6527;0.0759;"A"
"SOFT_MOB";"TOT";"ALL_REAS";"GEO";"CH";2015;"KM";2.8031;0.0440;"A"

Federal Statistical Office. Comportement de la population en matière de transports, tableaux de synthèse. https://opendata.swiss/fr/perma/18084205@bundesamt-fur-statistik-bfs

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

Extensible Markup Language (XML)

<?xml version="1.0" encoding="utf-8"?>
<rdf:RDF
  xmlns:foaf="http://xmlns.com/foaf/0.1/"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:dcat="http://www.w3.org/ns/dcat#"
  xmlns:dct="http://purl.org/dc/terms/"
  xmlns:vcard="http://www.w3.org/2006/vcard/ns#">
  <dcat:Dataset rdf:about="https://ckan.opendata.swiss/perma/121911@bundesamt-fur-statistik-bfs">
    <dcat:keyword xml:lang="it">lavoro-e-reddito</dcat:keyword>
    <dct:language>fr</dct:language>
    <dcat:distribution>
      <dcat:Distribution rdf:about="https://ckan.opendata.swiss/dataset/90beaddf-4f48-4211-9e34-ff68d4308f98/resource/f9eb3fb4-0a11-4a40-a995-0aa13f377011">
        <dct:rights rdf:resource="http://dcat-ap.ch/vocabulary/licenses/terms_by_ask"/>
        <dcat:downloadURL rdf:resource="https://dam-api.bfs.admin.ch/hub/api/dam/assets/121910/master"/>
        <dcat:accessURL rdf:resource="https://dam-api.bfs.admin.ch/hub/api/dam/assets/121910/master"/>
        <dct:format rdf:resource="http://publications.europa.eu/resource/authority/file-type/XLS"/>
        <dct:title xml:lang="de">Kanton Genf: Erwerbsleben und Ausbildung</dct:title>
        <dct:identifier>121910-master@bundesamt-fur-statistik-bfs</dct:identifier>
        <dct:language>de</dct:language>
        <dcat:mediaType rdf:resource="http://www.iana.org/assignments/application/vnd.ms-excel"/>
        <dct:description xml:lang="de">Dieser Dataset präsentiert die Zahlen zu Erwerbsleben und Ausbildung (Eidgenössische Volkszählung 2000)</dct:description>
        <dct:license rdf:resource="http://dcat-ap.ch/vocabulary/licenses/terms_by_ask"/>
      </dcat:Distribution>

Federal Statistical Office. Canton de Genève: Vie active et formation
https://opendata.swiss/fr/perma/121911@bundesamt-fur-statistik-bfs

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

Terse RDF Triple Language (Turtle)

@prefix ns1: <https://data.tg.ch/ld/ontologies/div-energie-6/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<https://data.tg.ch/ld/resources/div-energie-6/div-energie-6-record/1740b190a1beca6edcd04e0b143c380b885ee1de/> 
    a ns1:div-energie-6-record ;
    ns1:andere "2690"^^xsd:int ;
    ns1:einwohner "266510"^^xsd:int ;
    ns1:energiebezugsflaeche "25055573"^^xsd:int ;
    ns1:erdgas "308728"^^xsd:int ;
    ns1:erdoelbrennstoffe "307734"^^xsd:int ;
    ns1:jahr "2015-01-01"^^xsd:date ;
    ns1:kehricht "78654"^^xsd:int ;
    ns1:total "1291573"^^xsd:int ;
    ns1:treibstoffe "593764"^^xsd:int .

Kanton Thurgau. CO2-Gesamtemissionen nach Energieträgern (Ebene Kanton)
https://opendata.swiss/de/perma/div-energie-6@kanton-thurgau
https://data.tg.ch/ld/resources/div-energie-6/div-energie-6-record/1740b190a1beca6edcd04e0b143c380b885ee1de/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

JavaScript Object Notation (JSON)

[{
        "author": "Hemingway, Ernest",
        "available_at": [
            {
                "isil": "AG0066"
            }
        ],
        "bid": "AGR0005487",
        "cited": [
            "VEA1112819"
        ],
        "citing": [],
        "dewey_classifications": null,

EPFL. Citations extracted from monographs about the history of Venice. https://opendata.swiss/de/perma/EPFL-LinkedBooksMonographs@openglam

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Text-based Formats

JavaScript Object Notation for Linked Data (JSON-LD)

{
  "id": "https://lux.collections.yale.edu/data/group/8b757ad2-f853-425e-a30d-19686aa779ee",
  "type": "Group",
  "_label": "American Academy of Arts and Sciences",
  "@context": "https://linked.art/ns/v1/linked-art.json",
  "formed_by": {
    "type": "Formation",
    "timespan": {
      "type": "TimeSpan",
      "identified_by": [
        {
          "type": "Name",
          "content": "1780-05-04",
          "classified_as": [
            {
              "id": "https://lux.collections.yale.edu/data/concept/5088ec29-065b-4c66-b49e-e61d3c8f3717",
              "type": "Type",
              "_label": "Display Title" }

LUX. American Academy of Arts and Sciences
https://lux.collections.yale.edu/data/group/8b757ad2-f853-425e-a30d-19686aa779ee

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Binary-encoded Formats

Binary files are used to store non-text data, such as images, audio, or executable programs. These files do not contain human-readable text and are encoded in binary format.

• Image Formats: BMP, GIF, JPEG, JPEG2000, PNG, TIFF
• Vector Graphics Formats: EPS, PSD, SVG
• 3D Formats: 3MF, GLB, GLTF, OBJ, STL
• Audio Formats: AAC, FLAC, MP3, OGG, WAV
• Video Formats: AVI, FFV1/MKV, MOV, MP4, WEBM
• Documents: DOCX, ODT, PDF, PDF/A
• Scientific Data Formats: CDF, DICOM, FITS
• Archive File Formats: 7-ZIP, GZIP, TAR, ZIP

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

1706-11-30_Verzaglia_Giuseppe-Bernoulli_Johann_I
https://iiif.dasch.swiss/0801/4VjgCwiTn8p-CTaooIqSZBO.jpx/full/max/0/default.jpg

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Metadata standards / schemas

• Metadata standards are sets of rules and guidelines that dictate how metadata should be formatted and used. They ensure consistency and interoperability across different systems and platforms.
  • CIDOC Conceptual Reference Model (CIDOC-CRM), Dublin Core, Machine-Readable Cataloging (MARC), Preservation Metadata: Implementation Strategies (PREMIS)
• Metadata schemas are specific implementations of metadata standards. They outline the structure, elements, and attributes of metadata for a specific purpose.
  • Encoded Archival Description (EAD), Lightweight Information Describing Object (LIDO), Metadata Object Description Schema (MODS)

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Data Catalog Vocabulary (Application Profiles)

Data Catalog Vocabulary (DCAT)

• Resource Description Framework (RDF) vocabulary to facilitate interoperability between data catalogues published on the Web.
• Current version: DCAT 3

Data Catalog Vocabulary Application Profile (DCAT-AP)

Specifications based on DCAT for describing public sector datasets

• DCAT Application Profile for data portals in Europe: DCAT-AP 3.0
• DCAT Application Profile for the United States of America: DCAT-US - Version 3
• DCAT Application Profile for Data Portals in Switzerland (DCAT-AP CH): eCH-0200

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

DCAT

Seven classes ("entities")

1. dcat:Catalog represents a catalogue, which is a dataset in which each individual item is a metadata record describing some resource
2. dcat:Resource represents a dataset, a data service or any other resource that may be described by a metadata record in a catalogue.
3. dcat:Dataset represents a collection of data, published or curated by a single agent or identifiable community.
4. dcat:Distribution represents an accessible form of a dataset such as a downloadable file.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

5. dcat:DataService represents a collection of operations accessible through an interface (API) that provide access to one or more datasets or data processing functions.
6. dcat:DatasetSeries is a dataset that represents a collection of datasets that are published separately, but share some characteristics that group them.
7. dcat:CatalogRecord represents a metadata record in the catalogue, primarily concerning the registration information, such as who added the record and when.

https://www.w3.org/TR/vocab-dcat-3/#dcat-scope

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

DCAT

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

DCAT

ex:catalog
  a dcat:Catalog ;
  dcterms:title "Imaginary Catalog"@en ;
  dcterms:title "Catálogo imaginario"@es ;
  rdfs:label "Imaginary Catalog"@en ;
  rdfs:label "Catálogo imaginario"@es ;
  foaf:homepage <http://dcat.example.org/catalog> ;
  dcterms:publisher ex:transparency-office ;
  dcterms:language <http://id.loc.gov/vocabulary/iso639-1/en>  ;
  dcat:dataset ex:dataset-001 , ex:dataset-002 , ex:dataset-003 ;
  .

https://www.w3.org/TR/vocab-dcat-3/#basic-example

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

DCAT-AP CH

@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

# ---------- class Catalog --------------------------------------------------
<https://swisstopo/opendata/catalog>
  a dcat:Catalog ;

  # mandatory properties
  dct:description "Datenkatalog der Stadt Zurich"@de ;
  dct:publisher <https://publishers/swisstopo> ;
  dct:title "Open Data City of Zurich"@en ,
            "Offene Daten der Stadt Zurich"@de .

https://www.dcat-ap.ch/releases/2.0/dcat-ap-ch.html#Class:Catalog

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Documentation

Comprehensive and understandable information about the data, including its source, structure, context, and how to use it effectively.

Types of Documentation

• Data field descriptions/data models
• User guides
• Metadata
• Developer documentation
• Source code documentation

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Protocols

• Application Programming Interface (API): mechanism that enable two software components to communicate with each other
  • Representational State Transfer (REST): a style of API that uses HTTP requests for communication. REST is stateless, i.e. each request from a client to the server is treated as new. There is no stored memory of previous interactions. This means the server does not store any state about the client session on the server side.
  • Simple Object Access Protocol (SOAP): a protocol used for exchanging structured information in web services, offering high security and transactional reliability. SOAP can support both stateless and stateful operations.
• File Transfer Protocol (FTP): used for the transfer of data files, particularly large datasets, from one host to another.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Protocols

• Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH): a protocol for harvesting metadata descriptions of records in an archive, particularly used in digital libraries.
• Really Simple Syndication or Rich Site Summary (RSS) / Atom Feeds: used for regularly updating or publishing data that changes frequently. Feeds enable publishers to syndicate data automatically.
• SPARQL Protocol and RDF Query Language (SPARQL): used for querying and manipulating RDF

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – Imagine an ORD/OGD Platform

Conceptualising an Open Data Platform

In pairs or small groups, you will conceptualise a new platform for open data. Imagine what kind of datasets you would accept and showcase one example. Please consider the following:

• Accepted datasets
  • Criteria and subjects
• Accepted licence(s)
  • Open licence, possible restrictions
• Accepted (meta)data formats
• Metadata standard to describe the datasets
• Protocols/Services available on the platform

Associated Movements and Principles

• Open Access
• Open Science / Open Scholarship
• Open Source / Free Software / F(L)OSS
• FAIR Data Principles
• CARE Principles for Indigenous Data Governance
• Collections as Data
• Linked (Open) (Usable) Data

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Access

Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities

1. Authors and right holders must grant all users a free, irrevocable, worldwide, right of access to, and a license to copy, use, distribute, transmit and display the work publicly and to make and distribute derivative works, in any digital medium for any responsible purpose, subject to proper attribution of authorship as well as the right to make small numbers of printed copies for their personal use.
2. A complete version of the work and all supplemental materials, including a copy of the permission as stated above, in an appropriate standard electronic format is deposited in at least one online repository using suitable technical standards.

[Max Planck Society & European Cultural Heritage Online 2003]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Access

Definition

Open access (OA) is a broad international movement that seeks to grant free and open online access to academic information, such as publications and data. A publication is defined 'open access' when there are no financial, legal or technical barriers to accessing it - that is to say when anyone can read, download, copy, distribute, print, search for and search within the information, or use it in education or in any other way within the legal agreements.

https://www.openaccess.nl/en/what-is-open-access

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Access

Definition

OA is a publishing model for scholarly communication that makes research information available to readers at no cost, as opposed to the traditional subscription model in which readers have access to scholarly information by paying a subscription (usually via libraries).

https://www.openaccess.nl/en/what-is-open-access

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Access

Gold Open Access

Publications are made freely accessible by the publisher immediately upon publication. It often involves Article Processing Charges (APCs) paid by the author, their institution, or a funder.

→ Immediate OA via publisher

Green Open Access (Self-Archiving)

Authors publish their work in any journal and then self-archive an earlier version of the article (pre-print) for free public use in a repository (sometimes after an embargo period).

→ Immediate or delayed OA via self-archiving method/repository

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Access

Hybrid Open Access

Subscription-based journals allow authors to make their individual articles OA upon payment of an APC.

→ Immediate OA via publisher

Diamond/Platinum Open Access

Journals do not charge authors APCs and provide immediate OA to all their articles. It operates without direct cost to the authors; funding often comes from institutions, societies, or donations.

→ Immediate OA via publisher

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Access

Bronze Open Access

Articles made freely accessible on the publisher's website without an explicit OA licence.

Blue Open Access

Through blue OA, authors can archive the post-print or the publisher’s final version.

Black Open Access

It refers to the unauthorised distribution of published content through various channels, such as pirate sites or peer-to-peer networks.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Science

Definition

Open Science is the practice of science in such a way that others can collaborate and contribute, where research data, lab notes and other research processes are freely available, under terms that enable reuse, redistribution and reproduction of the research and its underlying data and methods.

[FOSTER 2019]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Science

[Morrison 2021, citing Persic 2021]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Scholarship

Open Scholarship: Expanding the Reach of Open Science

• Broader Approach
  • Extends beyond traditional scientific disciplines to include arts and humanities.
  • Engages not just the research community but also the wider public, including non-experts, educators, and policymakers.
• Supporting Collaboration and Innovation
  • Facilitates interdisciplinary collaboration across arts, humanities, and other fields.
  • Encourages the use of open educational resources for collaborative teaching and learning.
  • Advances open data practices for the sharing and reuse of cultural heritage resources.

[Tennant et al. 2020]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Source

Definition and Philosophy

Open Source refers to software with source code that can be inspected, modified, and enhanced by anyone. It emphasises collaboration and community-oriented development.

Key Characteristics

It includes free redistribution, access to source code, and allowance for derived works.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Source

Criteria

1. Free redistribution
2. Source code must be included
3. Derived works must be allowed
4. Integrity of the author's source code
5. No discrimation against persons or groups

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Source

6. No discrimation against fields of endeavour
7. Distribution of licence
8. Licence must not be specific to a product
9. Licence must not restrict other software
10. Licence must be technology-neutral

https://opensource.org/osd/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Free Software

• Free Software is centred around the idea of user freedom – the freedom to run, study, change, and distribute the software. "Free" refers to freedom, not price.
• It has four essential freedoms
  • The freedom to run the program as you wish, for any purpose (freedom 0).
  • The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
  • The freedom to redistribute copies so you can help others (freedom 2).
  • The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Free Software

“Free software” means software that respects users' freedom and community. Roughly, it means that the users have the freedom to run, copy, distribute, study, change and improve the software. Thus, “free software” is a matter of liberty, not price. To understand the concept, you should think of “free” as in “free speech,” not as in “free beer.” We sometimes call it “libre software,” borrowing the French or Spanish word for “free” as in freedom, to show we do not mean the software is gratis.

https://www.gnu.org/philosophy/free-sw.en.html

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

F(L)OSS

Free/Libre and Open Source Software

This is software for which the licensee can get the source code, and is allowed to modify this code and to redistribute the software and the modifications. Many terms are used: free, referring to the freedom to use (not to “free of charge”), libre, which is the French translation of Free/freedom, and which is preferred by some writers to avoid the ambiguous reference to free of charge, and open source, which focuses more on the access to the sources than on the freedom to redistribute. In practice, the differences are not great, and more and more scholars are choosing the term FLOSS to name this whole movement.

[Jullien 2009]

"Neutral stance": See https://www.gnu.org/philosophy/floss-and-foss.en.html

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

FAIR

FAIR Data Principles: https://www.go-fair.org/fair-principles/

[Wilkinson et al. 2016]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

FAIR

Findable

F1. (Meta)data are assigned a globally unique and persistent identifier

F2. Data are described with rich metadata (defined by R1)

F3. Metadata clearly and explicitly include the identifier of the data they describe

F4. (Meta)data are registered or indexed in a searchable resource

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

FAIR

Accessible

A1. (Meta)data are retrievable by their identifier using a standardised communications protocol

• A1.1 The protocol is open, free, and universally implementable

• A1.2 The protocol allows for an authentication and authorisation procedure, where necessary

A2. Metadata are accessible, even when the data are no longer available

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

FAIR

Interoperable

I1. (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.

I2. (Meta)data use vocabularies that follow FAIR principles

I3. (Meta)data include qualified references to other (meta)data

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

FAIR

Reusable

R1. (Meta)data are richly described with a plurality of accurate and relevant attributes

• R1.1. (Meta)data are released with a clear and accessible data usage license

• R1.2. (Meta)data are associated with detailed provenance

• R1.3. (Meta)data meet domain-relevant community standards

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

CARE

CARE Principles for Indigenous Data Governance: https://www.gida-global.org/care

[Carroll et al. 2020]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

CARE

Collective Benefit

C1. For inclusive development and innovation

C2. For improved governance and citizen engagement

C3. For equitable outcomes

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

CARE

Authority to Control

A1. Recognizing rights and interests

A2. Data for governance

A3. Governance of data

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

CARE

Responsibility

R1. For positive relationships

R2. For expanding capability and capacity

R3. For Indigenous languages and worldviews

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

CARE

Ethics

E1. For minimizing harm and maximizing benefit

E2. For justice

E3. For future use

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

CARE and FAIR

Operationalising CARE and FAIR

[Carroll et al. 2021]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Summits

• 2017: Santa Barbara Statement [Padilla et al. 2017]
• 2023: Vancouver Statement [Padilla, Scates Kettler, Varner, et al. 2023]

Main outputs

• 10 principles
• 'Part to Whole' Report
• Related checklist and initiatives

https://collectionsasdata.github.io/statement/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

10 Principles

1. Collections as Data development aims to encourage computational use of digitised and born digital collections.
2. Collections as Data stewards are guided by ongoing ethical commitments.
3. Collections as Data stewards aim to lower barriers to use.
4. Collections as Data designed for everyone serve no one.
5. Shared documentation helps others find a path to doing the work.
6. Collections as Data should be made openly accessible by default, except in cases where ethical or legal obligations preclude it.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

7. Collections as Data development values interoperability.
8. Collections as Data stewards work transparently in order to develop trustworthy, long-lived collections.
9. Data as well as the data that describe those data are considered in scope.
10. The development of collections as data is an ongoing process and does not necessarily conclude with a final version.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Collections as Data: Part to Whole

• Boundary Object Concept: Collections-as-data serve as flexible tools adaptable to various needs while maintaining a common identity.
• Ethical Considerations: Emphasis on ethical development and use of collections, especially concerning marginalized communities.
• Community Engagement: Essential for respecting and understanding the context of collections.
• Organisational Structure Support: Effective initiatives require collaboration across various organisational departments.

Boundary Object, cf. Star & Griesemer [1989]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Collections as Data: Part to Whole

• Documentation Importance: Crucial for understanding and maintaining collections in the future.
• Community of Practice: Emphasises the need for skill sharing and collaborative environments.
• Future Challenges and Opportunities
  • Integration of AI and computational tools in collections.
  • Navigating the balance between global collaboration and local cultural sensitivities.
  • Addressing financial and resource limitations for global community growth.
  • Potential and risks of using collections as data for AI training.

[Padilla, Scates Kettler, & Shorish, 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Checklist to publish Collections as Data in GLAM institutions

1. Provide a clear license allowing reuse of the dataset without restrictions
2. Provide a suggestion of how to cite your dataset
3. Include documentation about the dataset
4. Use a public platform to publish the dataset
5. Share examples of use as additional documentation
6. Give structure to the dataset

Galleries, Libraries, Archives, Museums (GLAM)

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Checklist to publish Collections as Data in GLAM institutions

7. Provide machine-readable metadata (about the dataset itself)
8. Include your dataset in collaborative edition platforms
9. Offer an API to access your repository
10. Develop a portal page
11. Add a terms of use

[Candela et al. 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Workflow

https://marketplace.sshopencloud.eu/workflow/I3JvP6

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Collections as Data

Implementation at the Royal Library of Belgium

• Data-level access to collections
• Digital Humanities Research

https://www.kbr.be/en/projects/data-kbr-be/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

An Open Vision of the Web

The [World Wide Web] project merges the techniques of information retrieval and hypertext to make an easy but powerful global information system. The project started with the philosophy that much academic information should be freely available to anyone.

[Berners-Lee 1991]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Data

Linked Data refers to a set of best practices for publishing structured data on the Web.

Linked Data Principles

• Use Uniform Resource Identifiers (URIs) as names for things
• Use HTTP URIs so that people can look up those names.
• When someone looks up a URI, provide useful information, using the standards (e.g. RDF, RDFS, SPARQL, etc.)
• Include links to other URIs so that they can discover more things.

[Berners-Lee 2006]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Open Data (LOD)

5-star deployment scheme for Open Data: https://5stardata.info/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

The Semantic Web or the Web of Data

The Semantic Web is an extension of the World Wide Web, through standards, to make it machine-readable.

Tweaked Semantic Web Layer Cake [Idehen 2017]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Resource Description Framework (RDF)

With RDF, everything goes in threes. Most of the triples' components have Uniform Resource Identifiers (URIs). Syntax: subject, predicate, object

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Open Usable Data (LOUD)

The concept of LOUD extends LOD by emphasising not just the openness and interlinking of data but also its usability.

LOUD

• The term was coined by Robert Sanderson [2018, 2019] who has been involved in the conception and maintenance of web standards, mainly in the cultural heritage field.
• LOUD's goal is to achieve the Semantic Web's intent on a global scale in a usable fashion by leveraging community-driven and JSON-LD-based specifications.
• It has five main design principles to make the data more easily accessible to software developers, who play a key role in interacting with the data and building software and services on top of it, and to some extent to academics.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Open Usable Data (LOUD)

LOUD design principles

• The right Abstraction for the audience
• Few Barriers to entry
• Comprehensible by introspection
• Documentation with working examples
• Few Exceptions, instead many consistent patterns

https://linked.art/loud/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

LOUD Standards

Specifications that follow the LOUD principles

• International Image Interoperability Framework (IIIF)
• W3C Web Annotation Data Model
• Linked Art

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

LOUD-driven Communities

IIIF and Linked Art: social fabrics of sound socio-technical practices

• Synergy of effective social and technical integration with an emphasis on usability
• Unified by shared expertise and leadership
• Collaboration beyond technical boundaries
• Inclusivity and diversity in participation
• Openness and friendliness as core values
• Commitment to transparency
• Organisation of online and face-to-face meetings

[Newbury 2018; Raemy 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

International Image Interoperability Framework (IIIF)

IIIF

• A model for presenting and annotating content
• A global community that develops shared application programming interfaces (APIs), implements them in software, and exposes interoperable content

https://iiif.io

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

A global community...

...but mostly from the Northern hemisphere

https://bit.ly/iiifmap

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Community

• State and National Libraries: Bavarian State Library, French National Library (BnF), British Library, National Library of Estonia, New York Public Library, Vatican Library, etc.

• Archives: Blavatnik Foundation Archive, Indigenous Digital Archive, Internet Archive, Swedish National Archives, Swiss Federal Archives, etc.

• Museums & Galleries: Art Institute Chicago, J. Paul Getty Trust, Smithsonian, Victoria & Albert Museum, MIT Museum, National Gallery of Art, Van Gogh Worldwide, etc.

• Universities & Research Institutions: Cambridge, Cornell University, Ghent University, Swiss National Data and Service Center for the Humanities (DaSCH), Kyoto University, Oxford, Stanford, University of Toronto, Yale University, etc.

• Aggregators/Facilitators: Europeana, Cuba-IIIF, Cultural Japan, OCLC ContentDM, etc.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Community

2019 IIIF Conference, Göttingen, Germany

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Community Practices

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Community Practices

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Images are fundamental carriers of information

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

The Problem

A world of silos and duplication

Image delivery on the Web has historically been hard, slow, expensive, disjointed, and locked-up in silos.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

The Problem

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Deep Zoom with Large Images

https://purl.stanford.edu/hs631zg4177

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Compare Images

Letter from Alexander Hamilton Papers (September 6, 1780), Library of Congress: https://prtd.app/#72f604db-6869-4c08-91ce-7c79502a7f35

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Reunify

https://demos.biblissima.fr/chateauroux/demo/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Search within

Franks, Kendal; Royal College of Surgeons of England. The Germ Theory. via Wellcome Library.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Storytelling

Storiiies: http://storiiies.cogapp.com/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Crowdsource

Crowdsourcing initiative from the National Library of Wales

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Machine-generated Annotations

See Cornut et al. [2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Beyond Images

https://ddmal.music.mcgill.ca/IIIF-AV-player/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF – Layers of digitisation

Leiden Collection's Curtain Viewer:
https://www.theleidencollection.com/viewer/david-and-uriah/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Application Programming Interface (API)

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Specifications

• Image API
• Presentation API
• Authorization Flow API
• Change Discovery API
• Content Search API
• Content State API

The Image and Presentation APIs are referred to as the core IIIF APIs

https://iiif.io/api

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Core IIIF APIs

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Image API

The IIIF Image API specifies a RESTful web service that returns an image in response to a standard HTTP(S) request.

https://iiif.io/api/image

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Image API

Uniform Resource Identifier (URI) Syntax

• Base URI
  {scheme}://{server}{/prefix}/{identifier}

• Image Request
  {$BASE}/{region}/{size}/{rotation}/{quality}.{format}
  https://sipi.participatory-archives.ch/SGV_12/SGV_12N_08589.jp2/full/1000,/0/default.jpg

• Image Information (Metadata)
  {$BASE}/info.json
  https://sipi.participatory-archives.ch/SGV_12/SGV_12N_08589.jp2/info.json

IIIF Presentation API

The IIIF Presentation API is a JSON-LD based web service which provides the necessary information about the object or collection structure and layout.

https://iiif.io/api/presentation

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Presentation API

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Presentation API

{
  "@context": "http://iiif.io/api/presentation/3/context.json",
  "id": "https://iiif.participatory-archives.ch/SGV_12N_08589/manifest.json",
  "type": "Manifest",
  "label": {
    "en": [
      "[Ringtanz während der Masüras auf der Alp Sura]"
    ]
  },
  "metadata": [
    {
      "label": {
        "de": [
          "Titel"
        ]
      },
      "value": {
        "de": [
          "[Ringtanz während der Masüras auf der Alp Sura]"
        ]
      }
    },

https://iiif.participatory-archives.ch/SGV_12N_08589/manifest.json

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Core IIIF APIs in Mirador

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Ecosystem

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

IIIF Ecosystem

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Web Annotation Data Model

https://www.w3.org/TR/annotation-model/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Web Annotation Data Model

Motivations

To uderstand the reasons why the Annotation was created, or why the Textual Body was included in the Annotation.

Some of the Motivations: commenting, highlighting, identifying, tagging

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Web Annotation Data Model in a IIIF setting

{
  "@context": "http://iiif.io/api/presentation/3/context.json",
  "id": "https://iiif.participatory-archives.ch/annotations/SGV_12N_08589-p1-list.json",
  "type": "AnnotationPage",
  "items": [
    {
      "@context": "http://www.w3.org/ns/anno.jsonld",
      "id": "https://iiif.participatory-archives.ch/annotations/SGV_12N_08589-p1-list/annotation-436121.json",
      "motivation": "commenting",
      "type": "Annotation",
      "body": [
        {
          "type": "TextualBody",
          "value": "person",
          "purpose": "commenting"
        },
        {
          "type": "TextualBody",
          "value": "Object Detection (vitrivr)",
          "purpose": "tagging"
        },
        {
          "type": "TextualBody",
          "value": "<br><small>Detection score: 0.9574</small>",
          "purpose": "commenting"
        }
      ],
      "target": {
        "source": "https://iiif.participatory-archives.ch/SGV_12N_08589/canvas/p1",
        "selector": {
          "type": "FragmentSelector",
          "conformsTo": "http://www.w3.org/TR/media-frags/",
          "value": "xywh=319,2942,463,523"
        },
        "dcterms:isPartOf": {
          "type": "Manifest",
          "id": "https://iiif.participatory-archives.ch/SGV_12N_08589/manifest.json"
        }}},

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art

Linked Art is a community and a CIDOC (ICOM International Committee for Documentation) Working Group collaborating to define a metadata application profile for describing cultural heritage, and the technical means for conveniently interacting with it (the API).

https://linked.art

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art Community

Institutions (some of them)

The American Numismatics Society, Europeana, The Frick Collection, J. Paul Getty Trust, The Metropolitan Museum of Art, The Museum of Modern Art (NY), National Gallery of Art (US), Oxford University (OERC), The Philadephia Museum of Art, Rijksmuseum (NL), University of Basel (Digital Humanities Lab), University of the Arts London, Victoria and Albert Museum, Yale Center for British Art

https://linked.art/community/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art Community Practices

https://groups.google.com/g/linked-art/c/8DcbDIExdS8/m/RTRQtOBsFQAJ

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Intent of Linked Art: finding the right balance

[Sanderson 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art Overview

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art Data Model Fundamentals

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art from 50k feet

[Raemy et al. 2023, adapted from Sanderson 2018]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art

Digital Object

https://linked.art/api/1.0/endpoint/digital_object/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art

Digital Object

{
  "@context": "https://linked.art/ns/v1/linked-art.json",
  "id": "https://linked.art/example/digital/0",
  "type": "DigitalObject",
  "_label": "Digital Image of Self-Portrait of Van Gogh",
  "classified_as": [
    {
      "id": "http://vocab.getty.edu/aat/300215302",
      "type": "Type",
      "_label": "Digital Image"
    }
  ],

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Art Digital Integration (with IIIF)

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

LOUD-Driven Infrastructure

[Felsing et al. 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Linked Open Usable Data (LOUD)

LOUD in a nutshell

• Grassroots development of IIIF and Linked Art with collaboration and transparency are one of the key factors, but implementations are needed to be conducted in parallel (specifications versus demonstrability).

• LOUD standards, when used in conjunction, enhances semantic interoperability, even if it comes at the cost of ontological purity.

• LOUD practices and standards should serve as common denominators for cultural heritage institutions, public bodies as well as research projects.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – Identifying the Movements and Principles

Match the movements and principles to these statements

In pairs or small groups, relate the movements and principles (OA, Open Data, Open Science, FLOSS, FAIR, CARE, Collections as Data, LOUD) to the following propositions (multiple answers possible).

1. Software development
2. Image dissemination
3. Metadata dissemination
4. Publication of scientific articles
5. Persistent identifier assignment
6. Digital object interoperability

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – Identifying the Movements and Principles

7. Development and maintenance of APIs
8. Documentation
9. Open license
10. Inclusivity
11. Machine-readable (meta)data
12. Collaboration
13. Ethical commitment
14. Semantic interoperability
15. Reusability

Platforms and Organisations

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Platforms

ORD Platforms

• Registry: re3data.org
• Cross-disciplinary: SwissUBase
• Humanities: DaSCH Service Platform
• Cross-institutional: OLOS
• Institutional: Yareta
• Generic: Zenodo

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

re3data.org

Registry of Research Data Repositories

• Platform launched in 2012
• Registry that includes data repositories from various academic disciplines
• Embeddable widgets and tools
  • Additional information, Data Accessibility, Terms of use and licences, Policy, Persistent Identifier (PID) system, Certification
• All metadata are available for open use under CC0. It also provides an API to access the content

http://www.re3data.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

SwissUbase

National cross-disciplinary solution

• Launched in 2021, superseding FORSbase
• Operated by FORS – Swiss Centre of Expertise in the Social Sciences – and the Universities of Lausanne, Neuchâtel and Zurich
  • Data catalogue, mostly from social sciences and linguistics
• Own metadata schema for studies, datasets, and data files
• Digital Object Identifier (DOI) at the dataset level

https://www.swissubase.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

DaSCH Service Platform (DSP)

Swiss National Data and Service Center for the Humanities

• Institutionalised in 2017 by the Digital Humanities Lab of the University of Basel
• Operated as a national research infrastructure since 2021 by DaSCH and primarily funded by the SNSF
• Project-based data models that rely on a core base ontology (Knora), own metadata schema describing the whole project
• RESTful API (JSON-LD), IIIF Image API, Archival Resource Keys (ARKs) with timestamps

https://app.dasch.swiss/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

OLOS

Consultation and archive portal of Switzerland

• National instance deployed in 2021
• Developed as part of the Data Life-Cycle Management (DLCM) project and operated by an association composed of the University of Fribourg, the HEG-GE and the HES-SO
• Dataset description based on the Datacite Metadata Schema
• DOI at the dataset level

https://olos.swiss/portal/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Yareta

Research Data Repository for Geneva's Higher Education Institutions

• Platform launched in 2019
• Developed as part of the DLCM project, based on OLOS, and operated by the University of Geneva

https://yareta.unige.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Zenodo

Generalist repository built at CERN

• Created by the CERN to be a generic solution for storing data
• Anyone can deposit data, with or without embargo
• Own metadata schema
• DOI per version

https://zenodo.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Platforms

OGD Platforms

Switzerland

• National: opendata.swiss
• Cantonal: Open Data Basel-Stadt
• Municipal: Stadt Zürich Open Data
• Public-Law Body: Open Data Portal of Geneva Public Transport

International

• EU: European Data
• USA: DATA.GOV

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

opendata.swiss

Swiss public administration’s central portal for OGD

• National platform launched in 2013 (first as opendata.admin.ch) under the direction of the Swiss Federal Archives. It exists as opendata.swiss since 2016 and is overseen by the Federal Statistical Office since 2019. It provides an overview of OGD published in Switzerland and is a joint project of the Confederation and the cantons.
• Source code accessible on GitHub
• Metadata accessible via a JSON API (CKAN) and an XML file according to the DCAT-AP CH standard. Unique identifier per dataset

https://opendata.swiss/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Basel-Stadt

Canton of Basel-Stadt's OGD

• Cantonal platform officially launched in 2019 (pilot project in 2017-2018)
• Plateform based on opendatasoft
• Own Metadata schema which comprises some DCAT and DCAT-AP CH properties. JSON API to explore the catalogue and the datasets. Dedicated dashboard. Opaque identifier per dataset.

https://data.bs.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Stadt Zürich Open Data

City of Zurich's OGD

• First OGD Platform in Switzerland, launched in 2012.
• Documentation and dedicated scripts on GitHub – Specialist Unit for Open Government Data Canton of Zurich
• Own metadata scheme which comprises DCAT-AP CH properties. (Geo)JSON APIs (see documentation). Non-opaque identifier per dataset

https://data.stadt-zuerich.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Portal of Geneva Public Transport

opendata.tpg

• Platform launched in 2022. First open data initiative in 2015 through their real-time transit time data API.
• Democratisation process: transparency, efficiency, innovation, and citizen participation
• Several metadata schemes and download possibilities including DCAT in RDF/XML. Dataset schema in JSON which comprises GeoJSON, a format for encoding a variety of geographic data structures. Non-opaque identifier per dataset.

https://opendata.tpg.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

European Data

The official portal for European data

• Platform launched in 2021 (beta version in 2015)
• Source code available on GitLab
• Metadata displayed using DCAT-AP (currently version 2.1.1) and accessible through a variety of APIs and an SPARQL endpoint (see documentation). Opaque identifier per dataset.

https://data.europa.eu/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

DATA.GOV

The Home of the U.S. Government's Open Data

• Platform launched in 2009 which provides access to datasets published by agencies across the federal government of the United States
• Based on open source applications (such as CKAN)
• Metadata displayed using DCAT-US Schema v1.1

https://data.gov/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Organisations

• Open Knowledge Network
• Opendata.ch
• Open Data Beer

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Knowledge Network

A Global Network for Open Data

• Launched in 2011
• Foundation which comprises several established chapters, members, and contributors from around the world
• Their mission is to create a fair, sustainable and open digital future, advancing open knowledge as a design principle beyond just data.

https://okfn.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Opendata.ch

Opendata.ch: Swiss Open Data Association

• Founded in 2011
• Swiss chapter of the Open Knowledge Network
• Dedicated working group for the GLAM (Galleries, Libraries, Archives, Museums) sector: OpenGLAM

https://opendata.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Beer

• Events since 2018 (around three events per year)
• Founded by Open Data practicioners in Switzerland
  • Federal Statistical Office
  • Canton of Basel-Stadt
  • Canton of Thurgau
  • Canton of Zurich
  • City of Zurich
  • SBB CFF FFS

https://opendatabeer.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

And much more...

Awesome Open Government Data Switzerland

A curated list of OGD portals, websites, APIs, tools and other related resources in Switzerland (and beyond)

https://github.com/rnckp/awesome-ogd-switzerland

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – Comparing Open Data Portals

Short comparative analysis of open data portals

In pairs or small groups, you will conduct a comparative analysis of one ORD portal and one OGD portal, neither of which has been previously discussed in our course. Your analysis will involve comparing these portals with similar ones that have already been presented, based on specific criteria.

1. Choose one ORD portal and one OGD portal now. Announce your chosen portals to ensure no overlap.
2. Dimensions to conduct the analysis: Launch Year, Purpose and Theme, Data Types, Access, Metadata Standards, Dataset Identifiers
3. Prepare a concise 5-minute presentation of your findings (with or without visual aids)

Assessment, Data Quality, and Best Practices

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Maturity (ODM)

Annual assessment

Exercise done by the EU since 2015 to measure the progress of European countries in promoting and facilitating the availability and reuse of public sector information (→ mostly OGD).

1. Policy – It investigates the open data policies and strategies in place in the participating countries, the national governance models for managing open data and the measures applied to implement those policies and strategies.
2. Impact – It analyses the willingness, preparedness and ability of countries to measure both the reuse of open data and the impact created through this reuse.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Open Data Maturity (ODM)

Annual assessment

3. Portal – It investigates the functionality of national open data portals, the extent to which users’ needs and behaviour are examined to improve the portal, the availability of open data across different domains and the approach to ensuring the portal’s sustainability.
4. Quality – It assesses the measures adopted by portal managers to ensure the systematic harvesting of metadata, the monitoring of metadata quality and compliance with the DCAT-AP metadata standard, and the quality of deployment of the published data on the national portal.

https://data.europa.eu/en/publications/open-data-maturity

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

ODM 2023

• 35 participating countries: EU-27, 3 European Free Trade Association Countries (Iceland, Norway and Switzerland), 5 candidate countries (Bosnia and Herzegovina, Montenegro, Albania, Serbia and Ukraine)
• Switzerland is 24th with an Open Data Maturity of 79%.

[Page et al. 2023]

https://data.europa.eu/en/publications/open-data-maturity/2023

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Measuring Impact

Impact Monitoring Framework

Used in Switzerland for the opendata.swiss platform

• Method to measure the value of OGD initiatives and projects.
• Based on a structured and consistent list of criteria
• Leverages the Social Return on Investment (SROI) approach to measure impact (input, output, outcome, impact)

[Stürmer 2016]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Data Quality and Best Practices

Data Quality

• Tracking the state of OGD: http://index.okfn.org/index.html
• Checklist Opquast - assurance qualité web
• Data Quality Score for the metadata evaluatin of the datasets, e.g. https://data.winnipeg.ca/Organizational-Support-Services/Data-Quality-Score/73sq-j2qi/about_data

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Data Quality and Best Practices

Best Practices, Toolkits

• The Annotated 8 principles of OGD (US): https://opengovdata.org/
• Open Government Data Toolkit

And of course the important principles from FAIR, CARE, Collections as Data, LOUD (and surely others)

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Periodic Table of Open Data Elements

English: https://odimpact.org/periodic-table.html
French: https://open.datactivist.coop/apps/periodic-table [Pichot Damon 2024]

Techniques, Software, and Tools

How to get and work with data?

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Data Scraping

• Process of extracting data from websites or other online sources, typically using automated software or scripts.
  • Advantages: It allows for efficient data collection from multiple sources, can automate repetitive tasks, and is capable of handling large volumes of data.
  • Challenges: Data scraping faces issues like website layout changes, legal and ethical considerations, as well as handling dynamic content loaded through JavaScript.
• Examples
  • Extracting exhibition data from museum websites using Beautiful Soup, a Python library;
  • Scraping historical records or archives from government websites with Scrapy.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

API Integration

• API integration involves connecting and pulling data from external services.
  • Advantages: It provides structured and often real-time access to data, allows for automation, and ensures data consistency and reliability.
  • Challenges: Complexity in handling API limits/rate limiting, maintaining integration after API updates, and managing data from disparate APIs.
• Examples
  • Integrating social media data from platforms like Instagram, LinkedIn or Mastodon;
  • Retrieving weather information from meteorological APIs.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Data Mining

• Data mining is the process of analysing large datasets to discover patterns, correlations, and insights.
  • Advantages: Helps in identifying trends, making predictions, and informing decision-making processes; can uncover hidden patterns in data.
  • Challenges: Requires significant computational resources, potential privacy concerns, and the need for skilled interpretation of results.
• Examples
  • Analysing visitor data patterns using RapidMiner, a Java-based data science platform;
  • Mining public opinion data from government surveys with WEKA (Waikato Environment for Knowledge Analysis), a Java-based Machine Learning software.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Data wrangling/munging

• Data wrangling, or munging, involves transforming and mapping raw data into a more structured and usable format.
  • Advantages: Makes data more accessible and useful for analysis, helps in cleaning and standardising data, and improves data quality.
  • Challenges: Time-consuming, requires expertise in data manipulation, and can be complex with large and diverse datasets.
• Examples
  • Formatting and combining different datasets for a research project using Python's pandas library;
  • Harmonising open government datasets from different departments for comparative analysis.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Data Integration

• Data integration involves combining data from different sources to provide a unified view of the data.
  • Advantages: Provides a comprehensive view of data, enhances data usability and analysis, and supports better decision-making.
  • Challenges: Managing data format and schema discrepancies, ensuring data quality and consistency, and handling large-scale integration.
• Examples:
  • Combining spatial data from various archaeological digs and historical GIS databases for comprehensive mapping and analysis;
  • Combining financial data from various business units.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Stream processing

• Stream processing is the technique of processing data in real-time as it flows in streams from various sources.
  • Advantages: Enables real-time data analysis and decision-making, can handle high throughput, and is suitable for time-sensitive data.
  • Challenges: Requires handling data velocity and volume, ensuring system scalability and reliability, and managing out-of-order data streams.
• Examples:
  • Real-time analysis of social media feeds using Apache Kafka;
  • Processing live public transport data for city management using Apache Flink.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Data Quality Management

• Data quality management involves ensuring the accuracy, completeness, and reliability of data in a dataset.
  • Advantages: Increases the trustworthiness of data, improves decision-making, and reduces the risk of errors in data analysis.
  • Challenges: Continuously maintaining data quality, especially with large and evolving datasets, and integrating quality management into existing processes.
• Examples
  • Using OpenRefine to clean and standardise metadata across different collections of historical artifacts;
  • Ensuring accuracy in patient data in healthcare databases.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Techniques

Extract, Transform, Load (ETL)

• ETL is a process where data is extracted from various sources, transformed into a suitable format, and loaded into a target system.
  • Advantages: Facilitates data consolidation, supports complex data transformations, and enables effective data storage and analysis.
  • Challenges: Managing data from disparate sources, ensuring data transformation accuracy, and maintaining ETL process performance.
• Examples
  • Extracting economic and demographic data from various government departments using Apache NiFi transforming it for consistency, and loading it into an aggregated portal.
  • Analysing sales data from different systems.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Comprehensive Knowledge Archive Network (CKAN)

Open Source Data Management System (DMS)

• CKAN is an open source DMS, mainly written in Python, for powering data hubs and portals. It is maintained by the Open Knowledge Foundation since 2006.
• It contains a PostgeSQL database, a Solr index, an API, and has several extensions.

https://ckan.org/
https://github.com/ckan/ckan

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Apache Tools

• Apache Kafka: a distributed event streaming platform designed for high-throughput, real-time data feeds, excelling as a scalable, durable, and fault-tolerant message broker for large-scale data integration and streaming
• Apache Flink: a stream processing framework optimised for stateful computations and complex event processing on unbounded data streams, offering robust event time processing, advanced windowing, and real-time analytics capabilities.
• Apache NiFi: a data flow management tool providing a user-friendly interface for automating, controlling, and monitoring data flows between systems, with strengths in data routing, transformation, and ensuring data provenance and compliance.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

OpenRefine

• Open source tool initally released in 2010 (first as Freebase Gridworks and then as Google Refine) for data cleanup and transformation
• It operates as a local web application to clean messy data and can be installed on Windows, macOS and Linux
• It handles various types of data (CSV, TSV, JSON, XML) and can connect to and import data from databases and other sources
• It supports scripting in languages like General Refine Expression Language (GREL) and Jython, allowing for advanced data manipulation
• It has various features: faceting/filtering, clustering, reconciliation, undo/redo

https://openrefine.org/
https://github.com/OpenRefine/OpenRefine

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – OpenRefine

Goal: gain hands-on experience in setting up and navigating OpenRefine

Step 1. Getting started with OpenRefine

1. Install the software (https://openrefine.org/docs)
2. Run it locally (accessible at http://127.0.0.1:3333/)
3. Have a look at the different pages and functionalities
4. Create a new project by importing any supported files

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – OpenRefine

Step 2. Create a project with an extract from the CAS photographic archives

5. Create a project by importing the data extract from the CAS photographic archives: ekws_extract.csv
6. Review the dataset
7. Clean the dataset by removing unnecessary columns
8. Undertake some reconciliation with external services for agents (people and institutions).

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – OpenRefine

Step 3. Going further

9. Create a new project by importing a dataset from one of the ORD/OGD portals
10. Analyse and curate the dataset

Alternatively: go through this tutorial from Library Carpentry: https://librarycarpentry.org/lc-open-refine/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Exercise – Analysing datasets from opendata.swiss

Starter code for all CSV datasets on opendata.swiss

1. Find a dataset from the preconfigured starter code files: https://rnckp.github.io/starter-code_opendataswiss/
2. Open it on Google Colab
3. Run the code snippets
4. You can also open it on GitHub (Jupyter Notebook or Rmd) and download the file

Explanation and Showcases: https://opendata.swiss/de/showcase/starter-code-fur-alle-csv-datensatze-auf-opendata-swiss

Assignment Workshop

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Assignement

Work on your assignment

Showcases

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Our World in Data

Research and data to make progress against the world’s largest problems

• Our World in Data is a collaborative effort between researchers at the University of Oxford and the non-profit organisation Global Change Data Lab (GCDL).
• It is a comprehensive online resource that presents empirical research and data on a wide array of global issues, focusing on large-scale problems like poverty, disease, hunger, climate change, war, existential risks, and inequality.
• The platform aims to provide accessible, comprehensible, and transparent data to inform readers about the state of the world and to support informed decision-making.

https://ourworldindata.org/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Infectious Diseases Dashboard (IDD)

• The Infectious Diseases Dashboard (IDD) is managed by the Federal Office of Public Health (FOPH)
• The IDD provides information on cases of infection and illness in Switzerland and the Principality of Liechtenstein caused by various pathogens.

https://idd.bag.admin.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Sportanlagen-Finder

The sports facility finder shows sports and exercise facilities operated by the canton of Basel-Stadt as well as all cantonal sports facilities outside the cantonal and national borders. The dataset also lists cantonal premises that are used and rented for sports activities.

• Dashboard: https://data.bs.ch/explore/embed/dataset/100151/custom/
• Data source: https://data.bs.ch/explore/dataset/100151/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Federal Popular Votes

Managed by the Federal Sttistical Office where results of popular votes in Switzerland are continuously updated.

https://abstimmungen.admin.ch/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Heile Preise

Launched in 2023 by Mario Zechner, this platform offers a comprehensive platform for comparing food prices across various supermarkets in Austria, tracking and analysing price trends over time.

• Platform: https://heisse-preise.io/
• News item: https://www.wired.com/story/heisse-preise-food-prices/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Animal Crossing Art Generator

• Innovative tool for integrating virtual art: The generator allows players to turn any image from the Getty Museum's open-access collection into miniature works of art for use in the Animal Crossing: New Horizons game.
• Creative Expression and Customisation: Players can use the tool to add famous artworks to their game by applying them to clothing, wallpaper, canvas, etc., enhancing their virtual environment with museum-quality art.
• Technical foundation and accessibility: Uses open source code from the Animal Crossing Pattern Tool and includes a IIIF manifest converter for broader art integration, making it easy to import art from various institutions into the game.

https://experiments.getty.edu/ac-art-generator/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Animal Crossing Art Generator

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

12 sunsets: Exploring Ed Ruscha's Archive

• Interactive platform launched in 2020 by the J. Paul Getty Trust to explore Sunset Boulevard throughout 60 years (between 1965 and 2007) as photographed by Ed Ruscha
• The 65,000 photographs are IIIF-compliant and are all linked to the Getty Research Institute

https://12sunsets.getty.edu/

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

12 sunsets: Exploring Ed Ruscha's Archive

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

LUX: Yale Collections Discovery

LUX provides a unified gateway to more than 41 million cultural heritage resources held by Yale's museums, archives and libraries: Yale University Library, Yale Center for British Art, Yale Peabody Museum, Yale University Art Gallery.

Built on open standards

• Linked Art, IIIF, W3C Activity Streams
• Widespread technologies: Python, JavaScript, Node.js, React, AWS
• Multimodal database (NoSQL): MarkLogic Server

https://lux.collections.yale.edu/

[Metcalfe Hurst 2023]

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

LUX: Yale Collections Discovery

Data pipeline and architecture

[Raemy & Sanderson 2023]

Data Transformation Pipeline Code: https://github.com/project-lux/data-pipeline

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

LUX: Yale Collections Discovery

Link to optimised video resolution

Conclusion

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

Long-term archiving of ORD

• Archival appraisal: Identifying which research data warrants long-term preservation, taking into account scientific, historical or legal significance as data volumes increase.

• Documentation and contextualisation: Ensuring comprehensive documentation for each dataset, including the context of its creation, to maintain its relevance and intelligibility over time.

• Infrastructure: Addressing the challenges of physical and software obsolescence, file format changes and the risk of content loss over time.

• Methodology/Process: Determining the most effective time and method for archiving, including periodic or project completion, to prevent data loss and ensure data integrity.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

Movements and principles impacting Open Data

• The rise of movements such as Open Access and Open Science, along with principles like FAIR and CARE, significantly shape the relevance and implementation of open data.

• However, achieving true openness requires not only adherence to these principles and movements but also the backing of sufficient funding and the cultivation of necessary skills among data practitioners. This is crucial for ensuring that open data is not just available but also meaningful and usable.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

The Evolution of Open Data

• While open data in itself is a commendable goal, the concept of Linked Open (Usable) Data takes it a step further.

• Linked Open Data enhances the value of open data by ensuring it is not only available but also interconnected, making it more discoverable and useful for a wider range of applications and analyses.

• LOUD is about enhancing usability and semantic interoperability leveraging community-driven standards and practices.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

OGD and ORD for GLAM institutions

• ORD and OGD can be viewed both as a service provided to the public and as a process that requires active management and continuous improvement.

• Institutions in the GLAM sector need to consider how these open data initiatives fit within their practices, both in terms of contributing data and utilising data for research, curation, and public engagement.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

Public Engagement and Empowerment

• Open data empowers the public by providing access to information that was previously inaccessible or difficult to obtain.

• This not only fosters a more informed citizenry but also enables individuals and communities to participate more actively in civic and cultural discourses.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

Transparency and Accountability

• Open data plays a pivotal role in enhancing transparency and accountability, particularly in sectors where public trust is paramount.

• By making data freely accessible, open data initiatives allow for greater scrutiny and analysis, leading to more accountable governance and institutional practices.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

AI and ML

• Open data serves as a critical fuel for AI systems, providing the large datasets necessary for training ML models. The availability of diverse, high-quality open datasets enables more robust and inclusive AI developments.

• By leveraging open data, AI can transcend a wide array of domains, from improving healthcare diagnostics to enhancing climate change models, thus contributing significantly to societal advancements and problem-solving.

• Open data plays a pivotal role in fostering transparency and ethical practices in AI. By using open datasets, AI researchers and developers can ensure a level of accountability in their models, allowing for external validation and reducing biases.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

Conclusion

Collaboration is Key

• Collaboration is a fundamental aspect of open data initiatives.

  • Discussing best practices grounded in collaboration, such as leveraging the Collections as Data checklist
  • Participating in the IIIF and Linked Art communities for the cultural heritage field (and beyond, notably for the STEM sector)
  • OGD meet-ups (Open Data Beer)
  • Etc.

• Such collaboration is vital for addressing global challenges, encouraging innovation, and ensuring the sustainable development of open data ecosystems.

Julien A. Raemy | 7C2-CT-4A Introduction to Open Data |

A multitude of tools

For a better understanding of the past,
Our images have to be enhanced,
A new dialogue in three dimensions,
Must have openness at its heart,
For somewhere within the archive
Of our aggregated minds
Are a multitude of questions
And a multitude of answers,
Simply awaiting to be found.

[Mr Gee 2023], Data Poet at EuropeanaTech 2023

References and Image Credits