Machine Learning-based Annotations Embedded into IIIF Presentation API 3.0 Resources

Use Case from the PIA Research Project

Zentralbibliothek Zürich | 20.02.2023

Julien A. Raemy, PhD Candidate in Digital Humanities
DHLab, University of Basel
ORCID Google Scholar GitHub Mastodon

PIA

Participatory Knowledge Practices in Analogue and Digital Image Archives

  • Institute for Cultural Anthropology and European Ethnology, University of Basel
  • Digital Humanities Lab, University of Basel
  • Bern Academy of the Arts, Bern University of Applied Sciences
Julien A. Raemy

Collections of the Swiss Society for Fokflore Studies (SSFS) within PIA

Developing a Citizen Science platform on the basis of:

  • SGV_05 Atlas der Schweizerischen Volkskunde - Cartography
  • SGV_10 Familie Kreis - Vernacular Photography
  • SGV_12 Ernst Brunner - Photojournalism
Julien A. Raemy

Linked Open Usable Data for Cultural Heritage

Perspectives on Community Practices and Semantic Interoperability

PhD Thesis - https://phd.julsraemy.ch - supervised by:

  • Peter Fornaro (University of Basel)
  • Walter Leimgruber (University of Basel)
  • Robert Sanderson (Yale University)
Julien A. Raemy

PIA Infrastructure / APIs

At the moment:

  • IIIF Image API 3.0
  • IIIF Presentation API 3.0
  • Omeka S API

In the future:

  • Linked Art API 1.0
  • IIIF Change Discovery API 1.0
Julien A. Raemy
Julien A. Raemy

Annotations

  • Machine Learning: Leveraging vitrivr to detect objects (Faster R-CNN Inception Resnet V2)
  • From humans: a proof-of-concept still needs to be undertaken

The goal is that all these annotation outputs are compatible with the Web Annotation Data Model.

Julien A. Raemy
Julien A. Raemy
Julien A. Raemy

Presentation API 3.0 Resources

  • Manifest: https://iiif.participatory-archives.ch/<id>/manifest.json
  • AnnotationPage: https://iiif.participatory-archives.ch/annotations/<id>-<listid>.json

Example of a Manifest displayed in our Mirador instance: https://mirador.participatory-archives.ch/?manifest=https://iiif.participatory-archives.ch/SGV_12N_19783/manifest.json

Julien A. Raemy
{
      "motivation": "commenting",
      "type": "Annotation",
      "body": [
        {
          "type": "TextualBody",
          "value": "person",
          "purpose": "commenting"
        },
        {
          "type": "TextualBody",
          "value": "Object Detection (vitrivr)",
          "purpose": "tagging"
        },
        {
          "type": "TextualBody",
          "value": "<br><small>Detection score: 0.7576504945755</small>",
          "purpose": "commenting"
        }
      ],
      "target": {
        "source": "https://iiif.participatory-archives.ch/SGV_12N_19783/canvas/p1",
        "selector": {
          "type": "FragmentSelector",
          "conformsTo": "http://www.w3.org/TR/media-frags/",
          "value": "xywh=1439,2597,417,944"
          }}}
Julien A. Raemy

Image Credits

  • [Blick auf das Spalentor]. Basel, 1938. Ernst Brunner. SGV_12N_00115
  • ["Steffenbach-Brücke" der Furka-Bahn: Bau und Erneuern der Brücke]. Kanton Wallis, 1950. Ernst Brunner. SGV_12N_36937

These images are part of the photographic archives of the Swiss Society for Folklore Studies.

Licence: CC BY-NC 4.0

This presentation is about the deployment of IIIF within the PIA research project

PIA is a Sinergia project funded by the Swiss National Science Foundation (SNSF) led by the University of Basel, the Uni, the Bern Academy of the Arts, and the Swiss Society for Folklore Studies. PIA wants to connect the world of data and things in an interdisciplinary manner. We explore the phases of the analogue and digital archive from the perspectives of cultural anthropology, technology and design. The common goal of this project is to design a visual interface with machine learning-based tools to make it easy to annotate, contextualize, organize, and link both images and their meta-information, to deliberately encourage the participatory use of archives.

I am doing my PhD in Digital Humanities on Linked Open Usable Data, with a focus on its (potential) use in the Humanities and the perspectives it could bring in terms of semantics and interoperability. My research is grounded as part of the Participatory Knowledge Practices in Analogue and Digital Image Archives (PIA) research project, which aims to develop a Citizen Science platform around three photographic collections of the Swiss Society for Folklore Studies (SSFS).

IIIF Workflow within PIA - We have developed an application for IIIF Resources based on Laravel, an opensource PHP Framework11 for generating IIIF resources (Manifests and Collections) as well as machine-learning Annotations, which are derived from vitrivr, a content-based multimedia retrieval system. To serve IIIF manifests, the application consumes metadata from our repositories. Our main database is managed through Omeka S

For object comparison, displaying annotations, as well as manipulating images, we decided to opt for Mirador. New cookbook recipes will be integrated into PIA, such as relevant patterns facilitating the display of geographical metadata within or pointing to maps. This monitoring work is indeed simplified as one of us is a member of the IIIF TRC, but a more automated setting without necessarily going through each recipe or the viewer matrix should in our opinion be sought