Product Coordination Group

PCG#2

March 2025

The two sessions of the meeting have been blended into a single report wherever possible for clarity and completeness. The text of this summary was edited with the help of AI.

Introduction

The OBIS Products Coordination Group met in two sessions, chaired by Jon Pye and Stephen Formel, with Secretariat support from Silas Principe and Laurent Chmiel. The agenda covered updates from the Secretariat, including the estimated costs of the products infrastructure, the design of the Products Catalogue, and how products can provide clear metadata and citation for the data they use.

Updates from the Secretariat

Costs of the products infrastructure. Costs vary a great deal with the size of the user base, so it is better to assess them once the infrastructure is in use. The community should be shown how usage affects costs, so that it can be used sensibly. The group discussed whether to open the infrastructure beyond OBIS Nodes to the broader biodiversity community, for example a metabarcoding initiative. It was considered important to start using it in a less restrictive way, respond to any large usage costs as they arise, and demonstrate and quantify who is using the infrastructure in order to seek funding on that basis.

→ It was suggested that Pangeo’s experience building out similar services could be useful, and that Rich Signell could be approached to share it (https://www.pangeo.io/steering-council).

→ Jon Pye explained that OTN had avoided much of this cost institutionally by provisioning an old server with virtual machines running Rundeck and JupyterHub. He was interested in OBIS’ costs over time, to understand how they compare with the cost of a cloud solution.

Latest developments. The Secretariat is working on several projects at once. One is particularly important for product development: the files on the OBIS server are moving from PostgreSQL to Parquet format, which will make it easier to generate products through the Airflow process. This was welcomed as a big improvement.

Products Catalogue

The Products Catalogue has been a priority since the first PCG meeting, and CKAN is the priority for action in the coming months. The software is installed and working (https://ckan.obis.org/) but not yet implemented: it holds only one or two example datasets. The Secretariat proposed developing a simplified first version that covers all expected uses, starting to use it to see what is needed, and improving it along the way.

Proposed design. The Secretariat shared figures and screens drawing on examples such as the Humanitarian Data Exchange and the CIOOS data catalogue. The proposal includes a search bar, highlight buttons, new products and news items, and a way to browse products by ocean variable, with products tagged to those categories. Browsing by ocean variable would be clean and simple, and would support OBIS’ aim of positioning itself as a place that helps users derive biodiversity metrics for CBD or national monitoring. It should be clear that everything in the catalogue is a product, as distinct from a dataset. Products would be pipelined from GitHub repositories through Airflow and the OBIS data lake, then expressed in CKAN and potentially GeoServer. The final address for the catalogue, such as products.obis.org or catalogue.obis.org, was on the agenda for discussion.

→ Dan Lear asked whether everything could be linked to ODIS. In principle this is possible, and the requirements will be checked. He also asked whether a Node-based CKAN would be possible, and noted that MBA is exploring options for its research data outputs.

→ Ben Donnelly said OBIS-SEAMAP had explored using CKAN in the past and looked forward to seeing how its products could connect to the catalogue. Jon Pye noted that CKAN is stackable, as in the CIOOS example.

→ Katherine Tattersall said that if OBIS builds great products and makes them available to the research partners who contribute data, and to government, policy and monitoring programmes, it will be an enormous success for OBIS and excellent for outreach and usability.

Tagging with EOVs and EBVs. CIOOS does something similar: users can tag their dataset with EOVs when publishing, followed by a curation step. OBIS Australia tags all its datasets with EOVs internally, though these tags are not currently passed through to Darwin Core.

Data flow. The group asked for the steps of the data flow to be set out in detail, with examples of how the infrastructure can be used to produce products, and noted that open data should be used in the data orchestration step.

Openness and licensing. The group discussed whether products could be built in a private setting. It also asked whether it would be reasonable to require a range of open, remixable software licences from users of OBIS resources, as OBIS already does for datasets.

Metadata and citation

The Secretariat presented an idea for discussion on how products can provide clear information about the data they use, prompted by a GitHub issue (https://github.com/iobis/mpaeu_sdm/issues/4).

A DOI service for products. Source datasets would carry DOIs, with a fallback where needed. A service would take a list of source DOIs as dependencies and produce a DOI for the product. The product’s DOI would contain the dataset IDs, and ideally the DOIs, of every dataset used, together with the version of each. This would make it possible to track provenance, to see the point at which the product was produced, and to trigger updates to the product.

→ Jon Pye pointed to the R package targets as an example of how changes to underlying datasets can trigger changes to a product.

→ Ben Donnelly asked how the DOI holds the information about the datasets. It is held in the metadata, with details to be set out by Pieter Provoost.

→ Dan Lear considered this technically a good approach, but stressed that how it is shown to users is critical. The NCG could contribute ideas on what Nodes want to see when individual datasets are showcased.

Visualizing attribution. Visualizing these products will be a priority, and showing attribution for very large numbers of datasets (thousands) will be a difficult problem. The MPA Europe citations page (https://iobis.github.io/mpaeu_docs/citations.html) was cited as an example of the current approach, which could be improved. Network diagrams could be a good solution.

Related resources. Suggestions included an ontology for the relationships between datasets and their derivations, on which Pierluigi Buttigieg, who has worked in this space building ENVO, may be able to advise; ROR (https://ror.org/) for institutional attribution; RAiD (https://raid.org/) for attributing research effort; CHORUS (https://www.chorusaccess.org/); Provena (https://docs.provena.io/); and the CiTO citation typing ontology (https://sparontologies.github.io/cito/current/cito.html).

Minimum metadata. The group discussed the minimum metadata required for products published in the catalogue. Darwin Core will not cover everything, and Dublin Core may be a good representation of the minimum.

Shared resources and experience

The table of resources and experience begun at the previous meeting was extended, adding OBIS-UK (Kevin Paxman) to those with GeoServer experience and OBIS-UK to those with CKAN experience. Members were asked to complete the second table, of product needs already identified, which can help in adapting and implementing the technical infrastructure.

Action items