OceanGliders tag adoption across glider data centres

Findings for data management across data centres

—

Written as a starting point for organising glider data management across data assembly centres, and for a federated OceanGliders ERDDAP. Every number below is computed from the same data as the charts above, so it follows each data refresh; the statements around them were reviewed against the 2026-10-27/28 snapshot (findings reviewed 2026-10-28), and the spot checks at the end record what was verified by opening files then.

Data centre profiles

Where data centres behave differently

Guidance for each data centre

For each data centre: what is missing for an OceanGliders dataset (tags, format, QC description, identifiers), and where it does the same thing differently from the others.

Recommendations for a federated OceanGliders ERDDAP, and what stands in its way today, are on the Federated ERDDAP tab.

—

A federated OceanGliders ERDDAP

The idea. Each data centre runs its own ERDDAP serving glider data in OG1.0 format. A federating ERDDAP lists all of them as one catalogue: it holds no data itself, and each of its datasets points to the data centre's ERDDAP (ERDDAP's EDDTableFromErddap), so a user query is passed on to the data centre and answered there. Users get one place to search and download all OceanGliders data; data stay with, and are maintained by, the data centres that hold them.

The demo. The OceanGliders data management team has explored the feasibility with a quick demo by Callum Rollo (VOTO): github.com/voto-ocean-knowledge/OG-GDAC-demo. It runs ERDDAP in docker, searches the VOTO and BODC ERDDAPs for OG1.0 datasets and adds each one as a pass-through dataset. Its own checklist: done: docker setup, federating from one and from several servers, search; not yet: checking datasets with the IOOS compliance checker, distinguishing real-time and delayed-mode datasets consistently, queries across all datasets, and a persistent server. The demo includes only VOTO and BODC because they are the only data centres serving OG1.0 today, not because the others failed a test.

What each data centre offers today

Measured while building this dashboard (checked 28 to 29 October 2026).

Querying tip. For large or slow datasets, ask ERDDAP for one time window at a time (for example &time>=2013-09-12T00:00:00Z&time<2013-09-19T00:00:00Z) and join the pieces: ERDDAP then reads only the files covering that window. Averaging on the server (orderByMean("profile_id")) returns one row per profile instead of every measurement.

Current bottlenecks

1. Format. A federation of OG1.0 data can include only the data centres that serve OG1.0: today VOTO and BODC. IOOS and OTN serve the IOOS Glider DAC format, Coriolis EGO and IMOS its own format.
2. No query service. Coriolis and IMOS publish files rather than ERDDAP datasets, so there is nothing for a pass-through federation to point to; they would need an ERDDAP over their files, or the federating ERDDAP would have to hold copies.
3. Speed is inherited. A pass-through dataset is as fast as its source. Where a data centre's ERDDAP assembles each dataset from thousands of per-profile files (OTN), a query over a whole older dataset can take longer than the server's 90-second gateway allows, at the source and in the federation alike. Two remedies: users and tools can bound queries in time (ERDDAP keeps each file's time range and opens only the files a query needs: one week of an OTN dataset that timed out as a whole answered in 4 seconds, and the whole deployment in 11 seconds week by week); and data centres can combine profile files into one file per deployment, or cache, so that whole-dataset queries are fast too.
4. Real time vs recovered vs delayed mode. Data centres label products differently ("delayed" means recovered data at IOOS, VOTO and OTN, and delayed-mode QC at IMOS), so a federated catalogue cannot yet tell users which product they are getting. The proposed OG1 data modes (R, P, D) would solve this if every data centre applies them the same way.
5. File access. Federation needs queries, but a full copy or a check of the files needs file access: IOOS blocks it, BODC's file collection has no byte-range access, OTN allows it.
6. The same deployment in several places. Some deployments are held by more than one data centre (for example NOC deployments at BODC and Coriolis, sometimes with different content); without a shared deployment identifier a federated catalogue would list them twice.
7. Metadata quality. Placeholder dates (1970, catalogue end dates years ahead), missing tags and free-text QC statements carry straight through a pass-through federation; checks at the data centre, or in the federation layer, are needed.

What would make it work

Recommendations for data centres, from the findings of this audit.

1. One dataset per deployment and stream, with a mandatory data mode. Publish real-time, recovered and delayed-mode products as separate datasets with a consistent name and a data_mode attribute that every data centre fills the same way (R real time, P recovered, D delayed-mode QC, as proposed for OG1). Stop using "delayed" for recovered data.
2. A controlled QC level. Replace free-text processing_level with a short controlled list (raw; automatic QC; delayed-mode QC), plus a citation of the QC procedure and the processing software and version.
3. One flag scheme, CF-described. Require flag_values and matching flag_meanings on every QC variable, from one agreed scheme linked to a vocabulary (SeaDataNet L20 has been proposed), with real flag values, not placeholders.
4. Required discovery attributes, as global attributes. network and site (validated against the BOON vocabulary), WMO ID, glider model, institution as a persistent identifier (e.g. EDMO or ROR), and time_coverage_start/end.
5. ERDDAP-to-ERDDAP harvesting. Every data centre exposes each product as an ERDDAP dataset (not a file tree), so a federated ERDDAP can aggregate them with EDDTableFromErddap; recovery files need byte-range access at minimum.
6. A shared deployment identifier. The same deployment appears at more than one data centre (e.g. NOC on BODC and Coriolis); a persistent deployment ID (OceanOPS ID, or WMO ID plus start date) avoids double counting and lets a federated catalogue link the copies.
7. Validation at ingestion. Check attributes, vocabulary values and time coverage when files arrive, and report back to the provider; most issues above would be caught automatically.

Audit and methods

For manual checks of this automatic audit, the file audit (CSV, or open the table below) lists every file and ERDDAP dataset read: its URL, the stream (real time or recovered) and the rule that set it, the QC level with the rule and the attribute text it matched (processing_level, data_mode, qc_manual, data_processing_chain_name), format with Conventions and format_version, the network and site values, the QC flag scheme, and whether the file was used for the track map (status, decimation, points kept and dropped).

Assumptions and limitations

• A snapshot of six data centres. Read on 2026-10-27/28 from IOOS, VOTO, BODC, OTN, Coriolis and IMOS. Glider data held only elsewhere (other national or institutional servers, EMODnet) are not counted, so totals describe these six, not the global fleet.
• Metadata, not data. Everything comes from what the files and catalogues state (global attributes, flag attributes, names, indexes). Work that was done but not recorded is invisible: a centre may apply delayed-mode QC that its files don't mention, and the dashboard will show "automatic" or "not stated".
• QC level is read from free text. Most data centres describe QC in free-text processing_level, classified by keywords (rules under Definitions). Unusual wording can be misclassified; QC flag values themselves were checked in only one file.
• Real time vs recovered follows each centre's naming. Dataset names, folders and data-mode letters were taken at face value. Coriolis data mode P was checked as full resolution on two Slocum deployments (Ifremer, SOCIB) and assumed for all P files; IMOS mission (FV01) folders are taken as recovered from their "Delayed Mode" label.
• What counts as a deployment. Records are grouped per data centre by its own key (IOOS dataset without -delayed, VOTO glider and mission, BODC and OTN deployment number, Coriolis and IMOS folder). A deployment split across several keys, or one key holding several deployments, is miscounted.
• Unique deployments are an upper bound. Copies on different centres are matched only on WMO ID and start date within 3 days. IMOS records no WMO IDs and OTN few, and wrong start dates break matches, so some duplicates remain.
• Tagged means the attribute is there. A record counts as tagged if network starts with "OceanGliders"; values outside the BOON vocabulary still count. Format is read from Conventions and format_version strings; files were not validated against the OG1.0 specification.
• CF check is shallow. "CF-described" means every flag variable has flag_values and a flag_meanings list of the same length; meanings were not checked against a vocabulary, nor flags against the data.
• Glider-days come from metadata time ranges. They can include pre- or post-deployment test data; spans under 0 or over 400 days are left out, and deployments without an end time (BODC folders without ERDDAP datasets, some Coriolis files) have none. The total sums unique deployments, taking the longest span in each match group.
• BODC file collections were not opened. BODC's recovery and older real-time OG1 files can't be read by byte range, so their tags and QC are unknown ("Not read") unless an ERDDAP dataset exists for the same deployment; they are assumed to be OG1 from the collection name.
• Platform and institution are text matching. Glider type is matched in attributes and titles (IOOS has no glider-model attribute, so many IOOS records are "Unknown"); wave gliders were dropped the same way. Institution names were merged by a hand-made alias list (raw names kept in the data); groupings such as "Multi-institution partnership" and separate OOI arrays are judgment calls.
• "Active since Oct 2024" uses the end date, or the start date where no end date is known.
• Built with an AI assistant. Harvest, classification and page were produced by Claude for Filipa Carvalho and spot-checked against raw metadata and files (table below); not independently reviewed. The code is public at github.com/FiIipaCarvalho/og-dac-metadata so the analysis can be re-run and checked.

Spot checks behind these findings

How this dashboard was made

The prompt below, synthesised from the requests made while building this dashboard, reproduces the harvest, the analyses and this page. The code, the data snapshot and this page are public at github.com/FiIipaCarvalho/og-dac-metadata (README, PROMPT.md, harvest scripts).

Build an interactive dashboard, for the OceanGliders steering team and glider data-centre managers, showing how far glider data across data centres meet OceanGliders' requirements (OG1.0 format, `network` and `site` tags from the BOON metadata guidelines, an approved data centre), and use it as a launchpad for organising data management across data assembly centres (DACs), with a federated OceanGliders ERDDAP in mind. Frame everything as readiness, not compliance: the `network`/`site` requirement was only agreed in October 2026.

## Data: harvest every glider deployment from six data centres, from the root of each

1. **IOOS Glider DAC** (https://gliders.ioos.us/erddap): `allDatasets`, then `info/<datasetID>/index.csv` for every dataset. Keep all global attributes, the `platform` variable's `wmo_id` and `type`, the variable list, and `flag_values`/`flag_meanings` of every flag variable.
2. **VOTO** (https://erddap.observations.voiceoftheocean.org/erddap): the `nrt_`, `delayed_`, `OG_nrt_` and `OG_complete_` datasets; same fields.
3. **BODC** (https://linkedsystems.uk/erddap): datasets with glider/Slocum/Seaglider in the title, plus the OG1 file collections `Public_OG1_Data_001` (real time) and `Public_OG1_Data_001_Recovery` (recovery), one EGO deployment JSON per folder (the files can't be read by byte range).
4. **OTN** (https://erddap.oceantrack.org/erddap): Trajectory/TrajectoryProfile datasets.
5. **Coriolis, the OceanGliders GDAC** (https://data-oceangliders.ifremer.fr/): `glider_traj_index.txt` (time coverage, WMO, glider model, owner per file), each deployment folder's JSON, and each NetCDF header read by HTTP byte range.
6. **IMOS ANFOG** (S3 `imos-data`, prefix `IMOS/ANFOG/`): one header per deployment folder (FV01 preferred) by byte range.

Gotchas to handle: some ERDDAPs publish a global attribute `id` that is not the datasetID (store the datasetID separately); netCDF4 isn't thread-safe (read files in processes); check a file's signature before choosing the classic or HDF5 reader (a wrong guess sits through long timeouts); EGO files have no time-coverage attributes and often fill `TIME` but not `JULD`; the Coriolis index `time_coverage_start` has 1970/2000 placeholders and earlier-deployment starts (use `deployment_start_date`, or the folder date when they disagree by more than 60 days); keep Coriolis to a few parallel requests.

## One row per deployment per data centre, with

- **Tag:** `network` starts with "OceanGliders"; mission and region from its `>` parts; **site** from `site` or `oceanGliders_site`.
- **Format:** OG1.0, EGO, IOOS DAC (incl. older Unidata/COARDS files), IMOS, Other; plus OG1 available on any file.
- **Data mode column:** from `data_mode`, else naming, with recovered datasets as P (never D from a name).
- **Stream:** real time (telemetered, often decimated) vs recovered (full resolution). Recovered: IOOS `-delayed`, VOTO `delayed_`/`OG_complete_`, OTN `_delayed`, BODC recovery collection, Coriolis data mode P (verify by sampling resolution of a P vs R file), IMOS mission folders. "Delayed" in names means recovered, not delayed-mode QC.
- **QC level per stream** (highest stated): Delayed mode (manually calibrated / QC'd in delayed mode, data_mode D, IMOS "Delayed Mode"); Automatic QC (automatic, QARTOD, Argo-method, EGO QC-tests manual DOI 10.13155/51485 or EGO processing chain, data_mode A); No QC (no QC, raw, Level 0, "as is", IMOS FV00); else Not stated; BODC folder-only deployments Not read. Check "no QC" wording before "delayed mode", because IOOS writes "Delayed mode dataset" for recovered raw data.
- **QC flags:** present (flag attributes or `_qc` names); CF-described (every flag variable has `flag_values` and an equal-length `flag_meanings`); scheme by majority of flag variables (IOC/QARTOD 1,2,3,4,9; Argo-style 0-9; SeaDataNet L20 with A/B/Q; Other; Not described).
- **Platform:** Slocum, Seaglider, SeaExplorer, Spray or Unknown from attributes, title/summary text and the Coriolis index; drop wave gliders everywhere.
- **Glider-days:** end minus start; leave out spans under 0 or over 400 days.
- **WMO ID**, **active since** two years before the snapshot, **institution** normalised (merge spelling variants and acronyms, keep the raw string), and a **match group** for the same deployment on more than one data centre (same WMO ID, start dates within 3 days).

## Dashboard

- Each data centre has its own colour throughout. Tagged records are drawn in that colour, untagged in grey; categories are told apart by **pattern** in the data centre's colour (solid for the target category, then stripes, dots, fine lines, faded, outline), never by shades of one colour.
- **Tiles**, in three labelled rows: deployments (records, unique, glider-days); OceanGliders network tag (tagged deployments, share, glider-days share, data centres with tags "N of M", institutions tagging "N of M"); OG1.0 format (the same five figures for OG1.0).
- **Deployments by data centre** (tagged solid, untagged faded).
- **By start year**, with a Highlight switch (Tagged | OG1.0), a Records | Glider-days switch and three views: highlighted vs all (grey all, highlighted part coloured by data centre), All by data centre, highlighted only.
- **Where the tags point:** mission > region.
- **Format**, **real-time and recovered data by QC level** (two rows per data centre), **QC flags** (present/CF and scheme) and **platforms**, all with the same three-way switch.
- **Largest institutions** split by data centre.
- **Readiness matrix** by institution and data centre (network tag, site tag, recovered data, delayed-mode QC, WMO ID, active, OG1.0, QC flags, CF flags, glider-days), sortable and searchable, with a Gradient | Traffic light switch (red under 50%, amber 50-80% with a strong amber, green 80% and over).
- **Definitions** for every criterion and caveat.
- **Assumptions and limitations** of the analysis (snapshot scope, metadata not data, free-text QC classification, naming-based streams, deployment keys, matching as an upper bound, shallow CF and format checks, glider-days from metadata, unread BODC files, text-matched platform and institution, AI-built and spot-checked), and **How this dashboard was made** with this prompt, both linked from the top of the page.
- **Findings report** at the end: a lede, data-centre profile table (traffic light), "where data centres behave differently" table (stream separation, meaning of "delayed", QC level statements, flag schemes, format, tags, WMO ID, glider model, time coverage, institution naming, access, processing software), a guidance card per data centre ("missing or incomplete" and "does differently", with its key numbers), recommendations for a federated OceanGliders ERDDAP (one dataset per deployment and stream with R/P/D data mode; controlled QC level; one CF-described flag scheme; required global discovery attributes validated against the BOON vocabulary; ERDDAP-to-ERDDAP harvesting; a shared deployment ID; validation at ingestion), a worked duplication example (NOC on BODC and Coriolis), and a table of the spot checks made by opening files.

## Verify before publishing

Spot-check at least one deployment per data centre against its raw metadata; check every claim in the guidance against the data (counts, shares, who holds the tags); state what was checked by opening files; and keep every number on the page computed from the data.