Metadata Schema¶
In plain words
Describe and share your data covers what Thyra records and how to prepare a METASPACE submission, in plain words. This page has every field.
Every store Thyra writes carries a versioned, ontology-mapped metadata
block: table.uns["msi_metadata"]. Its base fields mirror the
METASPACE submission form, so a converted
dataset already holds every answer its source reports; the rest, such as
the organism and the matrix, is added when you export (see
Completing the metadata).
The schema exists because MSI has had no structured metadata convention the way other spatial omics modalities do. Vendor files spell the same fact many ways, imzML metadata is free-form cvParams, and every pipeline re-invents its own dictionary. This block fixes the names, maps them to ontologies, and ships a validator.
Where this sits in the format stack
Raw/archival exchange is the domain of mzPeak (the HUPO-PSI working draft succeeding mzML); SpatialData is the analysis-ready layer and Thyra's output. Thyra is the bridge between the two, and this schema is the analysis-layer metadata contract: PSI CV-anchored like the raw layer, so nothing is lost crossing the bridge. Raw ragged spectra stay in the raw layer; they are deliberately not part of Thyra's output.
import spatialdata as sd
sdata = sd.read_zarr("output.zarr")
block = sdata.tables["msi_dataset_z0"].uns["msi_metadata"]
print(block["schema_version"]) # "0.14.0"
print(block["ms_analysis"]["pixel_size_um"]) # {"x": 20.0, "y": 20.0}
print(block["ms_analysis"]["ionisation_source"]) # "MALDI"
print(block["ms_analysis"]["ionisation_source_term"])
# {"accession": "MS:1000075",
# "name": "matrix-assisted laser desorption ionization"}
The document¶
Seven sections and a processing list. ms_analysis and provenance are
written by the converter for every store; acquisition, calibration and
alignment are each written when the reader reports at least one of their
facts and are absent otherwise; sample and preparation describe things no raw file
records (what the tissue was, how it was prepared) and are supplied by you --
see Completing the metadata. The one exception is
the sample's id and name, which a conversion writes when it is given them
(convert_msi(sample_id=..., sample_name=...) or --sample-id).
| Section | Field | Type | Ontology |
|---|---|---|---|
| (root) | schema_version |
MAJOR.MINOR.PATCH string; thyra validate requires it, the published JSON Schema does not |
-- |
sample |
sample_id |
text, your identifier for the sample; never read from the source | PSI-MS (MS:1000001) |
sample_name |
text, your name for the sample; never read from the source | PSI-MS (MS:1000002) |
|
organism, organism_term |
text + term | NCBITaxon | |
organism_part, organism_part_term |
text + term | UBERON | |
condition |
text | -- | |
sample_growth_conditions |
text | -- | |
preparation |
sample_stabilisation |
text | -- |
tissue_modification |
text | -- | |
matrix, matrix_term |
text + term | CHEBI | |
matrix_application |
text | -- | |
solvent |
text | -- | |
ms_analysis |
polarity, polarity_term |
"positive"/"negative" + term |
PSI-MS |
spectrum_representation, spectrum_representation_term |
"profile"/"centroid" + term, the representation the reader delivered |
PSI-MS (MS:1000525; MS:1000128 / MS:1000127) |
|
ionisation_source, ionisation_source_term |
text + term | PSI-MS | |
analyzer, analyzer_term |
text + term | PSI-MS | |
instrument_model |
text | -- | |
manufacturer |
text, the source's own spelling where it states one; set by Thyra for Waters, Rapiflex and PHI, and "Bruker" for a solariX file that does not say |
PSI-MS (MS:1001269) |
|
serial_number |
text, identifies one physical machine | PSI-MS (MS:1000529) |
|
detector_resolving_power |
{value, at_mz} |
-- | |
pixel_size_um |
{x, y}, required |
-- | |
n_spectra |
integer, the spectra the source holds as its reader counted them | -- | |
ion_mobility |
{present, separation, separation_term, unit_term, range_lower, range_upper, num_scans, resolved_table, grid} |
PSI-MS (MS:1002815 / MS:1002476, unit MS:1002814) |
|
fragmentation |
{present, ms_level, constant_across_pixels, merges_precursors, dissociation_term, windows, resolved_table} |
PSI-MS (MS:1000511; windows MS:1000827 / 828 / 829, MS:1000045, MS:1000133) |
|
acquisition |
acquisition_datetime |
ISO 8601 YYYY-MM-DDThh:mm:ss[.fff], with a UTC offset only when the source recorded one |
-- |
laser_power_percent |
number, percent of the laser's range | -- | |
laser_frequency_hz |
number, hertz | IMS (IMS:1006000) |
|
shots_per_pixel |
integer, laser shots summed into one pixel's spectrum | IMS (IMS:1006001) |
|
method_file |
text, the acquisition method's file name (never a path) | -- | |
calibration |
calibration_datetime |
ISO 8601 at the source's precision (YYYY-MM-DDThh:mm[:ss[.fff]]), with a UTC offset only when the source recorded one |
-- |
recalibrated |
boolean, a calibration made after the acquisition replaced the one it ran under | -- | |
original_calibration_datetime |
as calibration_datetime, for the calibration a recalibration replaced |
-- | |
software, software_version |
text, what made the calibration, in the source's words | -- | |
n_reference_peaks |
integer, the reference peaks the calibration was fitted to | -- | |
mz_standard_deviation_ppm |
number, their residual in ppm (see below); never zero | -- | |
lock_mass_corrected |
boolean, the m/z values were corrected against a lock mass | -- | |
alignment |
optical_image_file |
text, the file name of the optical image the raster is registered onto (never a path) | IMS (IMS:1006008) |
method |
text, how that image was aligned with the raster, e.g. "teaching points" |
IMS (IMS:1006017) |
|
teaching_points |
list of {image_x_px, image_y_px, stage_x_um, stage_y_um}: a pixel of the image and the stage position of the same feature |
-- | |
processing |
list of {name, action_term, software {name, version, uri}, parameters} |
ordered steps, oldest first | PSI-MS data processing action (MS:1001485 for m/z calibration) |
provenance |
thyra_version |
text, required | -- |
source_format |
"imzml", "bruker", ... |
-- | |
source_path |
text, the source's path in a store block and its name alone in a document | -- | |
pixel_size_source |
"automatic" / "manual" / "default" |
-- | |
mass_axis |
optional {sha256, n_bins}: a SHA-256 of the store's var["mz"] (little-endian float64 bytes, lower-case hex) and its length; written for every conversion, resampled or not; absent from a document and from stores older than 0.11.0 |
-- |
An ontology term is always the pair
{"accession": "MS:1000075", "name": "matrix-assisted laser desorption ionization"} --
a CURIE plus the term's label, so the block
is readable without resolving anything.
pixel_size_um is the one acquisition field that is required: conversion
refuses to run without a pixel size, so a document without it describes no
store Thyra ever wrote. It is a pair because a raster need not be square, and
the same pair reaches the rest of the store -- the root attrs, the
coordinate system and its affine, the element transforms and
obs["spatial_x"]/["spatial_y"].
Unknown fields are rejected
Validation refuses keys the schema does not define, so a typo fails
loudly instead of becoming an unread field. Anything genuinely
vendor-specific already has a home in the sections beside this block
(format_specific, acquisition_params, raw_metadata -- see
Output Format).
What is auto-populated¶
Auto-population is deliberately honest: a field the source does not report is left unset, never guessed. The only inferences are facts that follow from the format itself.
| Source | polarity | ionisation source | analyzer | instrument model |
|---|---|---|---|---|
| imzML | from MS:1000130 / MS:1000129 in fileContent, a parameter group or the spectra, when all agree (MS:1000128 named "positive scan", a known exporter mistake, counts as positive) |
-- | from the <analyzer> component cvParam |
from the instrumentConfiguration (model term or MS:1000031 value) |
| mzPeak | from file_description.contents and each spectrum's scan_polarity, when all agree |
-- | as for imzML | as for imzML |
Bruker timsTOF .d |
from Frames.Polarity, when every frame agrees |
MALDI, when the laser tables are present | TOF | from the DB |
Bruker solariX .d |
from the file | -- | -- | from the file |
| Bruker Rapiflex | -- | -- | -- | -- |
| PHI ToF-SIMS | from the header | SIMS | TOF | platform name |
Waters .raw |
-- | -- | -- | SELECT SERIES MRT on an MRT, Thyra's own label; otherwise unset |
spectrum_representation is the representation the reader delivers, so
resampling never changes it:
| Source | spectrum representation |
|---|---|
| imzML | from the file's declaration (MS:1000127 / MS:1000128), or --spectrum-type |
| mzPeak | centroid when read from the peaks member, profile when the archive fills both members, else the file's declaration |
Bruker timsTOF .d |
centroid for TSF and for TDF read with vendor_centroid; profile for TDF read with scan_sum (the default) |
Bruker solariX .d |
centroid |
| Bruker Rapiflex, PHI ToF-SIMS | profile |
Waters .raw |
what the reader returns (--waters-spectrum) |
An imzML that declares neither term leaves the field unset. Thyra still
guesses centroid for a processed-mode file, to pick a resampling method, but
that guess stays in uns["essential_metadata"]["spectrum_type"].
A timsTOF .d also fills ion_mobility: present: true for a TDF acquisition
(TIMS engaged), with the acquired 1/K0 range and the ramp length in scans,
and present: false for TSF. An imzML states it as well: present: true
with the array's terms when the file declares a mobility array, present:
false otherwise. A Waters run with drift bins states present: true with
drift time (MS:1002476, ms), its range and the bin count. Other formats,
and Waters runs without drift bins, leave the field unset, which means "not
reported", not "no mobility". The MSI table is always summed over
the ramp; how it was summed is the tdf_spectrum parameter of the
conversion processing step (see Supported Formats).
resolved_table names the mobility-resolved sibling table when one was
written beside the summed table, and grid ({law, lower, upper,
n_channels}) describes the common mobility grid such a table was binned onto
when it was built from per-pixel mobility values (--mobility-grid); both
are unset otherwise. grid is the only thing in the store that says which of
the two mechanisms filled that table -- reading it off a shared feature axis
bins nothing -- and it is a description, not a structural difference: the
table is the same shape either way.
The arrays that describe the axis itself and the mass-mobility heatmap live
outside this block, in uns["mobility_axis"] and uns["mobility_heatmap"]
(see Output Format): this block is versioned
and carries no arrays.
Bruker .d also fills fragmentation from Frames.MsMsType and whichever
precursor table the acquisition uses -- PasefFrameMsMsInfo for PASEF frames,
DiaFrameMsMsWindows for DIA frames (MsMsType 9), FrameMsMsInfo for
single-precursor ones -- in a TSF acquisition as well as a TDF one. What
stays TDF-only is the demultiplexed sibling table, which needs the mobility
scan ranges only PASEF and DIA frames record. present: false records a survey
acquisition; the field is left unset when the database cannot be asked at all,
which means "not reported", not "MS1". windows is stored as a JSON string
(a list of objects does not round-trip through AnnData/zarr) and decoded by
read_msi_metadata_blocks and thyra validate. The field names and terms are
mzPeak's, so an archive and a store describe a precursor the same way; see
Output Format for the array block
beside it and for what merges_precursors means for the stored spectrum.
Waters .raw fills it too, from the precursor m/z MassLynx reports per scan:
a file whose converted functions carry one reports it here, and a file that
also holds MS1 functions converts those alone, reports MS1, and lists the
MS/MS functions it left out under excluded_functions in the Waters-specific
block. The MS level MassLynx reports is not used on its own, because on a
raster it split across functions the level is a chunk artefact (see
Supported Formats).
The acquisition section is filled the same way, from the vendor values
the reader already keeps in uns["acquisition_params"], and only where
the value's meaning and unit were checked against an acquisition:
| Source | acquisition_datetime |
laser_power_percent |
laser_frequency_hz |
shots_per_pixel |
method_file |
|---|---|---|---|---|---|
| Bruker tsf/tdf | GlobalMetadata.AcquisitionDateTime, with offset |
MaldiFrameInfo.LaserPower |
MaldiFrameInfo.LaserRepRate |
MaldiFrameInfo.NumLaserShots |
GlobalMetadata.MethodName, name only |
| Bruker solariX | Properties.AcquisitionDateTime, with offset |
Spectra.LaserPower |
Spectra.LaserRepRate |
Spectra.NumSummations (it equals the method's NumLaserShots) |
the *.m directory's name |
| Bruker rapiflex | -- (the info file's Start Time format is unverified) |
-- (unit unverified) | -- | Number of Shots |
Method |
| PHI ToF-SIMS | AcqFileDate, no offset |
-- (ion gun) | -- | -- | -- |
Waters .raw |
MassLynx's acquisition date and time, no offset | -- | -- | -- | $$ MS Method in _header.txt, name only |
| imzML | -- | -- | -- | -- | -- |
A Bruker per-frame value that varied across the acquisition is reported in
acquisition_params as a [min, max] pair and leaves the schema field
unset: the section states one value per acquisition or none. A timestamp
in a format the builder does not parse stays unset too, with the raw
string untouched beside it. No time zone is ever assumed: PHI and Waters
record none, so their timestamps carry none.
resolved_table names the demultiplexed sibling table when one was written,
mirroring ion_mobility.resolved_table, so both kinds of sibling are
discoverable from this block alone.
A precursor's m/z here and in var["precursor_mz"] is the isolation window
target (MS:1000827) -- the m/z the quadrupole was set to -- and not
MS:1000744, a selected ion whose m/z was measured. mzPeak keeps the two in
separate files for the same reason; do not read either as a monoisotopic mass.
detector_resolving_power is filled when the source states both the value
and the m/z it is quoted at (an extractor reports them as resolving_power
and resolving_power_at_mz); one without the other is not written, because
a resolving power is not comparable without its reference m/z. No shipped
reader states the pair today, so for the formats Thyra converts it is
supplied by you like the fields below.
n_spectra is the reader's own count of the spectra the source holds -- the
ones present, not the positions the raster covers, and one region's when one
was read. It is left unset where the reader did not count (a PHI preview
decodes no events, so thyra metadata has no count for PHI), and the table can
have fewer rows than it says: a spectrum with nothing in it is not stored.
The calibration section describes how the source's m/z values were
calibrated. "The calibration" is the one they rest on: the most recent
recalibration when there is one, otherwise the calibration the acquisition
ran under. Every field comes from what the vendor states, and only where its
meaning was checked against real acquisitions:
| Source | calibration_datetime |
recalibrated |
software, software_version |
n_reference_peaks, mz_standard_deviation_ppm |
lock_mass_corrected |
|---|---|---|---|---|---|
| Bruker tsf/tdf | CalibrationInfo.CalibrationDateTime in the analysis database, with offset; for recalibrated data the latest calibration.sqlite state's time, with the database's as original_calibration_datetime |
from calibration.sqlite: its first state is written at acquisition, so only a second is a recalibration; unset without the file |
CalibrationSoftware, CalibrationSoftwareVersion; the latest state's source and version when recalibrated |
the reference masses with a corrected mass, and MzStandardDeviationPPM; unset for a recalibration, since no recalibrated acquisition was at hand to check what its state records |
-- |
| PHI ToF-SIMS | the appended recalibration's BlockAppendedDate, no offset; unset for the header's calibration, which has no date |
an appended block with new coefficients | -- | the calibrants of the calibration in force, from their measured and theoretical m/z | -- |
Waters .raw |
$$ Cal Date and $$ Cal Time in _header.txt, to the minute, no offset |
-- | -- | -- (Cal StdDev is 0 on every file read) |
MassLynx isLockmassCorrected |
| imzML, solariX, rapiflex, mzPeak | -- | -- | -- | -- | -- |
recalibrated counts states. A .d whose calibration.sqlite was created
by a recalibration -- DataAnalysis writes one into a .d that had none --
holds that recalibration as its only state, so recalibrated is false and
the section describes the analysis database's calibration. The
m/z calibration step still names the state a conversion applied.
mz_standard_deviation_ppm is the square root of the reference peaks'
summed squared m/z errors after the calibration, in ppm, over one less than
the number of peaks. That is Bruker's MzStandardDeviationPPM, recomputed
from the file's own arrays to six decimals on four acquisitions, and PHI's
calibrants give the same statistic: refitting their theoretical m/z against
the flight times the stated coefficients imply reproduces the coefficients
exactly, so each calibrant's measured m/z is its position under the fit and
the difference its residual. A value is never written where it has nothing
to say. A standard deviation of zero is left unset -- a timsTOF calibrated on
two peaks fits both exactly, and the online lock-mass state a MALDI timsTOF
writes records 0.000000 against a single reference mass and no measured
mass at all -- and so is one over fewer than two peaks. The count stands on
its own.
Three things are deliberately not in the section. The first is which
calibration a conversion applied: that is a processing step (below). On a
timsTOF, Bruker's library turns every index into m/z itself. With
--use-recalibrated (the default) it applies the newest state in
calibration.sqlite, on TSF and TDF alike; with --no-recalibrated it
applies the analysis database's own calibration and none of the states.
Measured on one acquisition of each: a DataAnalysis 6.1 recalibration moved
every ion of a TSF by +1.3 to +1.7 ppm, and a SCiLS Lab 2027a alignment,
written into a TDF as its second state, moved each frame by -2.3 to +0.1 ppm.
With --no-recalibrated both read exactly as acquired. lock_mass_corrected
stays unset for timsTOF, for two reasons. Whether the m/z carry the online
lock-mass correction depends on that option, so it describes a conversion,
not the source. And on a TDF the lock-mass state changed no m/z on either of
the two acquisitions checked, so what it corrects cannot be read off the
file. The mobility calibration's keys are not read: on the one imaging TDF
with TIMS engaged its measured voltages are all zero and its standard
deviation is 3578 %. And no person reaches the section: CalibrationUser
and MobilityCalibrationUser are not read, and neither is the reference
list's name, which is text the lab chose.
The alignment section says which optical image the source registers its
raster onto, and how. Only flexImaging states one, in the .mis file beside
a Bruker acquisition:
| Source | optical_image_file |
method |
teaching_points |
|---|---|---|---|
Bruker tsf/tdf, rapiflex, solariX, with a .mis |
<ImageFile>, name only |
"teaching points" when there are three or more |
each <TeachPoint> |
| Everything else | -- | -- | -- |
The stage positions are in micrometres, in the frame the teaching was done
in. The positions the acquisition records its spectra at are offset from that
frame, so the points place the image on the target, not on those positions.
The .mis Areas stay in raw_metadata: an acquisition can hold fewer of them
than the file lists. Whether a conversion placed the raster in the image's
pixels is for the store's coordinate_systems attribute to say (see
Coordinate Systems).
Everything else -- organism, tissue, condition, matrix -- cannot come from a raw file and stays empty until you provide it.
Processing history¶
processing is the dataset's processing provenance, modeled on
mzQC: an ordered list of steps, each
naming the software that performed it and the parameters it ran with, and
the PSI-MS data processing action it is (action_term) where one exists.
The converter records its own steps -- conversion always, m/z calibration
when the reader turned flight times or digitiser indices into m/z with a
calibration it chose, and mass axis resampling with the resolved resampling
parameters when resampling was enabled. Downstream tools (normalisation, peak
picking, annotation) append theirs when they modify the store.
In mass axis resampling, mass_width_da is the bin width at
reference_mz. It is the width asked for when that width set the bin count.
When a bin count or the 100-bin floor set it instead, it is the width the
count gives, and on a tof axis bins_per_fwhm is too. A constant axis
lays its count as points, so its step is (max_mz - min_mz) / (n - 1): a
recorded width that was asked for is that step up to a factor n / (n - 1).
The width law (tof_a, tof_b) is recorded on a tof axis only.
[
{"name": "conversion",
"software": {"name": "thyra", "version": "4.1.0"}},
{"name": "m/z calibration",
"action_term": {"accession": "MS:1001485", "name": "m/z calibration"},
"software": {"name": "thyra", "version": "4.1.0"},
"parameters": {"calibration": "appended"}},
{"name": "mass axis resampling",
"software": {"name": "thyra", "version": "4.1.0"},
"parameters": {"method": "nearest_neighbor", "target_bins": 50000,
"reference_mz": 1000.0}}
]
The conversion step's parameters say what the conversion kept:
| Parameter | When | Meaning |
|---|---|---|
tdf_spectrum: "scan_sum" / "vendor_centroid" |
Bruker tdf | how each frame's mobility scans became one spectrum |
intensity_threshold: a number |
--intensity-threshold was given |
peaks whose summed intensity is below it are not in the store |
The m/z calibration step is how a store says which calibration it holds,
as distinct from what the source states in calibration: a conversion can be
told to apply one the source does not consider current. Its parameters are
in the reader's own terms:
| Source | Parameter | Meaning |
|---|---|---|
| PHI ToF-SIMS | calibration: "appended" / "header" |
the coefficients Thyra computed m/z from: the recalibration appended to the file whenever there is one, unless the reader's use_appended_calibration was turned off |
| Bruker tsf/tdf | use_recalibrated_state: true / false |
the option Bruker's library was opened with (--use-recalibrated / --no-recalibrated) |
calibration: "calibration.sqlite" / "analysis.tsf" / "analysis.tdf" |
where the calibration the library applied is stored: the newest calibration.sqlite state with the option on, the analysis database with it off or without that file. Left out when the option is on and calibration.sqlite could not be read |
|
calibration_state_id |
the Id of the calibration.sqlite state applied, when one was |
Other readers take the m/z values the source stores, apply nothing and record no step.
On disk the list is stored as a JSON string (AnnData/zarr cannot
round-trip a list of objects -- the same reason uns["regions"] is
JSON); read_msi_metadata_blocks and validate_document decode it
transparently.
PSI CV alignment¶
Beyond the per-document *_term values, every schema field that
instantiates a PSI CV concept is bound to that concept's accession in
the JSON Schema itself (a cv annotation on the field definition), so
the claim "Thyra's output is annotated with the same PSI CV terms as
the raw file it came from" is machine-checkable from the committed
artifact alone:
| Field | CV concept |
|---|---|
ms_analysis.pixel_size_um.x |
IMS:1000046 pixel size (x) |
ms_analysis.pixel_size_um.y |
IMS:1000047 pixel size y |
sample.sample_id |
MS:1000001 sample number |
sample.sample_name |
MS:1000002 sample name |
ms_analysis.polarity |
MS:1000465 scan polarity |
ms_analysis.spectrum_representation |
MS:1000525 spectrum representation |
ms_analysis.ionisation_source |
MS:1000008 ionization type |
ms_analysis.analyzer |
MS:1000443 mass analyzer type |
ms_analysis.instrument_model |
MS:1000031 instrument model |
ms_analysis.manufacturer |
MS:1001269 instrument vendor |
ms_analysis.serial_number |
MS:1000529 instrument serial number |
ms_analysis.detector_resolving_power |
MS:1000800 mass resolving power |
ms_analysis.ion_mobility.separation |
MS:1002892 ion mobility attribute |
ms_analysis.fragmentation.ms_level |
MS:1000511 ms level |
ms_analysis.fragmentation.windows[].target |
MS:1000827 isolation window target m/z |
ms_analysis.fragmentation.windows[].lower_offset |
MS:1000828 isolation window lower offset |
ms_analysis.fragmentation.windows[].upper_offset |
MS:1000829 isolation window upper offset |
ms_analysis.fragmentation.windows[].collision_energy |
MS:1000045 collision energy |
acquisition.laser_frequency_hz |
IMS:1006000 repetition rate |
acquisition.shots_per_pixel |
IMS:1006001 laser shots per spectrum |
alignment.optical_image_file |
IMS:1006008 optical image location |
alignment.method |
IMS:1006017 method used to align optical image |
On the input side, every imzML file-description cvParam is preserved in
uns["raw_metadata"]["cvParams"] with its accession (and unit
accession where the source set one) -- the name alone cannot be resolved
back to the CV concept. The list is stored as a JSON string (see
Output Format for why); json.loads
hands back the list of terms. Polarity is read from these terms, from the
parameter groups and from every spectrum. A file that states both values
anywhere records neither. A metadata-only read (thyra metadata, the
preview) looks at the first spectrum only.
Candidate CV terms¶
Several imaging concepts this schema needs have no CV term yet. They
are tracked in CANDIDATE_CV_CONCEPTS as the vocabulary to raise in
the PSI/mzPeak imaging discussions, so this schema and the future
standard converge:
- pixel size semantics (raster pitch vs laser spot vs binned size)
- pixel size provenance (measured vs user-supplied vs default)
- coordinate origin and axis handedness
- stage offset of the raster origin
- ROI / acquisition region identity
- missing / empty pixel semantics
- continuous-vs-processed source provenance after conversion
- mass axis resampling provenance (method, axis law, target bins)
- mass axis identity (a hash of the stored m/z values)
- acquisition start timestamp (mzML has only the run's
startTimeStampattribute;MS:1000747is the completion time) - laser power as a percentage of the instrument's range (
MS:1000846pulse energy is in joules) - acquisition method identity (
MS:1002128names a method file format, not the method) - number of spectra of any MS level (
MS:4000059andMS:4000060count MS1 and MS2 spectra separately) - when an m/z calibration was made, whether a recalibration replaced it, and
what software made it (
MS:1001485m/z calibration is a processing action with no attributes;MS:1003200software version is scoped to spectral libraries) - an m/z calibration's fit: its residual in ppm and how many reference peaks
it rests on (
MS:1000014accuracy is an analyzer attribute,MS:4000072the error of one identified ion) - lock-mass correction of the m/z values
- a teaching point: a pixel of the optical image paired with the stage
position of the same feature (
IMS:1006017names the alignment method, with no term for the points it rests on)
MS:1001485 itself is used, as the action_term of the m/z calibration
processing step -- the one place the schema names a calibration as
something done rather than something stated.
LinkML rendering¶
The schema is also rendered as LinkML at
thyra/metadata/schema/msi_metadata.linkml.yaml -- classes, slots,
required flags, slot_uri CV bindings and the polarity enum with
meaning: accessions. The pydantic models remain the source of truth
for what Thyra writes and validates; the YAML is the discussion artifact
for LinkML-native settings (the PSI/mzPeak imaging work, the planned
spec repository), and a unit test keeps the two from drifting. A later
migration to LinkML-as-source changes no field names and nothing on
disk.
var column conventions¶
The MSI table's .var column names are fixed by the spec so every
consumer can rely on one spelling:
| Column | Written by | Meaning |
|---|---|---|
mz |
every converter, required | The common mass axis. Numeric, finite, strictly increasing -- except on a sibling table, where the pair is what is unique and sorted: (mz, mobility) on a mobility-resolved one, (precursor_mz, precursor_mobility, mz) on a demultiplexed MS/MS one, where precursor_index identifies the block. |
mobility |
the converter, on mobility-resolved tables only | The feature's ion mobility (1/K0 or drift time; see uns["mobility_axis"]). Its presence is what marks the table as mobility-resolved. |
precursor_mz |
the converter, on demultiplexed MS/MS tables only | The isolated m/z the feature's fragments came from (see uns["msms_schedule"]). Its presence is what marks the table as demultiplexed; such a table never carries mobility as well. |
mz_index |
the converter, on sibling tables only | Column of the feature's m/z on the summed MSI table's axis |
mobility_index |
the converter, on mobility-resolved tables only | The feature's position on the table's mobility axis: the rank of its mobility among the distinct values a shared-axis source lists, or the channel it fell in on a common grid (ion_mobility.grid) |
precursor_mobility |
the converter, on demultiplexed MS/MS tables only | The 1/K0 the precursor was isolated at (the middle of its mobility window). What tells two precursors sharing an m/z apart -- an isomer pair -- so they are never merged |
ccs |
the converter, on mobility-resolved tables whose axis is 1/K0 | The feature's collision cross section in square angstroms, computed by Mason-Schamp from its mz and mobility for an assumed charge of 1, in nitrogen at 305 K. uns["ccs"] records those assumptions. A drift-time table has no such column. Any table that carries it must hold numeric, finite, positive values |
precursor_index |
the converter, on demultiplexed MS/MS tables only | The precursor's position in this store's precursor axis, and the identity of its column block. Means nothing outside the store: align two stores on (precursor_mz, precursor_mobility) |
formula |
annotation tools | Molecular formula of the annotation |
adduct |
annotation tools | Adduct, e.g. +H, -H, +Na |
annotation_source |
annotation tools | Tool/database that produced the annotation |
fdr |
annotation tools | False discovery rate of the annotation |
Readers may add extra per-channel columns (a non-m/z native axis such as
flight time stays alongside mz), and annotation tools may add columns
beyond these -- but the reserved names above must never be reused with a
different meaning. thyra validate checks the mz contract on every
table of a store, and the ccs contract on every table that has one.
obs column conventions¶
One .obs name is fixed by the spec as well. The pixel columns beside it
(x, y, spatial_x, region_number and the rest) are described in
Output Format.
| Column | Written by | Meaning |
|---|---|---|
acquisition_order |
the converter, when the reader knows the order; absent otherwise | int64, grows with acquisition time and is unique within a table. The source's own spectrum number where it has one (Bruker Frames.Id, solariX Spectra.Id), else the spectrum's 0-based position in the order the source lists them. A row summed from repeated measurements takes the earliest. Sort by it; it is not a row index. See Acquisition order |
A missing column means the order is unknown. The name must never be reused with a different meaning.
Storage contract¶
- The block lives at
table.uns["msi_metadata"], in every table the store has. This location is stable, likeuns["essential_metadata"]and thecoordinate_systemsroot attribute; consumers read it from here and nowhere else. - It is written identically by every converter write path (the uns parity tests assert this), and records no time of the conversion itself -- the acquisition's start time comes from the source -- so converting the same input twice produces the same block.
- Sections with nothing in them are omitted rather than written empty, following the store-wide convention.
Versioning¶
schema_version follows semantic-version rules:
- Adding an optional field bumps the minor version.
- Renaming, removing, retyping, or making a field required bumps the major version.
- A validator implementing major version N rejects documents with a different major version, accepts older minors, and warns on newer minors.
Versions so far: 0.1.0 (initial), 0.2.0 (ms_analysis.ion_mobility
added), 0.3.0 (ion_mobility.resolved_table and ion_mobility.grid added),
0.4.0 (ms_analysis.fragmentation added), 0.5.0
(fragmentation.resolved_table added), 0.6.0 (the acquisition section
added), 0.7.0 (ms_analysis.manufacturer and ms_analysis.serial_number
added), 0.8.0 (the calibration section, ms_analysis.n_spectra and
the processing steps' action_term added), 0.9.0 (the alignment section
added), 0.10.0 (the obs column acquisition_order reserved; the document
is unchanged), 0.11.0 (provenance.mass_axis added), 0.12.0
(sample.sample_id, sample.sample_name and
ms_analysis.spectrum_representation added), 0.13.0 (the var column
ccs reserved; the document is unchanged), 0.14.0
(the coordinate system "acquisition" reserved; the document is unchanged).
The JSON Schema rendering is published at a fixed, versioned address,
which is also its $id:
Every version from 0.6.0 on has its own folder under that path, and a
published folder is never edited; there is deliberately no latest. The LinkML source of
the same version is served beside it as msi_metadata.linkml.yaml. A
program that writes the document without Thyra validates against that
address; see Writing the Metadata Document.
The same file is committed at
thyra/metadata/schema/msi_metadata_schema_v0_14.json and ships in the
wheel, so a Python consumer can validate documents offline without
importing Thyra:
from importlib import resources
import json
schema = json.loads(
resources.files("thyra.metadata.schema")
.joinpath("msi_metadata_schema_v0_14.json")
.read_text()
)
Unit tests keep the package artifact in sync with the models and the
published copy in sync with the package artifact; regenerate both with
python -m thyra.metadata.schema.generate after a model change. A
version bump creates a new folder under docs/schema/; the previous
version's folder stays as it is. docs/schema/SHA256SUMS records the
SHA-256 of every published file, and a unit test fails when any of them
changes, so regenerating a version that is already published fails the
suite instead of rewriting it. The pull request that adds a version
folder appends that folder's lines to SHA256SUMS; the failing test
prints them.
thyra metadata¶
Builds the same document from a raw source and writes it out, without
converting anything. INPUT is a file or folder in any format Thyra reads.
For every format but Waters, no spectra are decoded and no vendor SDK is
loaded, so it is a header read whatever the dataset's size; a Waters run is
described through the MassLynx library, which reads every scan. The default
output is stdout.
The document is the one a conversion would have stored, with three
differences, and a fourth for PHI data. Two follow from nothing having been
converted: processing is absent, and no sibling table is named; for PHI,
ms_analysis.n_spectra is absent too, because counting the pixels that
carry a spectrum means decoding the whole event stream. The third follows
from where
the document goes: provenance.source_path is the source's name,
where a store block carries the path it was converted from. A store sits
on the machine that wrote it and the path is provenance a reader can act
on; a document is written to be handed to somebody, and an absolute path
describes a filesystem they do not have while carrying a user directory
to a machine that has no use for it. Everything else -- the sections, the
ontology terms, the JSON shape -- is identical, so a document written
this way and the same dataset's block read back out of a store compare
directly.
An acquisition with no raster has no pixel size, which this schema version requires, so its document is reported as invalid and the command exits 1 -- while still writing the document, which is the only kind such an acquisition has. See Design Decisions.
thyra validate¶
PATH is a converted .zarr store or a standalone metadata .json
document. Only the metadata block (and, for stores, the var axis) is
read -- validating a 100 GB store is instant. For stores it also checks
the var column contract and that every table
carries a metadata block. Exit status is 0 when every document
conforms (warnings allowed) and 1 otherwise, so it can gate CI.
On Windows, a store whose files sit past the 260-character path limit is read through an extended-length path automatically; without that, Zarr would return empty terms for the keys it cannot open and the document would fail validation for the wrong reason (see long paths).
Errors mean the document does not conform: structural violations, unknown PSI-MS/IMS/UO accessions, version incompatibility. Warnings mean it conforms but says something suspicious: a term label that does not match its accession, or a term from an unexpected ontology.
thyra validate output.zarr
# msi_dataset_z0: OK
thyra validate output.zarr --json # machine-readable report on stdout
thyra export-metaspace¶
Writes the METASPACE submission metadata JSON (default:
<input>.metaspace.json next to the input; -o - for stdout). Required
fields the store cannot know are written empty, or left out in the case of
Detector_Resolving_Power, and reported as warnings on stderr -- the output
is a truthful starting point, never a fabricated record. Two values are
filled without being stated: matrix-free sources (DESI, SIMS) get
MALDI_Matrix: "none", and an unset solvent is written as "none".
thyra export-metaspace output.zarr
# warning: Sample_Information.Organism is required ... and is not set
# Wrote METASPACE metadata for msi_dataset_z0 to output.metaspace.json
Completing the metadata¶
The fields only you can know go in a small JSON overlay, merged at
validation or export time with --merge:
{
"sample": {
"organism": "Mus musculus",
"organism_term": {"accession": "NCBITaxon:10090", "name": "Mus musculus"},
"organism_part": "liver",
"organism_part_term": {"accession": "UBERON:0002107", "name": "liver"},
"condition": "wildtype"
},
"preparation": {
"sample_stabilisation": "fresh frozen",
"matrix": "2,5-dihydroxybenzoic acid (DHB)",
"matrix_term": {"accession": "CHEBI:17189", "name": "2,5-dihydroxybenzoic acid"},
"matrix_application": "ImagePrep"
},
"ms_analysis": {
"detector_resolving_power": {"value": 130000, "at_mz": 400}
}
}
thyra validate output.zarr --merge sample.json
thyra export-metaspace output.zarr --merge sample.json
The overlay merges key-by-key over the stored block, so it only needs the
fields you are adding. Keep it next to the store; it applies unchanged to
every dataset from the same study. Name each sample at conversion instead
(--sample-id, or sample_id and sample_name in convert_msi): that
name is stored, so it tells stores of one study apart. One conversion names
one sample, even when the slide holds several regions.
Python API¶
from thyra.metadata.schema import (
MSIMetadata, # the pydantic model
read_msi_metadata_blocks, # {table_name: block} from a store
validate_document, # (model | None, issues)
to_metaspace, # (submission_dict, warnings)
)
blocks = read_msi_metadata_blocks("output.zarr")
meta, issues = validate_document(blocks["msi_dataset_z0"])
submission, warnings = to_metaspace(meta)