DNA sequence sanitization

GBIF sanitizes the raw DNA/RNA sequences published in occurrence records before they are indexed and made searchable. The goal is to produce a normalised, comparable sequence together with a set of quality metrics, while keeping every transformation transparent and reproducible.

Each occurrence sequence is passed through a fixed, ordered pipeline. The original raw sequence is never altered in the source data — only a cleaned copy is derived for indexing. The cleaned sequence is hashed (MD5) to produce a stable nucleotideSequenceID used to group identical sequences across records.

Pipeline stages

The pipeline applies the following stages in order. Each stage takes the output of the previous stage as its input.

Stage Description Example

Stage	Description	Example
Normalise whitespace and case	All whitespace characters are removed, and the sequence is upper-cased. If any whitespace was present, this is recorded and contributes to the `gapsOrWhitespaceRemoved` flag.	`"acgt acgt"` → `"ACGTACGT"`
Detect natural language	The sequence is checked for configured marker words (for example primer or read-merging labels). A match flags the record but does not modify the sequence (see Sequence flags and validity.	`"ACGTUNMERGEDACGT"` → flagged
Remove gaps	Characters matching the gap pattern (`-` and `.`) are removed. If any gap was present, this contributes to the `gapsOrWhitespaceRemoved` flag.	`"ACGT-ACGT..ACGT"` → `"ACGTACGTACGT"`
RNA to DNA conversion	RNA sequences are normalised to DNA by replacing every `U` (uracil) with `T` (thymine), so that RNA and DNA representations of the same sequence collapse to a single form.	`"ACGU"` → `"ACGT"`
Question marks to N	Question marks, sometimes used to denote an unknown base, are replaced with the IUPAC ambiguity code `N`.	`"ACGT?ACGT"` → `"ACGTNACGT"`
Trim to anchors	Leading and trailing characters are removed until a run of valid nucleotide "anchor" characters is found at each end. Front trim: the sequence is scanned for the first run of at least `anchor_minrun` consecutive anchor characters (`anchor_chars`). Everything before that run is discarded. If no such run exists anywhere in the sequence, the sequence is considered unusable and is set to an empty string. Back trim: the sequence is then scanned for the last anchor run, and everything after it is discarded. If either end is trimmed (or the sequence is set to an empty string), `endsTrimmed` is set to `true`.	`"THISISMYGBIFSEQUENCEACGTACGTACGTNNNNNENDOFSEQUENCE"` → `"ACGTACGTACGT"`
Cap N-runs	Long stretches of `N` carry no information and can dominate similarity comparisons. Any run of `N` of length `nrun_cap_from` or greater is reduced to `nrun_cap_to` characters. The number of runs that were shortened is reported as `nRunsCapped`.	`"ACGTACGTNNNNNNNNNNNACGTACGT"` → `"ACGTACGTNNNNNACGTACGT"`

Normalise whitespace and case

All whitespace characters are removed, and the sequence is upper-cased. If any whitespace was present, this is recorded and contributes to the gapsOrWhitespaceRemoved flag.

"acgt acgt" → "ACGTACGT"

Detect natural language

The sequence is checked for configured marker words (for example primer or read-merging labels). A match flags the record but does not modify the sequence (see Sequence flags and validity.

"ACGTUNMERGEDACGT" → flagged

Remove gaps

Characters matching the gap pattern (- and .) are removed. If any gap was present, this contributes to the gapsOrWhitespaceRemoved flag.

"ACGT-ACGT..ACGT" → "ACGTACGTACGT"

RNA to DNA conversion

RNA sequences are normalised to DNA by replacing every U (uracil) with T (thymine), so that RNA and DNA representations of the same sequence collapse to a single form.

"ACGU" → "ACGT"

Question marks to N

Question marks, sometimes used to denote an unknown base, are replaced with the IUPAC ambiguity code N.

"ACGT?ACGT" → "ACGTNACGT"

Trim to anchors

Leading and trailing characters are removed until a run of valid nucleotide "anchor" characters is found at each end.

Front trim: the sequence is scanned for the first run of at least anchor_minrun consecutive anchor characters (anchor_chars). Everything before that run is discarded. If no such run exists anywhere in the sequence, the sequence is considered unusable and is set to an empty string.
Back trim: the sequence is then scanned for the last anchor run, and everything after it is discarded.

If either end is trimmed (or the sequence is set to an empty string), endsTrimmed is set to true.

"THISISMYGBIFSEQUENCEACGTACGTACGTNNNNNENDOFSEQUENCE" → "ACGTACGTACGT"

Cap N-runs

Long stretches of N carry no information and can dominate similarity comparisons. Any run of N of length nrun_cap_from or greater is reduced to nrun_cap_to characters. The number of runs that were shortened is reported as nRunsCapped.

"ACGTACGTNNNNNNNNNNNACGTACGT" → "ACGTACGTNNNNNACGTACGT"

Quality metrics

All metrics are calculated on the final cleaned sequence.

Field Type Description

Field	Type	Description
`sequence`	string	The final cleaned sequence. Set to `null` when the record is invalid.
`sequenceLength`	integer	Length of the cleaned sequence in bytes.
`nonIupacFraction`	float	Fraction of characters that are not valid IUPAC DNA codes.
`nonACGTNFraction`	float	Fraction of characters that are ambiguous IUPAC codes (anything other than A, C, G, T or N).
`nFraction`	float	Fraction of characters that are `N`.
`nRunsCapped`	integer	Number of N-runs that were shortened in the last stage.
`gcContent`	float	GC content, calculated over A/C/G/T bases only.
`naturalLanguageDetected`	boolean	Whether natural-language marker words were found.
`endsTrimmed`	boolean	Whether either end of the sequence was trimmed.
`gapsOrWhitespaceRemoved`	boolean	Whether any whitespace or gap characters were removed.
`nucleotideSequenceID`	string	MD5 hash of the final cleaned sequence, used for indexing and grouping. `null` when the record is invalid.
`invalid`	boolean	Whether the record is considered invalid (see below).

sequence

string

The final cleaned sequence. Set to null when the record is invalid.

sequenceLength

integer

Length of the cleaned sequence in bytes.

nonIupacFraction

float

Fraction of characters that are not valid IUPAC DNA codes.

nonACGTNFraction

float

Fraction of characters that are ambiguous IUPAC codes (anything other than A, C, G, T or N).

nFraction

float

Fraction of characters that are N.

nRunsCapped

integer

Number of N-runs that were shortened in the last stage.

gcContent

float

GC content, calculated over A/C/G/T bases only.

naturalLanguageDetected

boolean

Whether natural-language marker words were found.

endsTrimmed

boolean

Whether either end of the sequence was trimmed.

gapsOrWhitespaceRemoved

boolean

Whether any whitespace or gap characters were removed.

nucleotideSequenceID

string

MD5 hash of the final cleaned sequence, used for indexing and grouping. null when the record is invalid.

invalid

boolean

Whether the record is considered invalid (see below).

The fraction metrics are null when the cleaned sequence is empty, and gcContent is null when the sequence contains no A/C/G/T bases.

Sequence flags and validity

Sanitization records a set of flags describing what happened during cleaning, for example, whether the ends were trimmed or gaps removed, and whether the result is usable. A sequence is flagged invalid when, after cleaning, it still contains characters that are not valid IUPAC DNA codes (nonIupacFraction > 0) or when natural language is detected. Invalid sequences are not indexed: both sequence and nucleotideSequenceID are set to null, while the quality metrics are still reported so the reason can be inspected.

The individual flags and their definitions are documented alongside GBIF’s other occurrence flags — see Sequence issues.

Configuration

The pipeline is driven by a small set of configuration parameters, allowing the thresholds and character sets to be tuned without changing the logic.

anchor_chars: "ACGTU"        # Valid anchor characters used when trimming ends
anchor_minrun: 8             # Minimum consecutive anchors required to define an end
gap_regex: "[-\\.]"          # Characters removed as gaps
natural_language_regex: "REVERSE|REV|FWD|FORWARD|MERGED|UNMERGED"  # Marker words
iupac_rna: "ACGTURYSWKMBDHVN"  # Valid IUPAC RNA codes
iupac_dna: "ACGTRYSWKMBDHVN"   # Valid IUPAC DNA codes (used for nonIupacFraction)
nrun_cap_from: 6             # Cap N-runs of this length or longer
nrun_cap_to: 5               # ...down to this length

Example

Given the raw input "ACGT-ACGT NNNNNNNNNN ACGT" the sanitization stages are as follows:

Whitespace is removed → "ACGT-ACGTNNNNNNNNNNACGT"
Gaps are removed → "ACGTACGTNNNNNNNNNNACGT"
No U or ? are detected, and the sequence is left unchanged
Anchor runs are detected at both ends, and the sequence is left unchanged
Caps the run of ten N down to five → "ACGTACGTNNNNNACGT"

The result has sequenceLength 17, gcContent 0.5, nRunsCapped 1, and gapsOrWhitespaceRemoved true.

Example API response

The sanitized sequence and its metrics are delivered through the GBIF occurrence API as a nucleotideSequence array on each occurrence record. The array can also be used as a search filter, for example nucleotideSequence.invalid=false to return only records with at least one valid sequence.

"nucleotideSequence": [
  {
    "nucleotideSequenceID": "a3ebee35f7fc2883cf68b22c9e9f6ca4",
    "targetGene": "COI",
    "sequence": "CGTTATATTTTTTATTAGGGAGATGATCTGCGATAATAGGGACGGCTATGAGAGTTTTAATTCGGGTGGAGTTGGGGAGAACGGGAAGATTAATCGGGGATGATCATTTATATAATGTTGTTGTGACTGCTCATGCTTTAGTGATGATTTTTTTTATAGTTATGCCTATCTTAATTGGGGGATTTGGAAATTGGCTGGTTCCTTTAATATTGGGTGCACCGGATATAGCTTTTCCTCGTATGAATAACTTAAGATTTTGGTTATTACCTTTTTCAATAATGTTGTTGTTAATGTCTTCTATAATTGAGACAGGAGTGGGGGCAGGGTGGACTATTTATCCGCCTTTAGCCGGGTTGGAGGGGCATGGAGGAGTAAGTATGGATTTAGCGATTTTTTCATTACACTTGGCTGGGGCTTCATCTATTATGGGGGCTATTAATTTTATTTGTACTATTTTAAACATACGAATAGAGGGAATGACTTTAGATAAGATTCCTTTGTTTGTTTGGTCAGTGCTTATTACTGCAGTCTTATTGTTATTGTCATTACCAGTATTGGCGGGGGCTATTACTATACTTTTGACTGATCGTAATTTTAATACATCTTTTTTTGATCCGGCAGGTGGAGGAGATCCTGTGTTGTTTCAACATTTATT",
    "sequenceLength": 655,
    "gcContent": 0.37251908396946565,
    "nonIupacFraction": 0.0,
    "nonACGTNFraction": 0.0,
    "nFraction": 0.0,
    "nRunsCapped": 0,
    "naturalLanguageDetected": false,
    "endsTrimmed": false,
    "gapsOrWhitespaceRemoved": false,
    "invalid": false
  }
]

The targetGene field is not produced by the sanitization pipeline. It is the interpreted value of the publisher-supplied target gene, normalised against the GBIF target gene vocabulary (API). All other fields are as described above.