Question about GbifID

[niconoe]

Hi, I've posted the following message yesterday to the pipelines mailing list, but it doesn't appear really used anymore, so I thinkI might have a better chance here. Apologies if this is not the right place, don't hesitate to redirect.

Context:

We are currently developing an Early Alert system for invasive species
where all data transits through GBIF:

• one one side, we are helping multiple data providers to push their
  observations to GBIF at a high frequency (probably daily)
• on the other side, we have a web application
  https://github.com/riparias/early-warning-webapp that allows users to be
  alerted of specific species in specific locations. To do so, that web
  application refreshes its internal database daily, by dropping and
  recreating all occurrences based on an automated GBIF data download*. Data
  in the gbifId field is used as a long-term occurrence identifier in this
  web application.

It is crucial for our system that we can unambiguously distinguish new
occurrences from existing occurrences that were recently republished (and
maybe updated), and also detect occurrences that have been updated in the
source dataset.

Finally, the question:

in the GBIF infrastructure, when exactly is a new occurrence Gbif ID issued
(vs reused). Is it just a matter of following the publisher's occurrenceID,
or is the algorithm that chooses Gbif IDs more complex?

Or, more practically: what should I say to our data providers to guarantee
all new occurrences they publish get a new gbifId and existing occurrences
that get republished keep the same gbifId?

I'd be also interested to know if you think the concept of using GBIF ids
as stable identifiers in an external tool is fundamentally flawed (but I
hope not!)

[muttcg]

HI @niconoe, apologies for the waiting.

There are 2 main variations gbifIDs for DWCA:

1) DwcTerm.occurrenceID and triplet (a combination of DwcTerm.institutionCode, DwcTerm.collectionCode and DwcTerm.catalogNumber).

If the dataset contains unique occurrenceIDs and unique triplets data, we create pairs in key-value database:

datasetKey(registry.gbif.org) + occurrenceID -> gbifID
datasetKey(registry.gbif.org) + triplet (institutionCode| collectionCode|catalogNumber) -> the same gbifID

If data provider changes one of 4 fields we get a collision and drop that records from the index until the provider fix data or we break that pair and tell what to use as the main key (occurrenceID, triplet or generate new id)

2) DwcTerm.occurrenceID only.

If the dataset contains only unique occurrenceIDs.

datasetKey(registry.gbif.org) + occurrenceID -> gbifID

No extra checks, new occurrenceID equal new gbifID

gbifIDs consistency is very important for us and we try to keep it stable, but of course, sometimes we get collisions and new IDs when publishers change their data (popular issue use URL as an occurrenceID and replace http://url to https://url)

[niconoe]

Thanks a lot @muttcg, great explanation!

Just to be sure I got everything correctly, can you confirm me that for my use case, to be sure I can track down "which occurrences is which" in my webapp that consumes GBIF data, I should either:

1. ask my data publishers to never touch institutionCode, collectionCode and catalogNumber after initial publishing (so I can rely on gbifId and assume it'll only change if the occurrenceID change.
2. ask my data publishers to not populate institutionCode, collectionCode nor catalogNumber (so we are in your case 2), and I can also assume that in the dataset, only occurrenceID will affect the resulting gbifID
3. Take a different approach and make my app not reliant on gbifID, but rather compute my own hash of datasetKey+occurrenceID and use that value as a long term identifier. That way I am immune to changes to institutionCode, collectionCode and catalogNumber and I can pass a simpler message to my data providers ("just pay attention to the consistency of occurrenceID"). I'm tempted to go that route, since that would also allow me without too much problems to show data in my web app from other providers (I don't have control on those)

Is my understanding correct?

Thanks a lot for explaining the internals already, that's super helpful to build things on top of GBIF!

[muttcg]

Hi!
Everything is correct, I will only add extra comments for each point:

1. We use triplet only if there are no duplicates in the whole dataset, we do dataset validation before ingestion
2. It is ok and even if they add triplet in the future we link that data to existing occurrenceID->gbifID pair, of course, if they change triplet or occurrenceID in the future it will cause id collisions and records with collisions will be discarded
3. That can give you flexibility and control

I'd say usage of occurrenceID only is stable and predictable, we didn't have any issues with that, only related to changes in data. The couple occurrenceID+triplet was introduced mostly for non-dwca data standards like abcd/biocase/etc, at some point, people started to migrate data from non-dwca to dwca standards and we wanted to keep the same gbifIDs.

[niconoe]

Thanks a lot for the time you put in this, very helpful and appreciated!