move csv2bufr templates to separate repo

[tomkralidis]

Putting csv2bfur templates outside csv2bufr proper will alleviate the need to versions/publishing. As discussed, we should move csv2bufr templates back to https://github.com/wmo-im/csv2bufr-templates and have wis2box ref that setup again.

[david-i-berry]

Related PR in csv2bufr repo: wmo-im/csv2bufr#125

[david-i-berry]

To do:

• Update csv2bufr docs on installation of csv2bufr templates
• Integration testing with wis2box stack

[tomkralidis]

2024-01-30

• docs/testing for next week

[david-i-berry]

Changes tested with wis2box-api branch (https://github.com/wmo-im/wis2box-api/tree/wis2box-1.0b7).

Docs updated for csv2bufr.

WIS2box docs need updating (https://docs.wis2box.wis.wmo.int/en/1.0b6/reference/running/data-pipeline-plugins.html#wis2box-data-csv2bufr-observationdatacsv2bufr):

(@maaikelimper for info)

Add subsection on environment variables:

csv2bufr uses three environment variables, two to specify the originating centre and sub-centres for the BUFR message and one to specify the search path for the csv2bufr templates:

BUFR_ORIGINATING_CENTRE=65535 # defaults missing value
BUFR_ORIGINATING_SUBCENTRE=65535 # default missing value
CSV2BUFR_TEMPLATES=/custom/path/to/templates # ./ and /opt/csv2bufr/templates searched by default

Sample cs2vbufr templates can be downloaded from https://github.com/wmo-im/csv2bufr-templates and installed in /opt/csv2bufr/templates for use with csv2bufr, e.g.

mkdir /opt/csv2bufr # only required if directory does not exist
cd /opt/csv2bufr
wget https://github.com/wmo-im/csv2bufr-templates/archive/refs/tags/v0.1.tar.gz # download release v0.1 of templates
tar -zxf v0.1.tar.gz --strip-components=1 csv2bufr-templates-0.1/templates # extract the templates into the search path

[david-i-berry]

Linked to #536

[tomkralidis]

2024-02-06:

• @maaikelimper to provide doc update PR based on move csv2bufr templates to separate repo #546 (comment)
• backwards compatibility testing needed (@david-i-berry)

[tomkralidis]

2024-02-13:
@david-i-berry

• template ids are uuids
• use human readable strings back in data mappings
• update filenames for clarity

@maaikelimper

• API: add template lookup to API for UI workflow

[tomkralidis]

2024-03-05:

• fixed in API, will be fixed once API updates are merged/integrated to stack