feat: OpenTelemetry [DO NOT REVIEW]

[nasdf]

Relevant issue(s)

Resolves #293
Resolves #74

Description

This PR adds OpenTelemetry metrics and tracing.

Tasks

• I made sure the code is well commented, particularly hard-to-understand areas.
• I made sure the repository-held documentation is changed accordingly.
• I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
• I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

Manually testing with Jaeger

Specify the platform(s) on which this was tested:

• MacOS

[AndrewSisley]

thought: I think I agree with John's suggestion of extracting this out to a thin package so we can avoid scattering references throughout the code base to the same 3rd party package.

If we ever swap it out we'll probably still need to change the func signatures (I wouldn't put much thought into designing them), but at least everything will be in the same place, and it makes adding/standardising any middleware much easier.

[AndrewSisley]

suggestion: I do not know about performance/C-GO, but we can probably avoid having to manually specify "AddPolicy" by using the runtime package and stuff like runtime.CallersFrames instead.

[nasdf]

I can look into that.

[codecov]

Codecov Report

Attention: Patch coverage is 62.87129% with 75 lines in your changes missing coverage. Please review.

Project coverage is 78.23%. Comparing base (f24fda4) to head (5eef3d7).
Report is 3 commits behind head on develop.

Files with missing lines	Patch %	Lines
cli/telemetry.go	0.00%	50 Missing ⚠️
internal/metric/metric.go	44.44%	9 Missing and 6 partials ⚠️
cli/start.go	0.00%	4 Missing ⚠️
internal/db/collection.go	85.71%	3 Missing ⚠️
internal/db/collection_delete.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3404      +/-   ##
===========================================
+ Coverage    78.11%   78.23%   +0.12%     
===========================================
  Files          392      394       +2     
  Lines        36045    36209     +164     
===========================================
+ Hits         28155    28326     +171     
- Misses        6208     6218      +10     
+ Partials      1682     1665      -17     

Flag	Coverage Δ
all-tests	78.23% <62.87%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/db/collection_get.go	85.25% <100.00%> (+11.11%)	⬆️
internal/db/collection_index.go	87.79% <100.00%> (+0.29%)	⬆️
internal/db/collection_update.go	74.80% <100.00%> (+0.61%)	⬆️
internal/db/db.go	67.98% <100.00%> (+1.18%)	⬆️
internal/db/p2p_replicator.go	62.33% <100.00%> (+1.30%)	⬆️
internal/db/p2p_schema_root.go	80.92% <100.00%> (-0.78%)	⬇️
internal/db/store.go	73.89% <100.00%> (+6.49%)	⬆️
internal/metric/otel.go	100.00% <100.00%> (ø)
internal/db/collection.go	70.84% <85.71%> (+1.44%)	⬆️
internal/db/collection_delete.go	39.53% <0.00%> (-0.94%)	⬇️
... and 3 more

... and 16 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f24fda4...5eef3d7. Read the comment docs.