dbn-go - Golang bindings to DBN

Golang tooling for Databento's APIs and DBN format

This repository contains Golang bindings to Databento's file format Databento Binary Encoding (DBN), Historical API, and Live API. It also includes tools to interact with these services.

• Library Usage
• Reading DBN Files
• Reading JSON Files
• Historical API
• Live API
• Tools
  • Installation

NOTE: This library is not affiliated with Databento. Please be careful with commands which incur billing. We are not responsible for any charges you incur.

Library Usage

To use this library, import the following package, optionally with Databento Historical or Live support:

import (
    dbn "github.com/NimbleMarkets/dbn-go"
    dbn_hist "github.com/NimbleMarkets/dbn-go/hist"
    dbn_live "github.com/NimbleMarkets/dbn-go/live"
)

Most dbn-go types and enums parallel Databento's libraries. Available messages types are:

• Mbp0Msg
• MboMsg
• Mbp1Msg
• CbboMsg
• Mbp10Msg
• OhlcvMsg
• ImbalanceMsg
• SymbolMappingMsg
• ErrorMsg
• SystemMsg
• StatMsg
• StatusMsg
• InstrumentDefMsg

Reading DBN Files

If you want to read a homogeneous array of DBN records from a file, use the dbn.ReadDBNToSlice generic function. The generic argument dicates which message type to read.

file, _ := os.Open("ohlcv-1s.dbn")
defer file.Close()
records, metadata, err := dbn.ReadDBNToSlice[dbn.OhlcvMsg](file)

Alternatively, you can use the DBNScanner to read records one-by-one. Each record can be handled directly, or automatically dispatched to the callback method of a struct that implements the dbn.Visitor interface.

dbnFile, _ := os.Open("ohlcv-1s.dbn")
defer dbnFile.Close()

dbnScanner := dbn.NewDbnScanner(dbnFile)
metadata, err := dbnScanner.Metadata()
if err != nil {
	return fmt.Errorf("scanner failed to read metadata: %w", err)
}
for dbnScanner.Next() {
    header := dbnScanner.GetLastHeader()
    fmt.Printf("rtype: %s  ts: %d\n", header.RType, header.TsEvent)

    // you could get the raw bytes and size:
    lastRawByteArray := dbnScanner.GetLastRecord()
    lastSize := dbnScanner.GetLastSize()

    // you probably want to use this generic helper to crack the message:
    ohlcv, err := dbn.DbnScannerDecode[dbn.OhlcvMsg](dbnScanner)

    // or if you are handing multiple message types, dispatch a Visitor:
    err = dbnScanner.Visit(visitor)
}
if err := dbnScanner.Error(); err != nil && err != io.EOF {
    return fmt.Errorf("scanner error: %w", err)
}

Reading JSON Files

If you already have DBN-based JSON text files, you can use the generic dbn.ReadJsonToSlice or dbn.JsonScanner to read them in as dbn-go structs. Similar to the raw DBN, you can handle records manually or use the dbn.Visitor interface.

jsonFile, _ := os.Open("ohlcv-1s.dbn.json")
defer jsonFile.Close()
records, metadata, err := dbn.ReadJsonToSlice[dbn.OhlcvMsg](jsonFile)

jsonFile, _ := os.Open("ohlcv-1s.dbn.json")
defer jsonFile.Close()
jsonScanner := dbn.NewJsonScanner(jsonFile)
for jsonScanner.Next() {
    if err := jsonScanner.Visit(visitor); err != nil {
        return fmt.Errorf("visitor err: %w", err)
    }
}
if err := jsonScanner.Error(); err != nil {
    fmt.Errorf("scanner err: %w", err)
}

Many of the dbn-go structs are annotated with json tags to facilitate JSON serialization and deserialization using json.Marshal and json.Unmarshal. That said, dbn-go uses valyala/fastjson and hand-written extraction code.

Historical API

Support for the Databento Historical API is available in the /hist folder. Every API method is a function that takes an API key and arguments and returns a response struct and error.

databentoApiKey := os.Getenv("DATABENTO_API_KEY")
schemas, err := dbn_hist.ListSchemas(databentoApiKey, "DBEQ.BASIC")

The source for dbn-go-hist illustrates using this dbn_hist module.

Live API

Support for the Databento Live API is available in the /live folder.

The general model is:

1. Use dbn_live.NewLiveClient to create a LiveClient based on a LiveConfig and connect to a DBN gateway
2. client.Authenticate with the gateway.
3. client.Subscribe to a stream using one or many SubscriptionRequestMsg.
4. Begin the stream with client.Start.
5. Read the stream using client.GetDbnScanner.

The source for dbn-go-live illustrates using this dbn_live module.

Tools

We include some tools to make our lives easier:

• dbn-go-tui
• dbn-go-hist
• dbn-go-live.

Tool Instalation

These tools are available as:

• Binaries from the dbn-go releases page

• Homebrew packages via:

  • brew install NimbleMarkets/homebrew-tap/dbn-go

• Docker multi-architecture images on GitHub's Container Registry at ghcr.io/nimblemarkets/dbn-go:

  • Hist query: docker run -e DATABENTO_API_KEY --rm ghcr.io/nimblemarkets/dbn-go:0.0.12 /usr/local/bin/dbn-go-hist datasets
  • Simple Live feed handler: docker run -e DATABENTO_API_KEY -v ${pwd}/dbn --rm ghcr.io/nimblemarkets/dbn-go:0.0.12 /usr/local/bin/dbn-go-live -d DBEQ.BASIC -s ohlcv-1h -o /dbn/foo.dbn -v -t QQQ SPY

• Built-from-source to the ./bin folder with the command task go-build (install Taskfile).

Open Collaboration

We welcome contributions and feedback. Please adhere to our Code of Conduct when engaging our community.

• GitHub Issues
• GitHub Pull Requests

License

Released under the Apache License, version 2.0, see LICENSE.txt.

Portions adapted from databento/dbn databendo/databento-rs under the same Apache license.

Copyright (c) 2024-2025 Neomantra Corp.

---

Made with ❤️ and 🔥 by the team behind Nimble.Markets.