can the message library be used for financial trading protocols (SBE)?

[rmanaloto-tastytrade]

I've been reviewing this library and I find it very intriguing as some of the concepts for initialization and message dispatching seem like modern iterations of the in house libraries we have now.

I was wondering if there have been any inquiries on using the message library for some of the binary trading protocols out there. For example, the simple binary encoding (sbe) protocol

I've looked through the code a bit and the examples provided but I have not found anything regarding handling of string-like fields. I see that there is a restriction for a field to have a size <= 64, but I'm not sure if I'm properly reading the code correctly.
Even though sbe is a binary protocol, there are some fields that are strings and they can have lengths of more than 64 characters, such as error message text descriptions

Also, are there any other public repositories where this library is being used? That might help me understand how to use the library better to see it in action (vs simple one-off examples). More specifically, I am looking for encoding/decoding messages off the wire. Our underlying storage/data will reside in a char* array buffer (aka tcp socket buffers). I don't see any (or I don't know where exactly to look) on how to do the decoding/encoding where the messages are pulled from a char* buffer.

Thanks in advance,
Ray

[elbeno]

We have no use cases for variable-length messages/fields, and no use cases for non-fundamental types (this is the 64-bit max). Hence no support for either at the moment - this library is designed for embedded use cases.

Dealing with underlying 8-bit storage is supported though - relevant tests start at https://github.com/intel/compile-time-init-build/blob/main/test/msg/message.cpp#L358, also see #446

[rmanaloto-tastytrade]

thanks for the quick response.

bummer though. Some of these binary trading protocols mostly have fixed field fundamental types and a fairly strict layout policy (especially market data). The indexed message handling/dispatch would definitely be very useful in the trading space. I tend to write the same thing over and over again and finding a clean way to somewhat generate that code and super optimized really piqued my interest when I started reading up on this library.

Out of curiosity, if a message only had 64-bit fundamental types or smaller with a known layout up front, reading and writing these messages from a socket buffer would work though, correct? Just thinking if I could just write an adapter to bit_cast the string data since the typical use case is to just copy those bytes over to pass through to another system (like the error messaged text i mentioned above).

[elbeno]

I expect something like that could be made to work, yes.

I would also note that the field implementation here is also designed for bits rather than bytes; it's possible that a byte-aligned protocol (which in my experience most are, outside of bare metal) could be handled more efficiently when reading/writing fields in a message. Then again it's possible that the optimizer generates good code for the 8-bit alignment case here. I haven't checked, again because bit handling is the far greater use case.

[rmanaloto-tastytrade]

ok thanks again for the replies

I guess I just wanted to see if i could use the indexed service for the message dispatching and then use just a plain old value type/struct for my business logic/message protocol handling. Most of these protocols have a header type at the beginning of the message with a field for the message type and another for the message length. So, the index service/callback functionality in your library is a pretty good use case for me.

Some of our internal messaging protocols are at the bit level also in order to efficiently pack data and lower message payload sizes. And the newer reflection-like tricks to convert aggregate types into tuples to count the number of fields in a struct seem to fall apart for bit-packed fields. So, I have been exploring a different approach which is very similar to your field implementation that basically has metadata about each field and their bit sizes and just compile-time auto-generate the payload sizes from that with getters/setters for each field via their bit offset.

I definitely like where this library is headed. I really do see a lot of use cases for this outside the embedded world. There have very similar parallels in the low-latency/trading/financial space.