GitBOM Representations of Build Tools and Build Tool configuration in tree

[edwarnicke]

Problem

Currently, GitBOM artifact trees have been represented solely in terms of the input artifact to a build tool at each level

flowchart BT
    c1[.c] --> o1[.o]
    h1.1[.h] --> o1[.o]
    h1.2[.h] --> o1[.o]
    c2[.c] --> o2[.o]
    h2.1[.h] --> o2[.o]
    h2.2[.h] --> o2[.o]
    o1 --> executable
    o2 --> executable

Loading

Where each .o file is produce by a build tool step like:

flowchart LR
    .c --> compiler[[compiler]]
    *.h --> compiler[[compiler]]
    compiler --> .o

Loading

and the executable is produced by a build tool step like:

flowchart LR
    *.o --> linker[[linker]]
    linker --> executable

Loading

It has been repeatedly pointed out that this loses two pieces of information that are important for many use cases:

1. Precisely what build tool was used for each build step
2. Precisely how that build tool was configured for each build step

This information is minimally important for:

1. Reproducible builds
2. Vulnerabilities that are introduced by the build tools (rather than by the input artifacts)

Unfortunately, the most naive solution, capturing the build tool command line, fails on a number of important counts:

1. It only provides the filename of the build tool, which is its location in a particular file system, rather than its actual identity
2. It includes 'ephemeral' build tool configuration information that is largely orthogonal to the actual output being produced, such as the -I flag that indicates include directories for C compilers. Where the include files are is immaterial, what they contain is important. Building in one users home directory vs another is a non-sequitur to the actual questions being asked in most cases, and would lead to pointless tree divergence.

Proposal for Build Tool Information Document

Encourage build tools to create a very simple build tool information document (BTID) of a form:

Name: ${human readable name for the tool}
Id: ${gitoid of the tool}
GitBOM: ${gitbom id of the tool - optional, omit if not available}

Where 'Name' is however the build tool canonically choses to refer to itself. It may (and probably should) include its version.

The BTID could be written out to the .bom/metadata/ subdirectory in a distinguished location.
The BTID could be treated as a sibling of the other inputs:

flowchart LR 
    input1 --> buildtool[build tool] --> output
    input2 --> buildtool[build tool]
    input3 --> buildtool[build tool]
    btid[BTID] --> buildtool[build tool]

Loading

Resulting in a tree like:

flowchart BT
    c1[.c] --> o1[.o]
    h1.1[.h] --> o1[.o]
    h1.2[.h] --> o1[.o]
    compilerbtid1[Compiler BTID] --> o1[.o]
    c2[.c] --> o2[.o]
    h2.1[.h] --> o2[.o]
    h2.2[.h] --> o2[.o]
    compilerbtid2[Compiler BTID] --> o2[.o]
    o1 --> executable
    o2 --> executable
    linkerbtid[Linker BTID] --> executable

Loading

Note: This does not graft the build tool's artifact tree into the artifact tree of the output artifact (which would be improper) but merely adds the use of the build tool into the artifact tree.

Proposal for Build Tool Configuration Information

Build tool configuration will vary from build tool to build tool. Further, that build tool configuration which is non-ephemeral will be a subset (usually a proper subset) of that information. Each build tool will have to provide for itself a choice of configuration format and make its own decisions about what matters or doesn't matter.

For example, a C compiler almost certainly does not want to have 'include directories' in its build tool configuration document, but almost certainly does want to include any preprocessor flags provided to the build tool by the command line.

This proposal is to provide guidance to build tools about how to construct build tool configuration documents (BTCDs) and direction that build tools should:

1. Write the BTCD to .bom/metadata/
2. Include the BTCD into the artifact tree as siblings to the input

flowchart BT
    c1[.c] --> o1[.o]
    h1.1[.h] --> o1[.o]
    h1.2[.h] --> o1[.o]
    compilerbtid1[Compiler BTID] --> o1[.o]
    compilerbtcd1[BTCD1] --> o1[.o]
    c2[.c] --> o2[.o]
    h2.1[.h] --> o2[.o]
    h2.2[.h] --> o2[.o]
    compilerbtid2[Compiler BTID] --> o2[.o]
    compilerbtcd1[BTCD2] --> o1[.o]
    o1 --> executable
    o2 --> executable
    linkerbtid[Linker BTID] --> executable
    linkerbtcd[BTCD3] --> executable

Loading

[jsgf]

It has been repeatedly pointed out that this loses two pieces of information that are important for many use cases:

Precisely what build tool was used for each build step
Precisely how that build tool was configured for each build step

I think it's worse than that - it loses how many times a build tool was invoked at all. Since it only records the inputs, all tool invocations with the same inputs will look identical:

cc -c foo.c # uses foo.h
cat foo.[ch] > codeblob
sha1sum foo.[ch] > sum.txt

will all have the same GitBOM document, with only the derived artifact reference distinguishing them.

You could disambiguate this by (effectively) including the build tool as an input - either explicitly (ie hash the compiler binaries and include that OID as an input), or abstracted via some kind of tool identifier.

And while it's nice to know what the tool was, I think it's much more important to know what the outputs are. So I'd propose extending the GitBOM document to not only reference the inputs, but also explicitly enumerate all the outputs. And correspondingly change the definitions of inputs to reference a specific output of a specific bomdoc (which could be done with the existing reference, with the constraint that the listed derived artifact oid has to be a listed output of the referenced GitBOM document).

The BTID could be treated as a sibling of the other inputs: [...]
Note: This does not graft the build tool's artifact tree into the artifact tree of the output artifact (which would be improper) but merely adds the use of the build tool into the artifact tree.

I'm not sure what this means. Do you mean it will affect the OID of the GitBOM document? If the BTID contains an OID or a GitBOM manifest, and it is hashed into the referencing OID, then doesn't that effectively mean you are grafting the artifact tree in?

And if it doesn't get hashed into the GitBOM OID, then what effect will it have?

[dpp]

I'm all for tools emitting as much information as possible when doing their "thing".

I think it'd be useful to:

1 - Choose a standard file format for this data (e.g. YAML or JSON)
2 - Have standard fields as well as extension fields (like HTTP headers) for the tools to use

[fkautz]

Bumping this, what I think we are missing is a temporal ordering of the tools called. However, I don't think OmniBOR should be the thing that necessarily solves this. We could integrate something like in-toto to solve this issue. In-toto with OmniBOR support can give us input => output, order of what was called, and help link this metadata.

[jsgf]

@fkautz

I think we are missing is a temporal ordering of the tools called

I'm curious what that's useful for?

For a concurrent build system it may not be possible to put everything into a total order, only a partial order where write output -> consume input is a "happens before".

[alilleybrinker]

One thought I keep coming back to in these discussions of encoding additional data in an OmniBOR manifest is: can't we instead lean on other attestation formats which can refer to OmniBOR Artifact Identifiers?

If you want to know the build process, including configuration of the build tool, which produced some artifact, you could in theory produce a SLSA attestation which refers to the relevant OmniBOR data.

Isolating things in this way would help keep OmniBOR focused on solving the identity problem of software artifacts.