Skip to content

Commit

Permalink
Document MAP and ARRAY flat layout in the type classes
Browse files Browse the repository at this point in the history
  • Loading branch information
dain committed Sep 1, 2023
1 parent 1e02592 commit 60aac35
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 0 deletions.
22 changes: 22 additions & 0 deletions core/trino-spi/src/main/java/io/trino/spi/type/ArrayType.java
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,28 @@ public void writeObject(BlockBuilder blockBuilder, Object value)
});
}

// FLAT MEMORY LAYOUT
//
// All data of the array is stored in the variable width section. Within the variable width section,
// fixed data for all elements is stored first, followed by variable length data for all elements
// This simplifies the read implementation as we can simply step through the fixed section without
// knowing the variable length of each element, since each element stores the offset to its variable
// length data inside its fixed length data.
//
// In the current implementation, the element and null flag are stored in an interleaved flat record.
// This layout is not required by the format, and could be changed to a columnar if it is determined
// to be more efficient.
//
// Fixed:
// int positionCount, int variableSizeOffset
// Variable:
// byte element1Null, elementFixedSize element1FixedData
// byte element2Null, elementFixedSize element2FixedData
// ...
// element1VariableSize element1VariableData
// element2VariableSize element2VariableData
// ...

@Override
public int getFlatFixedSize()
{
Expand Down
23 changes: 23 additions & 0 deletions core/trino-spi/src/main/java/io/trino/spi/type/MapType.java
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,29 @@ public void writeObject(BlockBuilder blockBuilder, Object value)
});
}

// FLAT MEMORY LAYOUT
//
// All data of the map is stored in the variable width section. Within the variable width section,
// fixed data for all keys and values are stored first, followed by variable length data for all keys
// and values. This simplifies the read implementation as we can simply step through the fixed
// section without knowing the variable length of each value, since each value stores the offset
// to its variable length data inside its fixed length data.
//
// In the current implementation, the keys and values are stored in an interleaved flat record along
// with null flags. This layout is not required by the format, and could be changed to a columnar
// if it is determined to be more efficient. Additionally, this layout allows for a null key, since
// non-null keys is not always enforced, and null keys may be allowed in the future.
//
// Fixed:
// int positionCount, int variableSizeOffset
// Variable:
// byte key1Null, keyFixedSize key1FixedData, byte value1Null, valueFixedSize value1FixedData
// byte key2Null, keyFixedSize key2FixedData, byte value2Null, valueFixedSize value2FixedData
// ...
// key1VariableSize key1VariableData, value1VariableSize value1VariableData
// key2VariableSize key2VariableData, value2VariableSize value2VariableData
// ...

@Override
public int getFlatFixedSize()
{
Expand Down

0 comments on commit 60aac35

Please sign in to comment.