Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Further improvements in runtime metadata loading #1524

Merged
merged 4 commits into from
Nov 16, 2023
Merged

Further improvements in runtime metadata loading #1524

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6035796
Use zero copy for even faster json deserialization
mjeffryes Nov 14, 2023
b553dca
Generate a condensed metadata for runtime
mjeffryes Nov 15, 2023
34fc389
Add tests and flag
mjeffryes Nov 15, 2023
6a98a47
feedback
mjeffryes Nov 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ require (
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
github.com/segmentio/asm v1.1.3 // indirect
github.com/segmentio/encoding v0.3.5 // indirect
github.com/segmentio/encoding v0.3.5
github.com/sergi/go-diff v1.3.1 // indirect
github.com/skeema/knownhosts v1.2.0 // indirect
github.com/spf13/cast v1.5.0 // indirect
Expand Down
6 changes: 6 additions & 0 deletions pkg/tfbridge/info.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,12 @@ type ProviderInfo struct {
//
// See https://github.com/pulumi/pulumi-terraform-bridge/issues/1501
XSkipDetailedDiffForChanges bool

// Enables generation of a trimmed, runtime-only metadata file
// to help reduce resource plugin start time
//
// See also pulumi/pulumi-terraform-bridge#1524
GenerateRuntimeMetadata bool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why a provider would not want this flag enabled?

Put another way, if we were 100% confident that this would work as intended, would we enable it for all of our providers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not a lot of value for smaller providers, the data in the runtime config is strictly redundant with bridge-metadata.json (while being harder to read/edite) and it requires changing up the init a bit to use the runtime file for the runtime binary. So for now, I don't think it's worthwhile to try to enable this everywhere, lots of little frictions for minimal benefit on most providers.

We might revisit this as part of a larger effort to distinguish gentime/runtime though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense to me.

}

// Send logs or status logs to the user.
Expand Down
9 changes: 9 additions & 0 deletions pkg/tfbridge/metadata.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,12 @@ func (info *MetadataInfo) assertValid() {
contract.Assertf(info.Path != "", "Path must be non-empty")

}

// trim the metadata to just the keys required for the runtime phase
// in the future this method might also substitute compressed contents within some keys
func (info *MetadataInfo) ExtractRuntimeMetadata() *MetadataInfo {
data, _ := metadata.New(nil)
metadata.CloneKey("auto-aliasing", info.Data, data)
mjeffryes marked this conversation as resolved.
Show resolved Hide resolved
metadata.CloneKey("mux", info.Data, data)
return &MetadataInfo{"runtime-bridge-metadata.json", ProviderMetadata(data)}
}
34 changes: 34 additions & 0 deletions pkg/tfbridge/metadata_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
package tfbridge

import (
"testing"

md "github.com/pulumi/pulumi-terraform-bridge/v3/unstable/metadata"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

func TestMetadataInfo(t *testing.T) {
data, err := md.New(nil)
require.NoError(t, err)

err = md.Set(data, "hi", []string{"hello", "world"})
require.NoError(t, err)
err = md.Set(data, "auto-aliasing", []string{"1", "2"})
require.NoError(t, err)
err = md.Set(data, "mux", []string{"a", "b"})
require.NoError(t, err)

marshalled := data.Marshal()
require.Equal(t, `{"auto-aliasing":["1","2"],"hi":["hello","world"],"mux":["a","b"]}`, string(marshalled))

info := NewProviderMetadata(marshalled)
assert.Equal(t, "bridge-metadata.json", info.Path)
marshalledInfo := (*md.Data)(info.Data).Marshal()
assert.Equal(t, `{"auto-aliasing":["1","2"],"hi":["hello","world"],"mux":["a","b"]}`, string(marshalledInfo))

runtimeMetadata := info.ExtractRuntimeMetadata()
assert.Equal(t, "runtime-bridge-metadata.json", runtimeMetadata.Path)
runtimeMarshalled := (*md.Data)(runtimeMetadata.Data).Marshal()
assert.Equal(t, `{"auto-aliasing":["1","2"],"mux":["a","b"]}`, string(runtimeMarshalled))
}
24 changes: 12 additions & 12 deletions pkg/tfbridge/tokens_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -654,7 +654,7 @@ func TestMaxItemsOneAliasing(t *testing.T) {
// Save current state into metadata
autoAliasing(info, metadata)

v := string(metadata.Marshal())
v := string(metadata.MarshalIndent())
expected := `{
"auto-aliasing": {
"resources": {
Expand All @@ -681,14 +681,14 @@ func TestMaxItemsOneAliasing(t *testing.T) {

assert.True(t, *info.Resources["pkg_r1"].Fields["f1"].MaxItemsOne)
assert.False(t, *info.Resources["pkg_r1"].Fields["f2"].MaxItemsOne)
assert.Equal(t, expected, string(metadata.Marshal()))
assert.Equal(t, expected, string(metadata.MarshalIndent()))

// Apply metadata back into the provider again, making sure there isn't a diff
autoAliasing(info, metadata)

assert.True(t, *info.Resources["pkg_r1"].Fields["f1"].MaxItemsOne)
assert.False(t, *info.Resources["pkg_r1"].Fields["f2"].MaxItemsOne)
assert.Equal(t, expected, string(metadata.Marshal()))
assert.Equal(t, expected, string(metadata.MarshalIndent()))

// Validate that overrides work

Expand Down Expand Up @@ -716,7 +716,7 @@ func TestMaxItemsOneAliasing(t *testing.T) {
}
}
}
}`, string(metadata.Marshal()))
}`, string(metadata.MarshalIndent()))
}

func TestMaxItemsOneAliasingExpiring(t *testing.T) {
Expand All @@ -741,7 +741,7 @@ func TestMaxItemsOneAliasingExpiring(t *testing.T) {
// Save current state into metadata
autoAliasing(info, metadata)

v := string(metadata.Marshal())
v := string(metadata.MarshalIndent())
expected := `{
"auto-aliasing": {
"resources": {
Expand Down Expand Up @@ -786,7 +786,7 @@ func TestMaxItemsOneAliasingExpiring(t *testing.T) {
}
}
}
}`, string(metadata.Marshal()))
}`, string(metadata.MarshalIndent()))

}

Expand Down Expand Up @@ -817,7 +817,7 @@ func TestMaxItemsOneAliasingNested(t *testing.T) {
// Save current state into metadata
autoAliasing(info, metadata)

v := string(metadata.Marshal())
v := string(metadata.MarshalIndent())
expected := `{
"auto-aliasing": {
"resources": {
Expand Down Expand Up @@ -851,7 +851,7 @@ func TestMaxItemsOneAliasingNested(t *testing.T) {
info = provider(false, true)
autoAliasing(info, metadata)

assert.Equal(t, expected, string(metadata.Marshal()))
assert.Equal(t, expected, string(metadata.MarshalIndent()))
assert.True(t, *info.Resources["pkg_r1"].Fields["f2"].Elem.Fields["n1"].MaxItemsOne)
assert.False(t, *info.Resources["pkg_r1"].Fields["f2"].Elem.Fields["n2"].MaxItemsOne)
}
Expand Down Expand Up @@ -919,7 +919,7 @@ func TestMaxItemsOneAliasingWithAutoNaming(t *testing.T) {
}
}
}
}`, string((*md.Data)(p.MetadataInfo.Data).Marshal()))
}`, string((*md.Data)(p.MetadataInfo.Data).MarshalIndent()))
}

t.Run("auto-named-then-aliased", func(t *testing.T) {
Expand Down Expand Up @@ -1000,7 +1000,7 @@ func TestMaxItemsOneDataSourceAliasing(t *testing.T) {
}
}
}
}`, string((*md.Data)(p.MetadataInfo.Data).Marshal()))
}`, string((*md.Data)(p.MetadataInfo.Data).MarshalIndent()))
}

t.Run("auto-named-then-aliased", func(t *testing.T) {
Expand Down Expand Up @@ -1120,15 +1120,15 @@ func TestAutoAliasingChangeDataSources(t *testing.T) {
return func(t *testing.T) {
p := provider(t, current, name)
require.JSONEq(t, expected,
string((*md.Data)(p.MetadataInfo.Data).Marshal()))
string((*md.Data)(p.MetadataInfo.Data).MarshalIndent()))

// Regardless of the input and output, once we apply some name to
// our state, reapplying the same name to the new state should be
// idempotent.
t.Run("idempotent", func(t *testing.T) {
p := provider(t, expected, name)
require.JSONEq(t, expected,
string((*md.Data)(p.MetadataInfo.Data).Marshal()))
string((*md.Data)(p.MetadataInfo.Data).MarshalIndent()))
})
}
}
Expand Down
16 changes: 8 additions & 8 deletions pkg/tfbridge/x/token_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,7 +531,7 @@ func TestMaxItemsOneAliasing(t *testing.T) {
err = AutoAliasing(info, metadata)
require.NoError(t, err)

v := string(metadata.Marshal())
v := string(metadata.MarshalIndent())
expected := `{
"auto-aliasing": {
"resources": {
Expand Down Expand Up @@ -559,15 +559,15 @@ func TestMaxItemsOneAliasing(t *testing.T) {

assert.True(t, *info.Resources["pkg_r1"].Fields["f1"].MaxItemsOne)
assert.False(t, *info.Resources["pkg_r1"].Fields["f2"].MaxItemsOne)
assert.Equal(t, expected, string(metadata.Marshal()))
assert.Equal(t, expected, string(metadata.MarshalIndent()))

// Apply metadata back into the provider again, making sure there isn't a diff
err = AutoAliasing(info, metadata)
require.NoError(t, err)

assert.True(t, *info.Resources["pkg_r1"].Fields["f1"].MaxItemsOne)
assert.False(t, *info.Resources["pkg_r1"].Fields["f2"].MaxItemsOne)
assert.Equal(t, expected, string(metadata.Marshal()))
assert.Equal(t, expected, string(metadata.MarshalIndent()))

// Validate that overrides work

Expand Down Expand Up @@ -596,7 +596,7 @@ func TestMaxItemsOneAliasing(t *testing.T) {
}
}
}
}`, string(metadata.Marshal()))
}`, string(metadata.MarshalIndent()))
}

func TestMaxItemsOneAliasingExpiring(t *testing.T) {
Expand All @@ -623,7 +623,7 @@ func TestMaxItemsOneAliasingExpiring(t *testing.T) {
err = AutoAliasing(info, metadata)
require.NoError(t, err)

v := string(metadata.Marshal())
v := string(metadata.MarshalIndent())
expected := `{
"auto-aliasing": {
"resources": {
Expand Down Expand Up @@ -669,7 +669,7 @@ func TestMaxItemsOneAliasingExpiring(t *testing.T) {
}
}
}
}`, string(metadata.Marshal()))
}`, string(metadata.MarshalIndent()))

}

Expand Down Expand Up @@ -702,7 +702,7 @@ func TestMaxItemsOneAliasingNested(t *testing.T) {
err = AutoAliasing(info, metadata)
require.NoError(t, err)

v := string(metadata.Marshal())
v := string(metadata.MarshalIndent())
expected := `{
"auto-aliasing": {
"resources": {
Expand Down Expand Up @@ -737,7 +737,7 @@ func TestMaxItemsOneAliasingNested(t *testing.T) {
err = AutoAliasing(info, metadata)
require.NoError(t, err)

assert.Equal(t, expected, string(metadata.Marshal()))
assert.Equal(t, expected, string(metadata.MarshalIndent()))
assert.True(t, *info.Resources["pkg_r1"].Fields["f2"].Elem.Fields["n1"].MaxItemsOne)
assert.False(t, *info.Resources["pkg_r1"].Fields["f2"].Elem.Fields["n2"].MaxItemsOne)
}
Expand Down
6 changes: 5 additions & 1 deletion pkg/tfgen/generate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -946,7 +946,11 @@ func (g *Generator) UnstableGenerateFromSchema(genSchemaResult *GenerateSchemaRe
files = map[string][]byte{"schema.json": bytes}

if info := g.info.MetadataInfo; info != nil {
files[info.Path] = (*metadata.Data)(info.Data).Marshal()
files[info.Path] = (*metadata.Data)(info.Data).MarshalIndent()
if g.info.GenerateRuntimeMetadata {
runtimeInfo := info.ExtractRuntimeMetadata()
files[runtimeInfo.Path] = (*metadata.Data)(runtimeInfo.Data).Marshal()
}
}
case PCL:
if g.skipExamples {
Expand Down
73 changes: 48 additions & 25 deletions unstable/metadata/metadata.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,31 +15,44 @@
package metadata

import (
"encoding/json"
"github.com/json-iterator/go"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/segmentio/encoding/json"
)

// The underlying value of a metadata blob.
type Data struct{ m map[string]*json.RawMessage }
type Data struct{ m map[string]json.RawMessage }

func New(data []byte) (*Data, error) {
m := map[string]*json.RawMessage{}
m := map[string]json.RawMessage{}
if len(data) > 0 {
jsoni := jsoniter.ConfigCompatibleWithStandardLibrary
err := jsoni.Unmarshal(data, &m)
_, err := json.Parse(data, &m, json.ZeroCopy)
if err != nil {
return nil, err
}
}

return &Data{m}, nil
}

func (d *Data) Marshal() []byte {
return d.marshal(false)
}

func (d *Data) MarshalIndent() []byte {
return d.marshal(true)
}

func (d *Data) marshal(indent bool) []byte {
if d == nil {
d = &Data{m: make(map[string]*json.RawMessage)}
d = &Data{m: make(map[string]json.RawMessage)}
}
var bytes []byte
var err error
if indent {
bytes, err = json.MarshalIndent(d.m, "", " ")
} else {
bytes, err = json.Marshal(d.m)
}
bytes, err := json.MarshalIndent(d.m, "", " ")
// `d.m` is a `map[string]json.RawMessage`. `json.MarshalIndent` errors only when
// it is asked to serialize an unmarshalable type (complex, function or channel)
// or a cyclic data structure. Because `string` and `json.RawMessage` are
Expand All @@ -59,13 +72,12 @@ func Set(d *Data, key string, value any) error {
delete(d.m, key)
return nil
}
jsoni := jsoniter.ConfigCompatibleWithStandardLibrary
data, err := jsoni.Marshal(value)
data, err := json.Marshal(value)
if err != nil {
return err
}
msg := json.RawMessage(data)
d.m[key] = &msg
d.m[key] = msg
return nil
}

Expand All @@ -75,29 +87,40 @@ func Get[T any](d *Data, key string) (T, bool, error) {
if !ok {
return t, false, nil
}
jsoni := jsoniter.ConfigCompatibleWithStandardLibrary
err := jsoni.Unmarshal(*data, &t)
_, err := json.Parse(data, &t, json.ZeroCopy)
return t, true, err
}

func CloneKey(key string, from, to *Data) {
data, ok := from.m[key]
if !ok {
delete(to.m, key)
return
}
to.m[key] = cloneRawMessage(data)
}

func Clone(data *Data) *Data {
if data == nil {
return nil
}
m := make(map[string]*json.RawMessage, len(data.m))
m := make(map[string]json.RawMessage, len(data.m))
for k, v := range data.m {
dst := make(json.RawMessage, len(*v))
n := copy(dst, *v)
// According to the documentation for `copy`:
//
// Copy returns the number of elements copied, which will be the minimum
// of len(src) and len(dst).
//
// Since `len(src)` is `len(dst)`, and `copy` cannot copy more bytes the
// its source, we know that `n == len(v)`.
contract.Assertf(n == len(*v), "failed to perform full copy")
m[k] = &dst
m[k] = cloneRawMessage(v)
}
return &Data{m}
}

func cloneRawMessage(m json.RawMessage) json.RawMessage {
dst := make(json.RawMessage, len(m))
n := copy(dst, m)
// According to the documentation for `copy`:
//
// Copy returns the number of elements copied, which will be the minimum
// of len(src) and len(dst).
//
// Since `len(src)` is `len(dst)`, and `copy` cannot copy more bytes the
// its source, we know that `n == len(v)`.
contract.Assertf(n == len(m), "failed to perform full copy")
return dst
}
Loading