-
Notifications
You must be signed in to change notification settings - Fork 738
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor FPD Merge Functions #3601
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't find the benchmark in the contributed code. Should we include them?
) | ||
|
||
err := MergeClone(imp, []byte(`{"banner":nul}`)) | ||
require.EqualError(t, err, "cannot unmarshal openrtb2.Imp.Banner: expect ull") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it read expect null
instead of expect ull
? Does tryExtractErrorMessage
have an off-one bug when building the error message string?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not a bug in this PR. That is the error message from json-iter. This test is specifically targeting the failure branches of iter.ReadNil
. I'll add a comment to clarify.
|
||
err := MergeClone(imp, []byte(`{"banner":malformed}`)) | ||
require.EqualError(t, err, "cannot unmarshal openrtb2.Imp.Banner: expect { or n, but found m") | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we add this test case, it'll pass. Do we want MergeClone
to have this behavior with empty non null
arrays? In order to save memory, should we modify in order to get a nil
imp.IframeBuster
instead?
t.Run("nil-existing-empty-incoming", func(t *testing.T) {
var (
imp = &openrtb2.Imp{}
)
err := MergeClone(imp, []byte(`{"iframeBuster":[]}`))
require.NoError(t, err)
assert.Equal(t, []string{}, imp.IframeBuster, "new-val")
})
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No. It is proper to parse an empty json array to an empty slice. There is sometimes a difference between a null and empty array.
// token, so must be handled in this decoder. | ||
if iter.ReadNil() { | ||
*(*unsafe.Pointer)(ptr) = nil | ||
d.mapType.UnsafeSet(ptr, d.mapType.UnsafeNew()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this call initializing ptr
to a new empty map? If so, slices and pointers in lines 80 and 104 are simply set to nil
. Should slices in line 104 be initialized too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not fully understand why setting the pointer to nil
isn't sufficient for maps. This is copy I copied/pasted from json-iter, see here. Perhaps it's an incorrect assumption, but I assume this is done for a reason.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Replaces the FPD merge methods with a smarter implementation.
With the goal of parsing the FPD overlay json only once, the exisitng algorithm cloned the entire app / site / user object before unmarshalling the overlay. A clone is necessary because we share object references between bidder requests to save on allocations, so we need to make a clone to a pointer field before we make a bidder specific change.
Instead of cloning the entire object up front, we can save on allocations and processing time in most cases by tying into the unmarshal process and making a clone as needed. This new approach (now possible with the move to json-iter) also builds ext merging into the unmarshal process and eliminates the need for the extMerger helper.
Benchmarks:
This new algorithm uses less cpu time and allocates less in almost all cases, except for the worst case. I wouldn't expect the worst case to ever occur, but sharing the benchmark for full transparency. The new algorithm really shines for the medium / average cases.