Setting state inside gesture handler is causing significant slow down

[MicahDavid]

I'm migrating from version 1 to 2. The new v2 gesture handlers are working as expected, but I"m seeing a major slow down. This is occurring as I"m setting state variables on the pan gesture.

Here's the pan Gesture

    const panGesture = Gesture.Pan()
        .runOnJS(true)
        .onUpdate((e) => {
            const offsetY = e.translationY
            if (scrollIsTop && offsetY > 0) {
                setHeaderMarginTop(initialHeaderMarginTop + offsetY )
                setContentMarginTop(initialContentMarginTop + offsetY )
            }
        })
        .simultaneousWithExternalGesture(scrollRef)

I'm detecting the pan gesture on the ScrollView, and setting margin top values on the SwipeUpDownModal:

        <SwipeUpDownModal
            modalVisible={visible}
            PressToanimate={animateModal}
            HeaderStyle={{marginTop:headerMarginTop}}
            ContentModalStyle={{marginTop: contentMarginTop}}
            ContentModal={   
              <View>           
                    <GestureDetector gesture={panGesture}>
                        <ScrollView
                            ref={scrollRef}
                            style={styles.modalContent}
                            scrollEventThrottle={16}
                            onScroll={checkOverScroll}
                            bounces={false}
                        >
                         ...
                        </ScrollView>
                     </GestureHandler>
                </View>
        </SwipeUpDownModal>

If I take out:

                setHeaderMarginTop(initialHeaderMarginTop + offsetY )
                setContentMarginTop(initialContentMarginTop + offsetY )

Everything runs smoothly

But, when setting the states, its causing a substantial performance hit and really slowing things down. I didn't see this performance hit in version 1.

Any suggestions on how to get around this performance hit.