psrfits made by vcsbeam is affected more by RFI compared with vcstools

[mengyaoxue]

When I beamform toward the same pointing position using the same (legacy) data, the psrfits data products made by vcstools have higher quality compared with the psrfits made by vcsbeam (pulsars detections have higher presto sigma folded with the same ephemeris).
According to the example below, it seems that vcsbeam is affected more by RFI compared with vcstools. Not sure if that is because of the definition of 'scale' in

vcsbeam/src/form_beam.cpp

Line 437 in 6fcc7e4

scale = (max - min) / 256.0; // (for 8-bit unsigned values)

[robotopia]

I think you are correct, @mengyaoxue . There is definitely a difference between the way the output values are clipped and scaled in VCSTools vs VCSBeam. If memory serves, in VCSTools, the max and min are set differently, possibly based on the standard deviation of the values (within a 1-second time frame), and as a result, abnormally large values get clipped. VCSBeam's output is "closer to the truth", which unfortunately means (impulsive) RFI is also more "truthfully" represented.

It might be worth adding an extra option to the beamformer to let the user decide on which strategy should be used for setting the max and min. It would be an easy way to mitigate strong RFI, but should only be used if the pulsar signal itself would not also get clipped. The max and min decision is made on a per (fine) channel basis.

[fxzjshm]

For reference, the flatten bandpass code in VCSTools is here in flatten_bandpass_{I, C}_kernel, which appears to set mean intensity to 32 for each channel, if I understand it correctly.

[bwmeyers]

So, for the moment, I think this is less of an issue w.r.t. RFI and more that VCSBeam more realistically represents the dynamic range of the data.

I'm labelling it as "wontfix" for the moment. Sammy's suggestion to add an option for how the normalisation/scaling is done is a good suggestion but nominally not on the critical path for VCSBeam development/maintenance.