Token-level CodeSearchNet preprocessing option

[bzz]

Add --token-level-sources and --print options that would result in

token-level sources, used to reproduce OpenNMT seq2seq baseline from literature

'fastPathOrderedEmit                     ' - 'final Observer < ? super V > observer = '
'amb                                     ' - 'ObjectHelper . requireNonNull ( sources '
'ambArray                                ' - 'ObjectHelper . requireNonNull ( sources '
'concat                                  ' - 'ObjectHelper . requireNonNull ( sources '
'concat                                  ' - 'ObjectHelper . requireNonNull ( sources '
'concatArray                             ' - 'if ( sources . length == 0 ) { return em'
'concatArrayDelayError                   ' - 'if ( sources . length == 0 ) { return em'
'concatArrayEager                        ' - 'return concatArrayEager ( bufferSize ( )'
'concatArrayEager                        ' - 'return fromArray ( sources ) . concatMap'
'concatArrayEagerDelayError              ' - 'return fromArray ( sources ) . concatMap'

word-level source and char-level target (default)

'f a s t P a t h O r d e r e d E m i t   ' - 'final Observer<? super V> observer = dow'
'a m b                                   ' - 'ObjectHelper.requireNonNull(sources, "so'
'a m b A r r a y                         ' - 'ObjectHelper.requireNonNull(sources, "so'
'c o n c a t                             ' - 'ObjectHelper.requireNonNull(sources, "so'
'c o n c a t                             ' - 'ObjectHelper.requireNonNull(sources, "so'
'c o n c a t A r r a y                   ' - 'if (sources.length == 0) {\n            '
'c o n c a t A r r a y D e l a y E r r o ' - 'if (sources.length == 0) {\n            '
'c o n c a t A r r a y E a g e r         ' - 'return concatArrayEager(bufferSize(), bu'
'c o n c a t A r r a y E a g e r         ' - 'return fromArray(sources).concatMapEager'
'c o n c a t A r r a y E a g e r D e l a ' - 'return fromArray(sources).concatMapEager'

after 6cfc883 - token-level source and target

'fast path ordered emit                  ' - 'final Observer < ? super V > observer = '
'amb                                     ' - 'ObjectHelper . requireNonNull ( sources '
'amb array                               ' - 'ObjectHelper . requireNonNull ( sources '
'concat                                  ' - 'ObjectHelper . requireNonNull ( sources '
'concat                                  ' - 'ObjectHelper . requireNonNull ( sources '
'concat array                            ' - 'if ( sources . length == 0 ) { return em'
'concat array delay error                ' - 'if ( sources . length == 0 ) { return em'
'concat array eager                      ' - 'return concatArrayEager ( bufferSize ( )'
'concat array eager                      ' - 'return fromArray ( sources ) . concatMap'
'concat array eager delay error          ' - 'return fromArray ( sources ) . concatMap'

External dependencies like PyTorch (or https://github.com/microsoft/dpu-utils) are dropped for the CLI helper as it does not use it in any way (but implement a similar interface)

[m09]

Looks good! I added a few stylistic comments but we can merge as is.

[bzz]

@m09 thank you for the prompt review! All feedback addressed in aa65cc4

6cfc883 is a hack bringing in --token-level-targets, PR description updated with example.

[m09]

Looks handsome! Good to merge for me :)