New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question about two implementation details #1

Open

SKT-T1-Thecai opened this issue Aug 18, 2023 · 0 comments

SKT-T1-Thecai commented Aug 18, 2023

Hi, @OwenNJU. Nice code implimentation! I have reproduced this program on the opus dataset. But there is still two details which I do not fully understand.
First, https://github.com/OwenNJU/INK/blob/a6520e3d9af523872b4005554b9631a4865f00df/fairseq/models/transformer.py#L1022-L1024
It seems that the nearest neighbor is removed and other neighbors are kept and I wonder the reason.
Second,
https://github.com/OwenNJU/INK/blob/a6520e3d9af523872b4005554b9631a4865f00df/fairseq/criterions/label_smoothed_cross_entropy.py#L98-L102
When calculating the probability of the knn-search results, it seems that the K neighbors are truncated and only the first 8 neighbors of each token are used, could you please explain the reason why not all of the neighbors are used?
Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment