Missing information from paper

[ozppupbg]

Hello,

I don't know if you have done this, but looking through the paper, there are a lot of implementation details, that are missing.
Here is my current list:

• what are the model dimensions?
• who many layers (of LMM, MAC, etc.) did they use?
• In case the memory layer is updated and retrieved at the same time, is the output the retrieval before or after the update?
• How are the architecture components connected in detail?
• How is the neural memory initialized?

Did you identify more gaps?
And how did you fill them in in your implementation?

[lucidrains]

for your third bulletpoint, I too have questions about this, as it appears in a diagram they retrieve, attend, then store + gate

but I feel like it could make sense to first store then retrieve? I'm not sure but can offer both

[ozppupbg]

It could depend on, what the retrieved result is used for.
But my intuition for the general use-case as "memory" is that I would be mainly interested in what I already know about the thing I currently have and not, what I have just stored.

[lucidrains]

Your intuition makes sense. I'll just make sure it can do both ways and allow grad student descent to occur