Predicting memory usage - Memory Access Violation

[AgentSmithers]

Description

When running InferAsync with 16Gb of ram I hit a peek of 100% memory usage. I added memory to resolve the issue but was wondering if there is a way to predict the memory usage by calculating the size of the token input to cancel and give feedback that additional ram is required instead of triggering a memory corruption/violation error. Anyone know if this calculation is possible?

[martindevans]

I'd recommend asking this upstream in llama.cpp, whatever they say will apply to LLamaSharp too.

[AgentSmithers]

Thanks for the feedback. I'll give it a go with them.