Performance and Accuracy Benchmarks

[jamescho72]

Setup and run benchmarks against our continue.dev/ollama/granite environment.
Run baselines against our competitors Deepseek2.5 2.4B active and 21B active, codestral-mamba 7B, llama3-8B-instruct, and granite 8B instruct 128k context length.

Find 100 line code example
Ask chat to document
Measure latency (how long to complete)
Measure accuracy (How many lines of documentation was generated, How accurate/correct was the documentation IE 9/10 lines correctly)
Measure CPU consumption, Memory consumption
Automate/standardize the test as much as possible

[harshmittalibm]

I have put my initial findings here -

https://ibm.box.com/s/l69aksjokmnwdb6u2frpd715d6537pq8

It consists of the latency comparison between different models. I will update it with latency of documentation and its accuracy.