mistralrs-bench

[lucasavila00]

Closes #164

llama.cpp

$ ./llama-bench -m /home/lucas/.cache/huggingface/hub/models--TheBloke--Mistral-7B-Instruct-v0.1-GGUF/snapshots/731a9fc8f06f5f5e2db8a0cf9d256197eb6e05d1/mistral-7b-instruct-v0.1.Q4_K_M.gguf 
| model                          |       size |     params | backend    | ngl | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------- | ---------------: |
| llama 7B Q4_K - Medium         |   4.07 GiB |     7.24 B | CUDA       |  99 | pp 512     |  1682.64 ± 58.17 |
| llama 7B Q4_K - Medium         |   4.07 GiB |     7.24 B | CUDA       |  99 | tg 128     |     64.46 ± 1.25 |

mistral.rs

$ cargo run --release --features cuda
+----------------------------------------+---------+--------+----------+-----------+-------------+--------------+
| model                                  | backend | test   | t/s      | ms/t      | concurrency | throughput/s |
+----------------------------------------+---------+--------+----------+-----------+-------------+--------------+
| TheBloke/Mistral-7B-Instruct-v0.1-GGUF | CUDA    | tg 128 | 59.78263 | 16.727266 |           1 |     59.78263 |
+----------------------------------------+---------+--------+----------+-----------+-------------+--------------+
| TheBloke/Mistral-7B-Instruct-v0.1-GGUF | CUDA    | pp 512 | 654.6095 | 1.5276283 |           1 |     654.6095 |
+----------------------------------------+---------+--------+----------+-----------+-------------+--------------+

[github-actions]

Code Metrics Report

  ───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Rust                        61     20225     1446       860    17919       1157
───────────────────────────────────────────────────────────────────────────────
Total                       61     20225     1446       860    17919       1157
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop 59,209
Estimated Schedule Effort 11.027891 months
Estimated People Required 4.505036
───────────────────────────────────────────────────────────────────────────────
Processed 685030 bytes, 0.685 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────
  

[lucasavila00]

@EricLBuehler I made model an argument of the benchmark, by moving that part to core.

I would like to merge this ASAP.

The PR is too big already.

There's a lot to improve, but I'd prefer to make the improvements in further smaller PRs

[EricLBuehler]

Great, thanks for adding this,