Feature idea: perf analyzer behavior control

[Talavig]

We would like to suggest an idea we thought about and we do not think is currently available in model analyzer or perf analyzers:
When using model analyzer, we want to replicate the behavior of the prod environment when testing. For example, we want to send a lot of requests at once (a local spike of usage), or a gradual, more controlled increased in requests in specific timestamps, or any other pattern of usage.
We would like to propose the following solution: add a field in the config file with that can take on of two kinds of values: either keywords that represent different usage behaviors like spikes or gradual increase, or a path to a file. This file can contain a json generated by the customer with specific "directions" to the wanted behavior. For example, for 3 seconds send x request and then send a spike of requests.
Is such a feature possible to implement? It can be very valuable to us.

[tgerdesnv]

I do believe that Perf Analyzer supports something similar to what you are asking for via custom interval mode:
https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/perf_analyzer/docs/inference_load_modes.md#custom-interval-mode

[Talavig]

We tried using this option through model analyzer, and got the following error:

Cannot use --concurrency-range or --request-rate-range with --request-intervals.

We added it to the config.yaml in the following way:

perf_analyzer_flags:
  {request-intervals: ./time_intervals.txt}

Do you have any idea why this may happen?

[tgerdesnv]

That is likely a bug :(. MA generally specifies the concurrency with every request and probably needs to be updated to not specify it when request-intervals are specified. I don't think it will be hard to fix. I'll try to reproduce and fix today.

[tgerdesnv]

@Talavig Are you running into this from a brute search? Or quick search?

[Talavig]

We have originally encountered it using brute search, but we have tried using quick search as well and encountered the same error.

[eladamittai]

Hello, I'm currently experiencing the same issue with model analyzer 24.02. Is there an update on the progress or a certain thing I need to do in the yaml to enable it?

[YaliEkstein]

That is likely a bug :(. MA generally specifies the concurrency with every request and probably needs to be updated to not specify it when request-intervals are specified. I don't think it will be hard to fix. I'll try to reproduce and fix today.

Hey, I as well try to use it and the bug still exists. Is there any news on an update or a solve to this issue?

[tgerdesnv]

I am going to try to find time to get this fixed this week

[eladamittai]

That will be awesome! Thanks for the update! Can you tell whether it'll come out as part of the 24.03 image?

[tgerdesnv]

It will definitely not be 24.03. That being said, once I get a fix in you can always work with the main branch if you want it asap:
https://github.com/triton-inference-server/model_analyzer/blob/main/docs/install.md#alternative-installation-methods

[eladamittai]

Great! Thank you so much!

[tgerdesnv]

This work is still in-progress

[eladamittai]

Hey, is there an update?

[eladamittai]

@tgerdesnv hey, is the feature done?