feat: support scalar resources metric

[zedongh]

Add metric

queue_allocated_scalar_resources{queue_name="<queue>", resource="<resource>"}
queue_request_scalar_resources{queue_name="<queue>", resource="<resource>"}
queue_deserved_scalar_resources{queue_name="<queue>", resource="<resource>"}

/close #3931

Need discuss:
Shall we merge cpu, memory bulitin resources into scalar resource ?

[JesseStutler]

ScalerResources is used for extension resources. Keep CPU, and memory not to merge them into scalerResources are fine.

[JesseStutler]

/lgtm

[JesseStutler]

/area stable-metrics

[volcano-sh-bot]

@JesseStutler: The label(s) area/stable-metrics cannot be applied. These labels are supported: ``

In response to this:

/area stable-metrics

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[archlitchi]

/lgtm

[yccharles]

how about capacity ? such as :

volcano_queue_capacity_cpu
volcano_queue_capacity_mem
volcano_queue_capacity_scalar_resource

[JesseStutler]

how about capacity ? such as :

volcano_queue_capacity_cpu volcano_queue_capacity_mem volcano_queue_capacity_scalar_resource

cc @Monokaix

[Monokaix]

how about capacity ? such as :

volcano_queue_capacity_cpu volcano_queue_capacity_mem volcano_queue_capacity_scalar_resource

You mean add the capability of each dimension resource? The cpu and memory metrics is already exposed and we cannot modify its metric name for compatibility: )

[yccharles]

how about capacity ? such as :
volcano_queue_capacity_cpu volcano_queue_capacity_mem volcano_queue_capacity_scalar_resource

You mean add the capability of each dimension resource? The cpu and memory metrics is already exposed and we cannot modify its metric name for compatibility: )

there is no metrics about capacity in both vc-sheduler and vc-controller.

any other place ?

[JesseStutler]

how about capacity ? such as :
volcano_queue_capacity_cpu volcano_queue_capacity_mem volcano_queue_capacity_scalar_resource

You mean add the capability of each dimension resource? The cpu and memory metrics is already exposed and we cannot modify its metric name for compatibility: )

there is no metrics about capacity in both vc-sheduler and vc-controller.

any other place ?

https://github.com/volcano-sh/volcano/blob/master/docs/design/metrics.md

[Monokaix]

Plesse also update doc https://github.com/volcano-sh/volcano/blob/master/docs/design/metrics.md and volcano-monitoring-development.yaml to include the new metrics: )

[lowang-bh]

Any test results will be more appreciated:

1. queue's allocated gpu become zero, so does the metric
2. queue is removed, so does the metric

[volcano-sh-bot]

New changes are detected. LGTM label has been removed.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign k82cn
You can assign the PR to them by writing /assign @k82cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zedongh]

some changes update:

1. update queue metric doc
2. add unit test for queue allocated/request/deserved resource

[zedongh]

some changes update:

1. update queue metric doc
2. add unit test for queue allocated/request/deserved resource

@Monokaix @lowang-bh

And

volcano-monitoring-development.yaml to include the new metrics
need clearify.

Update volcano-monitoring-latest.yaml dashboard?

The original cpu/memeory metrics are not present in dashboard. Update later if necessary.

[yccharles]

how about capacity ? such as :
volcano_queue_capacity_cpu volcano_queue_capacity_mem volcano_queue_capacity_scalar_resource

You mean add the capability of each dimension resource? The cpu and memory metrics is already exposed and we cannot modify its metric name for compatibility: )

there is no metrics about capacity in both vc-sheduler and vc-controller.
any other place ?

https://github.com/volcano-sh/volcano/blob/master/docs/design/metrics.md

We also need export new metrics to describe each resource's capacity of a queue.
Current there is no metrics about capacity metrics.

scene：

1. queue's UsageRate = request / capacity
2. add alert when queue's capacity is almost exhaust

The proposal has been agreed by @JesseStutler . Can you implement this feature？ @zedongh

[lowang-bh]

/ok-to-test

[Monokaix]

volcano-monitoring-latest.yaml

If its not present in volcano-monitoring-latest.yaml we can skip add it, and add what @yccharles said is ok: )

[JesseStutler]

how about capacity ? such as :
volcano_queue_capacity_cpu volcano_queue_capacity_mem volcano_queue_capacity_scalar_resource

You mean add the capability of each dimension resource? The cpu and memory metrics is already exposed and we cannot modify its metric name for compatibility: )

there is no metrics about capacity in both vc-sheduler and vc-controller.
any other place ?

https://github.com/volcano-sh/volcano/blob/master/docs/design/metrics.md

We also need export new metrics to describe each resource's capacity of a queue. Current there is no metrics about capacity metrics.

scene：

1. queue's UsageRate = request / capacity
2. add alert when queue's capacity is almost exhaust

The proposal has been agreed by @JesseStutler . Can you implement this feature？ @zedongh

@yccharles Your capacity should mean the set value of Guarantee/Deserved/Capability, right?
You can also describe it in detail in the original issue #3931, and then ask @zedongh to confirm the requirements.

[zedongh]

New commit for queue capacity metric. @yccharles

[volcano-sh-bot]

@yccharles: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JesseStutler]

The capacity you ask is capability here? @yccharles I misunderstood the meaning of capacity before and wanted to confirm it.

[yccharles]

Thank you for the reminder.

1. what i want is queue's capacity . not realCapability.xxx
2. but, if realCapability.xxx can be exported is really a good idea. such as : queue_real_capability_xxx ?

[zedongh]

Thank you for the reminder.

1. what i want is queue's capacity . not realCapability.xxx
2. but, if realCapability.xxx can be exported is really a good idea. such as : queue_real_capability_xxx ?

updated with support both metrics

[JesseStutler]

@yccharles capacity refers to the capability value set by the user, right? Because there are too many words now, I am afraid that capacity and capability will be confused, we need to be clear. Besides, do we also need the deserved and guarantee values ​​set by the user?

[JesseStutler]

Also add queue_capacity_xxx metrics here?

[zedongh]

updated with queue_capacity_xxx and queue_real_capacity_xxx metric docs

[lowang-bh]

please squash commits to only one.

[zedongh]

please squash commits to only one.

capacity metric part need disscuss. Split it to another issue to make all things done?
@JesseStutler @yccharles