[Observability] Added object refs Task is dependent on to TaskInfoEntry
#48234
+77
−50
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexeykudinkin
requested review from
a team,
pcmoritz and
raulchen
as code owners
jjyao
reviewed
Oct 23, 2024
| @@ -637,6 +637,7 @@ CoreWorker::CoreWorker(const CoreWorkerOptions &options, const WorkerID &worker_ | |||
| [this] { | |||
| RAY_LOG(INFO) << "Event stats:\n\n" | |||
| << io_service_.stats().StatsString() << "\n\n" | |||
| << task_execution_service_.stats().StatsString() << "\n\n" | |||
There was a problem hiding this comment.
Haha, I have another PR that also adds this.
jjyao
reviewed
Oct 24, 2024
src/ray/protobuf/common.proto
Outdated
| // this task is dependent on and does NOT contain | ||
| // - Args passed by value (inlined) | ||
| // - ObjectRefs of the args passed by value | ||
| repeated ObjectReference dependent_args_refs = 27; |
There was a problem hiding this comment.
I think we can only record the ObjectID instead of the entire ObjectRef?
There was a problem hiding this comment.
Surely, but why not entire ObjectRef?
There was a problem hiding this comment.
I think we only care about ObjectID so trying to reduce memory usage: we need to store lots of events in GCS also each task can potentially have unlimited number of args.
|
There is test failure. |
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
…ds on Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
… tasks API Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
Signed-off-by: Alexey Kudinkin <[email protected]>
alexeykudinkin
force-pushed
the
ak/tsk-evts-ctx-fix
branch
from
42ff9d3
to
2f0c056
Compare
Signed-off-by: Alexey Kudinkin <[email protected]>
jjyao
reviewed
Oct 28, 2024
| // NOTE: This list only contains `ObjectReference`s passed in as arguments | ||
| // this task is dependent on and does NOT contain | ||
| // - Args passed by value (inlined) | ||
| // - ObjectRefs of the args passed by value |
There was a problem hiding this comment.
Inlined objects
| "task_id": "31323334", | ||
| "parent_task_id": "", | ||
| "args_object_ids": [ | ||
| arg_ref.hex(), |
There was a problem hiding this comment.
Don't we store the binary format of the object id instead of hex?
There was a problem hiding this comment.
We decode in the API
alexeykudinkin
force-pushed
the
ak/tsk-evts-ctx-fix
branch
2 times, most recently
from
0b64417
to
ccd8b8c
Compare
Signed-off-by: Alexey Kudinkin <[email protected]>
alexeykudinkin
force-pushed
the
ak/tsk-evts-ctx-fix
branch
from
ccd8b8c
to
29659a6
Compare
jjyao
approved these changes
Oct 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Currently when listing current tasks using
ray get task <task-id>we're not reflecting the args the task might be dependent on and therefore waiting to become available.That substantially complicates troubleshooting when the task is blocked on an argument passed in as an object-ref.
This change adds arguments passed in by reference to the
TaskInfoEntryto subsequently make it available to our StateHead APIs.Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.