Skip to content

Commit

Permalink
"hq read" renamed to "hq output-log"
Browse files Browse the repository at this point in the history
  • Loading branch information
spirali committed Sep 20, 2024
1 parent a7f619d commit 893efe5
Show file tree
Hide file tree
Showing 11 changed files with 86 additions and 63 deletions.
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
## Removed

* Because worker streaming fully replaces original streaming, the original server streaming was removed.
For most cases, you can rename `--log` to `--stream` and `hq log` to `hq read`. See the docs for more details.
For most cases, you can rename `--log` to `--stream` and `hq log` to `hq output-log`. See the docs for more details.

## Fixes

Expand Down Expand Up @@ -373,7 +373,7 @@ an old client/worker to a new server (Connecting a new client/worker to an old s

### Stream log

* Basic export of stream log into JSON (`hq read <log_file> export`)
* Basic export of stream log into JSON (`hq output-log <log_file> export`)

### Server

Expand Down
4 changes: 2 additions & 2 deletions crates/hyperqueue/src/bin/hq.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use hyperqueue::client::commands::job::{
output_job_summary, JobCancelOpts, JobCatOpts, JobCloseOpts, JobForgetOpts, JobInfoOpts,
JobListOpts, JobTaskIdsOpts,
};
use hyperqueue::client::commands::reader::command_reader;
use hyperqueue::client::commands::outputlog::command_reader;
use hyperqueue::client::commands::server::command_server;
use hyperqueue::client::commands::submit::command::{open_job, SubmitJobConfOpts};
use hyperqueue::client::commands::submit::{
Expand Down Expand Up @@ -474,7 +474,7 @@ async fn main() -> hyperqueue::Result<()> {
}) => command_task_info(&gsettings, opts).await,
#[cfg(feature = "dashboard")]
SubCommand::Dashboard(opts) => command_dashboard_start(&gsettings, opts).await,
SubCommand::Read(opts) => command_reader(&gsettings, opts),
SubCommand::OutputLog(opts) => command_reader(&gsettings, opts),
SubCommand::AutoAlloc(opts) => command_autoalloc(&gsettings, opts).await,
SubCommand::EventLog(opts) => command_event_log(&gsettings, opts).await,
SubCommand::GenerateCompletion(opts) => generate_completion(opts),
Expand Down
2 changes: 1 addition & 1 deletion crates/hyperqueue/src/client/commands/mod.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
pub mod autoalloc;
pub mod event;
pub mod job;
pub mod reader;
pub mod outputlog;
pub mod server;
pub mod submit;
pub mod wait;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use clap::Parser;
use std::path::PathBuf;

#[derive(Parser)]
pub struct ReadOpts {
pub struct OutputLogOpts {
/// Path of log file
path: PathBuf,

Expand Down Expand Up @@ -85,7 +85,7 @@ pub enum Channel {
Stderr,
}

pub fn command_reader(gsettings: &GlobalSettings, opts: ReadOpts) -> anyhow::Result<()> {
pub fn command_reader(gsettings: &GlobalSettings, opts: OutputLogOpts) -> anyhow::Result<()> {
let mut stream_dir = StreamDir::open(&opts.path, opts.server_uid.as_deref())?;
match opts.command {
StreamCommand::Summary(_) => {
Expand Down
4 changes: 2 additions & 2 deletions crates/hyperqueue/src/common/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ use crate::client::commands::job::{
JobCancelOpts, JobCatOpts, JobCloseOpts, JobForgetOpts, JobInfoOpts, JobListOpts,
JobTaskIdsOpts,
};
use crate::client::commands::reader::ReadOpts;
use crate::client::commands::outputlog::OutputLogOpts;
use crate::client::commands::server::ServerOpts;
use crate::client::commands::submit::command::SubmitJobConfOpts;
use crate::client::commands::submit::{JobSubmitFileOpts, JobSubmitOpts};
Expand Down Expand Up @@ -199,7 +199,7 @@ pub enum SubCommand {
/// Commands for controlling HyperQueue workers
Worker(WorkerOpts),
/// Operations with log
Read(ReadOpts),
OutputLog(OutputLogOpts),
/// Automatic allocation management
#[command(name = "alloc")]
AutoAlloc(AutoAllocOpts),
Expand Down
2 changes: 1 addition & 1 deletion crates/hyperqueue/src/stream/reader/streamdir.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::client::commands::reader::{CatOpts, Channel, ExportOpts, ShowOpts};
use crate::client::commands::outputlog::{CatOpts, Channel, ExportOpts, ShowOpts};
use crate::common::arraydef::IntArray;
use crate::common::error::HqError;
use crate::server::event::bincode_config;
Expand Down
36 changes: 29 additions & 7 deletions docs/imgs/cheatsheet.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 9 additions & 8 deletions docs/imgs/streaming.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 10 additions & 10 deletions docs/jobs/streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,23 +77,23 @@ HyperQueue lets you inspect the data stored inside the stream file using various
the following structure:

```bash
$ hq read <stream-path> <subcommand> <subcommand-args>
$ hq output-log <stream-path> <subcommand> <subcommand-args>
```

### Stream summary

You can display a summary of a log file using the `summary` subcommand:

```bash
$ hq read <stream-path> summary
$ hq output-log <stream-path> summary
```

### Stream jobs

To print all job IDs that streaming in the stream path, you can run the following command:

```bash
$ hq read <stream-path> jobs
$ hq output-log <stream-path> jobs
```

### Printing stream content
Expand All @@ -102,7 +102,7 @@ If you want to simply print the (textual) content of the log file, without any a
`cat` subcommand:

```bash
$ hq read <stream-path> cat <job-id> <stdout/stderr>
$ hq output-log <stream-path> cat <job-id> <stdout/stderr>
```

It will print the raw content of either `stdout` or `stderr`, ordered by task id. All outputs will be concatenated one
Expand All @@ -119,7 +119,7 @@ If you want to inspect the contents of the log, along with its inner metadata th
has produced which part of the data, you can use the `show` subcommand:

```commandline
$ hq read <log-file-path> show
$ hq output-log <log-file-path> show
```

The output will have the form `J.T:C> DATA` where `J` is a job id, `T` is a task id and `C` is `0` for `stdout` channel
Expand All @@ -132,7 +132,7 @@ You can filter a specific channel with the `--channel=stdout/stderr` flag.
Log can be exported into JSON by the following command:

```commandline
$ hq read <log-file-path> export
$ hq output-log <log-file-path> export
```

This prints the log file into a JSON format on standard output.
Expand All @@ -145,14 +145,14 @@ hence HyperQueue streaming is able to avoid mixing
outputs from different executions of the same task, when a task is restarted.

HyperQueue automatically marks all output from previous instance of a task except the last instance as *superseded*.
You can see statistics about superseded data via `hq read <stream-path> summary` command.
You can see statistics about superseded data via `hq output-log <stream-path> summary` command.
In the current version, superseded data is ignored by all other commands.

## More server instances

HyperQueue supports writing streams from the different server instances into the same directory.
If you run `hq read` commands over such directory then it will detect the situation and prints all server uids
that writes into the directory. You have to specify the server instance via `hq read --server-uid=<SERVER_UID> ...`
If you run `hq output-log` commands over such directory then it will detect the situation and prints all server uids
that writes into the directory. You have to specify the server instance via `hq output-log --server-uid=<SERVER_UID> ...`
when working with such a stream directory.

!!! note
Expand All @@ -163,4 +163,4 @@ when working with such a stream directory.
## Working with non-shared file system

You do not need to have a shared file system when working with streaming path. It is just your responsibility to
collect all generated files into one directory before using `hq read` commands.
collect all generated files into one directory before using `hq output-log` commands.
2 changes: 1 addition & 1 deletion tests/test_placeholders.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ def test_stream_submit_placeholder(hq_env: HqEnv):
hq_env.start_workers(1)
wait_for_job_state(hq_env, 1, "FINISHED")

lines = set(hq_env.command(["read", "log-1", "show"], as_lines=True))
lines = set(hq_env.command(["output-log", "log-1", "show"], as_lines=True))
assert "1.0:0> Hello" in lines


Expand Down
Loading

0 comments on commit 893efe5

Please sign in to comment.