Multiple plot with TensorboardStatsHandler for train and validation curves

[o-fares]

Hello,

Using TensorboardStatsHandler, is there a way to plot both validation and training metric curves on the same graph ?
The following code show what I tried.

Code:

trainer = create_supervised_trainer(net, opt, loss, device, False, prepare_batch=prepare_batch)
evaluator = create_supervised_evaluator(net, validation_metrics, device, non_blocking=True,
                                            prepare_batch=prepare_batch)

train_loader = DataLoader(train_dataset, batch_size)
val_loader = DataLoader(validation_dataset, batch_size)

@trainer.on(Events.EPOCH_COMPLETED)
    def run_validation(trainer):
          evaluator.run(train_loader)
          evaluator.run(val_loader)

val_tensorboard_stats_handler = TensorBoardStatsHandler(log_dir=tensorboard_logdir,
                                    output_transform=lambda x: None,
                                    global_epoch_transform=lambda x: trainer.state.epoch)
val_tensorboard_stats_handler.attach(evaluator)

Expected :

https://i.ibb.co/brn1M7z/roc-expected.png
Two separate curves for validation and training

Results obtained:

The resulting curve is this one : https://i.ibb.co/WBWRmRM/roc.png
For each epoch, there are two values on the same curve when there should be one for each curve (train and validation).

Do you know how to split the train and validation curves in that case?

Thanks

[Nic-Ma]

Hi @o-fares ,

Thanks for your interest here.
Actually, you can define an unique summary writer and pass it to several TensorBoardStatsHandlers:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/handlers/tensorboard_handlers.py#L38

Thanks.

[o-fares]

Hi @Nic-Ma,
Thank you for your quick answer.

I tried to use 2 evaluators and separate the TensorBoardStatsHandlers (one for training and one for evaluation). I added a SummaryWriter this way :

writer = SummaryWriter(tensorboard_logdir)
val_tensorboard_stats_handler = TensorBoardStatsHandler(summary_writer = writer, log_dir=tensorboard_logdir,
                                output_transform=lambda x: None,
                                global_epoch_transform=lambda x: trainer.state.epoch)
val_tensorboard_stats_handler.attach(evaluator)

But it didn't do any change and I have a curve similar to the first one in my previous post.

Do you have any advice on how to use SummaryWriter in that case ?

Thanks

[Nic-Ma]

Hi @o-fares ,

I think it's expected behavior of TensorFlow TensorBoard to save data in the different charts with different tags.
@rijobro do you know more advanced features in TenaorBoard, I am not very familiar with it.

Thanks in advance.

[o-fares]

Hi @Nic-Ma @rijobro ,

I succeeded in obtaining 2 separate plots on Tensorboard for the validation and training metrics. However, I don't find any example of how to use SummaryWriter with multiple plots. Do you have any insight on that ?

Thank you in advance.

[Nic-Ma]

Hi @o-fares ,

I think using the same SummaryWriter will save data into 1 single TensorBoard event file.
Maybe you can find some helpful guide here: https://www.tensorflow.org/tensorboard/get_started.

Thanks.

[o-fares]

Fixed it by saving tensorboard events to two different directories and passing them as SummaryWriter:

writer_1 = SummaryWriter(tensorboard_logdir)
writer_2 = SummaryWriter(tensorboard_logdir_val)


train_tensorboard_stats_handler = TensorBoardStatsHandler(summary_writer=writer_1,
                                output_transform=lambda x: x,
                                global_epoch_transform=lambda x: trainer.state.epoch,tag_name="metric")
train_tensorboard_stats_handler.attach(evaluator_tb_train)

val_tensorboard_stats_handler = TensorBoardStatsHandler(summary_writer=writer_2,
                                output_transform=lambda x: x,
                                global_epoch_transform=lambda x: trainer.state.epoch, tag_name="metric")
val_tensorboard_stats_handler.attach(evaluator_tb_val)

https://i.ibb.co/PWYf7kf/tensorboard-plot.png

[Nic-Ma]

Cool, glad to see the solution!

Thanks.