What is the function of inspect_grad?

[KelleyYin]

dlcl/fairseq/models/dlcl_transformer.py

Line 149 in c632ee2

self.inspected_grads = OrderedDict() if getattr(args, 'inspect_grad', False) else None

Hey man,

I don't understand the function of inspect_grad. It doesn't seem to be used in your running script.
Could you please to explain it?

Thanks!

[wangqiangneu]

dlcl/fairseq/models/dlcl_transformer.py

Line 149 in c632ee2

self.inspected_grads = OrderedDict() if getattr(args, 'inspect_grad', False) else None

Hey man,

I don't understand the function of inspect_grad. It doesn't seem to be used in your running script.
Could you please to explain it?

Thanks!

Thanks for your interest!

Yes，inspect_grad only works during training to record the gradients in the collection, e.g., the gradient of every encoder layer. Then, you can easily watch the gradient flow in Tensorboar. Please refer to:

dlcl/fairseq/trainer.py

Lines 354 to 365 in c632ee2

	def _update_gnorm_detail(self, meters, model):
	if not hasattr(model, 'inspected_grads'):
	return
	#gnorms = {}
	for name, p in model.inspected_grads().items():
	if p.grad is None:
	continue
	gn = utils.item(torch.norm(p.grad.data))
	name = 'grad_%s' % name
	if name not in meters:
	meters[name] = AverageMeter()
	meters[name].update(gn)

and

dlcl/train.py

Line 188 in c632ee2

write_tensorboard(args, stats['num_updates'], stats, is_training=True)