Summary

The main update is the new distances module, which adds an extra level of modularity to loss functions. It is a pretty big design change, which is why so many arguments have become obsolete. See the documentation for a description of the new module.

Other updates include support for half-precision, new regularizers and mixins, improved documentation, and default values for most initialization parameters.

Breaking Changes

Dependencies

This library now requires PyTorch >= 1.6.0. Previously there was no explicit version requirement.

Losses and Miners

All loss functions

`normalize_embeddings` has been removed

If you never used this argument, nothing needs to be done.
normalize_embeddings = True: just remove the argument.
normalize_embeddings = False: remove the argument and instead pass it into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(normalize_embeddings=False))

ContrastiveLoss, GenericPairLoss, BatchHardMiner, HDCMiner, PairMarginMiner

`use_similarity` has been removed

If you never used this argument, nothing needs to be done.
use_similarity = True: remove the argument and:

### if you had set normalize_embeddings = False ###
from pytorch_metric_learning.distances import DotProductSimilarity
loss_func = ContrastiveLoss(distance=DotProductSimilarity(normalize_embeddings=False))

#### otherwise ###
from pytorch_metric_learning.distances import CosineSimilarity
loss_func = ContrastiveLoss(distance=CosineSimilarity())

`squared_distances` has been removed

If you never used this argument, nothing needs to be done.
squared_distances = True: remove the argument and instead pass power=2 into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = ContrastiveLoss(distance=LpDistance(power=2))

squared_distances = False: just remove the argument.

ContrastiveLoss, TripletMarginLoss

`power` has been removed

If you never used this argument, nothing needs to be done.
power = 1: just remove the argument
power = X, where X != 1: remove the argument and instead pass it into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(power=2))

TripletMarginLoss

`distance_norm` has been removed

If you never used this argument, nothing needs to be done.
distance_norm = 2: just remove the argument
distance_norm = X, where X != 2: remove the argument and instead pass it as p into a distance object. For example:

from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(p=1))

NPairsLoss

`l2_reg_weight` has been removed

If you never used this argument, nothing needs to be done.
l2_reg_weight = 0: just remove the argument
l2_reg_weight = X, where X > 0: remove the argument and instead pass in an LpRegularizer and weight:

from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = NPairsLoss(embedding_regularizer=LpRegularizer(), embedding_reg_weight=0.123)

SignalToNoiseRatioContrastiveLoss

`regularizer_weight` has been removed

If you never used this argument, nothing needs to be done.
regularizer_weight = 0: just remove the argument
regularizer_weight = X, where X > 0: remove the argument and instead pass in a ZeroMeanRegularizer and weight:

from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = SignalToNoiseRatioContrastiveLoss(embedding_regularizer=ZeroMeanRegularizer(), embedding_reg_weight=0.123)

SoftTripleLoss

`reg_weight` has been removed

If you never used this argument, do the following to obtain the same default behavior:

from pytorch_metric_learning.regularizers import SparseCentersRegularizer
weight_regularizer = SparseCentersRegularizer(num_classes, centers_per_class)
SoftTripleLoss(..., weight_regularizer=weight_regularizer, weight_reg_weight=0.2)

reg_weight = X: remove the argument, and use the SparseCenterRegularizer as shown above.

WeightRegularizerMixin and all classification loss functions

If you never specified regularizer or reg_weight, nothing needs to be done.
regularizer = X: replace with weight_regularizer = X
reg_weight = X: replace with weight_reg_weight = X

Classification losses

For all losses and miners, default values have been set for as many arguments as possible. This has caused a change in ordering in positional arguments for several of the classification losses. The typical form is now:

loss_func = SomeClassificatinLoss(num_classes, embedding_loss, <keyword arguments>)

See the documentation for specifics

Reducers

ThresholdReducer

threshold has been replaced by low and high

Replace threshold = X with low = X

Regularizers

All regularizers

`normalize_weights` has been removed

If you never used this argument, nothing needs to be done.
normalize_weights = True: just remove the argument.
normalize_weights = False: remove the argument and instead pass normalize_embeddings = False into a distance object. For example:

from pytorch_metric_learning.distances import DotProductSimilarity
loss_func = RegularFaceRegularizer(distance=DotProductSimilarity(normalize_embeddings=False))

Inference

MatchFinder

`mode` has been removed

Replace mode="sim" with either distance=CosineSimilarity() or distance=DotProductSimilarity()
Replace mode="dist" with distance=LpDistance()
Replace mode="squared_dist" with distance=LpDistance(power=2)

New Features

Distances

Distances bring an additional level of modularity to building loss functions. Here's an example of how they work.

Consider the TripletMarginLoss in its default form:

from pytorch_metric_learning.losses import TripletMarginLoss
loss_func = TripletMarginLoss(margin=0.2)

This loss function attempts to minimize [d_ap - d_an + margin]₊.

In other words, it tries to make the anchor-positive distances (d_ap) smaller than the anchor-negative distances (d_an).

Typically, d_ap and d_an represent Euclidean or L2 distances. But what if we want to use a squared L2 distance, or an unnormalized L1 distance, or completely different distance measure like signal-to-noise ratio? With the distances module, you can try out these ideas easily:

### TripletMarginLoss with squared L2 distance ###
from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(margin=0.2, distance=LpDistance(power=2))

### TripletMarginLoss with unnormalized L1 distance ###
loss_func = TripletMarginLoss(margin=0.2, distance=LpDistance(normalize_embeddings=False, p=1))

### TripletMarginLoss with signal-to-noise ratio###
from pytorch_metric_learning.distances import SNRDistance
loss_func = TripletMarginLoss(margin=0.2, distance=SNRDistance())

You can also use similarity measures rather than distances, and the loss function will make the necessary adjustments:

### TripletMarginLoss with cosine similarity##
from pytorch_metric_learning.distances import CosineSimilarity
loss_func = TripletMarginLoss(margin=0.2, distance=CosineSimilarity())

With a similarity measure, the TripletMarginLoss internally swaps the anchor-positive and anchor-negative terms: [s_an - s_ap + margin]₊. In other words, it will try to make the anchor-negative similarities smaller than the anchor-positive similarities.

All losses, miners, and regularizers accept a distance argument. So you can try out the MultiSimilarityMiner using SNRDistance, or the NTXentLoss using LpDistance(p=1) and so on. Note that some losses/miners/regularizers have restrictions on the type of distances they can accept. For example, some classification losses only allow CosineSimilarity or DotProductSimilarity as their distance measure between embeddings and weights. To view restrictions for specific loss functions, see the documentation

There are four distances implemented (LpDistance, SNRDistance, CosineSimilarity, DotProductSimilarity), but of course you can extend the BaseDistance class and write a custom distance measure if you want. See the documentation for more.

EmbeddingRegularizerMixin

All loss functions now extend EmbeddingRegularizerMixin, which means you can optionally pass in (to any loss function) an embedding regularizer and its weight. The embedding regularizer will compute some loss based on the embeddings alone, ignoring labels and tuples. For example:

from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = MultiSimilarityLoss(embedding_regularizer=LpRegularizer(), embedding_reg_weight=0.123)

WeightRegularizerMixin is now a subclass of WeightMixin

As in previous versions, classification losses extend WeightRegularizerMixin, which which means you can optionally pass in a weight matrix regularizer. Now that WeightRegularizerMixin extends WeightMixin, you can also specify the weight initialization function in object form:

from ..utils import common_functions as c_f
import torch

# use kaiming_uniform, with a=1 and mode='fan_out'
weight_init_func = c_f.TorchInitWrapper(torch.nn.kaiming_uniform_, a=1, mode='fan_out')
loss_func = SomeClassificationLoss(..., weight_init_func=weight_init_func)

New Regularizers

For increased modularity, the regularizers hard-coded in several loss functions were separated into their own classes. The new regularizers are:

LpRegularizer
SparseCentersRegularizer
ZeroMeanRegularizer

Support for half-precision

In previous versions, various functions would break in half-precision (float16) mode. Now all distances, losses, miners, regularizers, and reducers work with half-precision, float32, and double (float64).

New collect_stats argument

All distances, losses, miners, regularizers, and reducers now have a collect_stats argument, which is True by default. This means that various statistics are collected in each forward pass, and these statistics can be useful to look at during experiments. However, if you don't care about collecting stats, you can set collect_stats=False, and the stat computations will be skipped.

Other updates

You no longer have to explicitly call .to(device) on classification losses, because their weight matrices will be moved to the correct device during the forward pass if necessary. See issue #139
Reasonable default values have been set for all losses and miners, to make these classes easier to try out. In addition, equations have been added to many of the class descriptions in the documentation. See issue #140
Calls to torch.nonzero have been replaced by torch.where.
The documentation for ArcFaceLoss and CosFaceLoss have been fixed to reflect the actual usage. (The documentation previously indicated that some arguments are positional, when they are actually keyword arguments.)
The tensorboard_folder argument for utils.logging_presets.get_record_keeper is now optional. If you don't specify it, then there will be no tensorboard logs, which can be useful if speed is a concern.
The loss dictionary in BaseTrainer is now cleared at the end of each epoch, to free up GPU memory. See issue #171

v0.9.90

********** Summary **********

********** Breaking Changes **********