Finetune SigLIP on custom dataset for retrieval task.

[hungphongtrn]

Hi everyone, wish you a good new day.

I am trying to finetune SigLIP and CLIP for a text-based retrieval task. However, when finetuned SigLIP with sigmoid loss, the metrics seem to be worse than finetune with the InfoNCE loss. Here are some snippets and information I think will be useful for debugging and understand the behavior:

• Calculation of contrastive loss with optional soft-labelling/EMA inspired from ALBEF

# contrastive calculation
def compute_constrative(
    image_features,
    text_features,
    image_features_stopped,
    text_features_stopped,
    sim_targets,
    alpha,
    logit_scale,
    use_sigmoid,
    logit_bias,
):
    """
    Compute constrative loss for image-text pairs
    with soft labeling mechanism

    Args:
        image_features: image features after pooling
        text_features: text features after pooling
        image_features_stopped: stopped gradients image features
        text_features_stopped: stopped gradients text features
        sim_targets: similarity targets for the image-text pairs
        alpha: scaling factor for the similarity targets
        logit_scale: scaling factor for the logits
        use_sigmoid: use sigmoid or softmax for the similarity targets
        logit_bias: bias for the logits if using sigmoid
    """
    if alpha != 0:
        with torch.no_grad():
            # Soft labels for the similarity targets
            sim_i2t_s = logit_scale * image_features_stopped @ text_features.t()
            sim_t2i_s = logit_scale * text_features_stopped @ image_features.t()
            # Soft + hard labels for the similarity targets
            if use_sigmoid:
                sim_targets = alpha * F.sigmoid(sim_i2t_s) + (1 - alpha) * sim_targets

            else:
                sim_i2t_targets = (
                    alpha * F.softmax(sim_i2t_s, dim=1) + (1 - alpha) * sim_targets
                )
                sim_t2i_targets = (
                    alpha * F.softmax(sim_t2i_s, dim=1) + (1 - alpha) * sim_targets
                )
    # If alpha is 0, use the hard labels
    else:
        sim_i2t_targets = sim_targets
        sim_t2i_targets = sim_targets

    if use_sigmoid:
        # Compute the cosine similarity between the image and text features as the logits
        logits = logit_scale * image_features @ text_features.t() + logit_bias
        # Commpute the negative log-likelihood loss
        loss = -F.logsigmoid(sim_targets * logits).sum() / image_features.shape[0]
        return loss

    else:
        # Compute the cosine similarity between the image and text features as the logits
        sim_i2t = logit_scale * image_features @ text_features.t()
        sim_t2i = logit_scale * text_features @ image_features.t()
        # Commpute the negative log-likelihood loss
        loss_i2t = -torch.sum(
            F.log_softmax(sim_i2t, dim=1) * sim_i2t_targets, dim=1
        ).mean()
        loss_t2i = -torch.sum(
            F.log_softmax(sim_t2i, dim=1) * sim_t2i_targets, dim=1
        ).mean()

        # Loss is the average of the two losses
        loss = (loss_i2t + loss_t2i) / 2
        return loss

• Preparation of the sim_targets

    def prepare_sim_targets(self, pids, use_sigmoid=False):
        """
        Prepare similarity targets for constrative learning.

        Args:
            pids (torch.Tensor): Tensor containing the person IDs.
        Returns:
            torch.Tensor: Tensor containing the similarity targets.
        """
        sim_targets = torch.eq(pids.view(-1, 1), pids.view(1, -1)).float()
        if use_sigmoid:
            sim_targets = (
                -torch.ones_like(sim_targets) + 2 * sim_targets
            )  # -1 if different, 1 if same
            return sim_targets

        return sim_targets / sim_targets.sum(
            dim=1, keepdim=True
        )  # normalize the true matching distribution

• Logit scale and logit bias is trainable and inited respectively at log(10) and -10.
• Generally, finetune SigLIP with CLIP's loss yield significantly better result compared to sigmoid loss:

Currently, I do not have any idea to explain this so I do not have any clear question. Looking forward to see you advice and feel free to ask me for any supplementary information.