Record Metrics for Reminder #4831

Vyom-Yadav · 2024-10-27T17:44:16Z

Summary

Provide a brief overview of the changes and the issue being addressed.
Explain the rationale and any background necessary for understanding the changes.
List dependencies required by this change, if any.

Fixes #(related issue)

Change Type

Mark the type of change your PR introduces:

Bug fix (resolves an issue without affecting existing features)
Feature (adds new functionality without breaking changes)
Breaking change (may impact existing functionalities or require documentation updates)
Documentation (updates or additions to documentation)
Refactoring or test improvements (no bug fixes or new functionality)

Testing

Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.

Review Checklist:

Reviewed my own code for quality and clarity.
Added comments to complex or tricky code sections.
Updated any affected documentation.
Included tests that validate the fix or feature.
Checked that related changes are merged.

evankanderson

A bunch of comments, mostly aiming to help simplify your life. 😁

evankanderson · 2024-10-28T18:44:56Z

internal/reminder/metrics/metrics.go

+	// For tracking average calculation
+	// TODO: consider persisting this to avoid reset on restart (maybe)
+	totalBatches   int64
+	totalReminders int64


Rather than storing these as int64s, why not use an Int64Counter? That will automatically record the total, but also that the metric is cumulative over time.

evankanderson · 2024-10-28T18:46:18Z

internal/reminder/metrics/metrics.go

+	// Current number of reminders in the batch
+	BatchSize metric.Int64Gauge


Why not use an Int64Histogram for this, which will also give you the Average metric you're tracking next (approximately, but with more-detailed frequency information, which is probably interesting).

evankanderson · 2024-10-28T18:48:05Z

internal/reminder/metrics/metrics.go

+	0,    // immediate
+	10,   // 10 seconds
+	20,   // 20 seconds
+	40,   // 40 seconds
+	80,   // 1m 20s
+	160,  // 2m 40s
+	320,  // 5m 20s
+	640,  // 10m 40s
+	1280, // 21m 20s


While powers of 2 are generally handy, I'd tend to align this more with human units, e.g.:

0, 10, 30, 60, 120, 300, 600, 900, 1800, 3600, 7200

I'm not sure what frequency we'll run reminder at, but 2 hours seems reasonably pessimistic for now -- it's really annoying when you run into problems and your frequency distribution has always been "it's not so good".

Given that our expected value for latency is 1/2 scan interval, I could even be convinced that < 1 minute is not particularly interesting (which could trim a couple buckets if you're worried about it -- I'm not!).

Updated with:

var delayBuckets = []float64{ 60, // 1 minute 300, // 5 minutes 600, // 10 minutes 1800, // 30 minutes 3600, // 1 hour 7200, // 2 hours 10800, // 3 hours 18000, // 5 hours 25200, // 7 hours 36000, // 10 hours }

evankanderson · 2024-10-28T18:52:34Z

internal/reminder/metrics/metrics.go

+	// Update running average
+	m.totalBatches++
+	m.totalReminders += size


You should aim to get the metrics code to do this for you automatically -- in particular, if we end up with two reminder processes running, getting a global average from reminder_avg_batch_size is impossible without knowing the numerator and denominator for each process. Exporting them as two cumulative totals allows aggregation across multiple processes.

Good point, I think this should be doable now for multiple reminder processes as we can aggregate metrics which are being exposed.

evankanderson · 2024-10-28T18:55:35Z

internal/reminder/reminder.go

+				remMetrics.SendDelay.Record(ctx, (time.Now().Sub(reminderLastSent.Time) - r.cfg.RecurrenceConfig.MinElapsed).Seconds())
+			} else {
+				// TODO: Should the send delay be zero if the reminder has never been sent?
+				remMetrics.SendDelay.Record(ctx, 0)


This will produce a bunch of +Inf values in the histogram bucket, which will disturb any sort of Nth percentile.

It's probably better to either export nothing here, or export it under some different counter metric ("new reminders").

Exposing a new metric now.

evankanderson · 2024-10-28T19:11:48Z

internal/reminder/metrics/provider.go

+	otel.SetMeterProvider(mp)
+
+	// Create metrics
+	meter := mp.Meter("reminder-service")


Suggested change

meter := mp.Meter("reminder-service")

meter := mp.Meter("reminder")

evankanderson · 2024-10-28T19:14:07Z

internal/reminder/metrics/provider.go

+}
+
+// Shutdown gracefully shuts down the metrics server
+func (p *Provider) Shutdown(ctx context.Context) error {


If you make this method private, then cancelling the context to Start or NewProvider becomes the only way to reach this code, which simplifies your lifecycle over having multiple callers.

(among other things, you can guarantee that p is not nil, because you don't call the shutdown method unless p is non-nil.)

evankanderson · 2024-10-28T19:17:35Z

internal/reminder/metrics/provider.go

+func (p *Provider) Metrics() *Metrics {
+	return p.metrics
+}


My main suggestion is to remove the Provider object in favor of a static setup function with cancellation, but if you do keep it, I'd suggesting making Metrics exported, rather than needing to add an accessor function.

evankanderson · 2024-10-28T19:18:32Z

internal/reminder/reminder.go

@@ -143,6 +163,11 @@ func (r *reminder) sendReminders(ctx context.Context) error {
 		return fmt.Errorf("error creating reminder messages: %w", err)
 	}

+	remMetrics := r.metricsProvider.Metrics()
+	if remMetrics != nil {
+		remMetrics.RecordBatch(ctx, int64(len(repos)))


If you make batches a histogram, this can just be a Record() on that histogram.

evankanderson · 2024-10-28T19:20:45Z

internal/reminder/metrics/metrics.go

+func (m *Metrics) RecordSendDelay(ctx context.Context, delaySeconds float64) {
+	m.SendDelay.Record(ctx, delaySeconds)
+}


It looks like this function isn't doing anything -- you're already exposing SendDelay, and there's not much semantic benefit to foo.RecordSendDelay(...) over foo.SendDelay.Record(...). That also saves you from needing to write a comment.

If you transform BatchSize into a histogram as well, then you might be able to get rid of all the methods on Metrics, which would be nice.

github-actions · 2024-11-28T02:42:53Z

This PR needs additional information before we can continue. It is now marked as stale because it has been open for 30 days with no activity. Please provide the necessary details to continue or it will be closed in 30 days.

Vyom-Yadav · 2025-01-03T07:24:29Z

Opening this, I'm working on this.

coveralls · 2025-01-03T12:39:51Z

coverage: 56.436% (-0.2%) from 56.664%
when pulling eb5c7d0 on Vyom-Yadav:add-metrics-to-reminder
into 6d4a5df on mindersec:main.

Vyom-Yadav

@evankanderson Because we reused eventer code in a6d9b22, which created pubWithMetrics using minder namespace, there are some non-reminder metrics being outputted by reminder (e.g. minder_eventer_publish_time_seconds_bucket).

I think this can cause problems, but I assume them to use the same infra, so not sure about how big of a problem that is. (Is this something we want to tackle in this PR?)

Also updated the code to remove provider and have a fire and forget setupMetricServer function

Vyom-Yadav · 2025-01-04T17:28:48Z

internal/reminder/reminder.go

+		if r.metrics != nil {
+			// sendDelay = Now() - ReminderLastSent - MinElapsed
+			reminderLastSent := repo.ReminderLastSent
+			if reminderLastSent.Valid {
+				r.metrics.SendDelay.Record(ctx, (time.Since(reminderLastSent.Time) - r.cfg.RecurrenceConfig.MinElapsed).Seconds())


I'm still not sure about this metric. Let's say a repo has been frequently updated every hour (due to GH webhook events) and r.cfg.RecurrenceConfig.MinElapsed is 2 hours. In this case, let's say after sending a reminder 18 hours back, every hour the repo was updated by GH webhook and wasn't eligible for a reminder. Now when it becomes eligible for the reminder SendDelay would be recorded as 18h which seems to be very wrong in this case. (I think we discussed this previously too)

Yeah, this feels like it should be time.Since(<last evaluation>) here. In fact, that feels like it should be at the top of the "if" block replacing the comment on line 177.

Vyom-Yadav · 2025-01-04T17:29:49Z

internal/reminder/reminder.go

+				// Recording metric for first time reminders (after how long was the initial reminder sent)
+				newSendDelay := time.Since(repoToLastUpdated[repo.ID]) - r.cfg.RecurrenceConfig.MinElapsed
+				r.metrics.NewSendDelay.Record(ctx, newSendDelay.Seconds())


Added a new metric for first time reminders. (I'm thinking this is the definition of sendDelay and what we have above is wrong)

evankanderson

A few notes -- I think you're right about determining the metrics for "sendDelay".

If you want to be particularly cute, you could write:

sendDelay := time.Since(...) - ...MinElapsed
recorder := r.metrics.SendDelay
if !reminderLastSent.Valid {
  // Use a different metric for first-time reminders
  recorder = r.metrics.NewSendDelay
}
recorder.Record(ctx, sendDelay.Seconds())

evankanderson · 2025-01-06T23:41:28Z

internal/reminder/metrics/metrics.go

+	totalBatches, err := meter.Int64Counter(
+		"total_batches",
+		metric.WithDescription("Total number of batches processed"),
+	)
+	if err != nil {
+		return nil, err
+	}


If we have batch_size as a histogram, the +Inf bucket will show all batches that have been sent, since it's storing a CDF (cumulative distribution function), IIRC. (If it's actually storing a histogram, summing the buckets will give the same result).

Note that total reminders can't be directly calculated from the histogram, because the buckets cover a range of values, so that metric is still independently useful.

reminder_total_batches_total and reminder_batch_size_count give the same result. reminder_total_reminders_total and reminder_batch_size_sum give the same result. So, I don't think those two extra metrics are useful. I've removed them.

evankanderson · 2025-01-06T23:49:13Z

internal/reminder/metrics_server.go

+		logger.Err(err).Msg("error shutting down metrics provider")
+	}
+
+	return server.Shutdown(shutdownCtx)


This shutdown sequence is a little weird, because ListenAndServe is potentially returning before this point, which seems unusual: https://pkg.go.dev/net/http#Server.Shutdown

Maybe ctx.Done() and r.stop should call server.Shutdown?

It's especially weird since you don't actually wait for startMetricsServer to exit in the main thread. (Which seems okay -- if you cut off a metrics HTTP connection in the middle, no real harm done.)

Yeah, the main goroutine goes home without waiting for his younger sibling, let me change it to send both of them back home together.

As mentioned, I'm not too worried about hanging up in the middle of a metrics request, so don't contort the code too much to make it happen. exit() is a very effective garbage collector.

Oh, now I see the other concern -- you may want to shut down the HTTP server before the metrics provider. Generally, I follow a LIFO order on teardown, with the idea that items constructed earlier probably don't depend on later items, but the other way may happen.

evankanderson · 2025-01-06T23:51:39Z

internal/reminder/reminder.go

+		}()
+
+		var err error
+		r.metrics, err = metrics.NewMetrics(otel.Meter("reminder"))


It feels like startMetricsServer should maybe return a metrics.Metrics, but I don't feel strongly.

No, you should feel strongly. There is a race condition here: (I wouldn't have checked until you pointed 🥇 )

if r.cfg.MetricsConfig.Enabled { go func() { if err := r.startMetricServer(ctx); err != nil { logger.Error().Err(err).Msg("error starting metrics server") } }() // Race condition here var err error r.metrics, err = metrics.NewMetrics(otel.Meter("reminder")) if err != nil { return err } }

otel.SetMeterProvider(mp) might not have been executed by the time otel.Meter("reminder") is run.

// If this is called before a global MeterProvider is registered the returned // Meter will be a No-op implementation of a Meter. When a global MeterProvider // is registered for the first time, the returned Meter, and all the // instruments it has created or will create, are recreated automatically from // the new MeterProvider.

Hmm, although technically, it is a race condition, but I think it should be fine (still I'll change it). Evan, can it be possible that few metrics get recorded on the no-op implementation, and then when the MeterProvider is registered for the first time, those metrics are lost?

That's a reasonable implementation for OTel, in my opinion.

My main reaction was that you went through the process of creating a MeterProvider in startMetricsServer and then 2 lines later in the outer function you go through OTel globals to fish it out again. If the code was more separated, I'd say the globals made sense, but at this proximity, I'd go for explicit over spooky action at a distance.

evankanderson · 2025-01-06T23:52:54Z

internal/reminder/reminder.go

+		if r.metrics != nil {
+			// sendDelay = Now() - ReminderLastSent - MinElapsed
+			reminderLastSent := repo.ReminderLastSent
+			if reminderLastSent.Valid {
+				r.metrics.SendDelay.Record(ctx, (time.Since(reminderLastSent.Time) - r.cfg.RecurrenceConfig.MinElapsed).Seconds())


Yeah, this feels like it should be time.Since(<last evaluation>) here. In fact, that feels like it should be at the top of the "if" block replacing the comment on line 177.

evankanderson · 2025-01-07T05:54:02Z

internal/reminder/reminder.go

 		}
 	}

-	return eligibleRepos, nil
+	return eligibleRepos, eligibleReposLastUpdated, nil


Do you need to copy idToLastUpdate into a new map, or could you just return idToLastUpdate?

I could just return idToLastUpdate as well, but that contained non eligible repos as well (so my initial thought was to create a new map of only eligible repos - helps in debugging with breakpoints - but it's fine either way). I've changed now to return idToLastUpdate.

My thought was primarily to avoid a bunch of copies and allocations. You could also clear out the entries from idToLastUpdate as you filter them if you don't want to return the extra data?

Signed-off-by: Vyom Yadav <[email protected]>

Vyom-Yadav · 2025-01-17T07:13:51Z

internal/reminder/reminder.go

+			if err := r.startMetricServer(ctx, metricsProviderReady); err != nil {
+				logger.Fatal().Err(err).Msg("failed to start metrics server")
+			}


This will call exit() even when server.Shutdown(shutdownCtx) return an error, is this correct? (This will prevent the deferred stop call)

Since startMetricServer blocks, this seems reasonable. The other option would be to have startMetricServer return a chan error and execute asynchronously within the function (i.e. move 68-86 into the goroutine started by startMetricServer).

Vyom-Yadav · 2025-01-17T07:14:11Z

internal/reminder/reminder.go

+		select {
+		case <-metricsProviderReady:
+			var err error
+			r.metrics, err = metrics.NewMetrics(otel.Meter("reminder"))
+			if err != nil {
+				return err
+			}
+		case <-ctx.Done():
+			logger.Info().Msg("reminder stopped")
+			return nil
+		}


I hope this isn't me complicating stuff.

I do think you're being overly conscientious for shutdown from main, but I'll allow it.

Vyom-Yadav · 2025-01-17T07:14:30Z

internal/reminder/reminder.go

+			if r.cfg.MetricsConfig.Enabled {
+				<-metricsServerDone
+			}


Same for this one (making sure I'm not complicating things)

Here, you might more easily do the following:

if r.cfg.MetricsConfig.Enabled { ... } else { close(metricsServerDone) }

And then you won't need the conditional here.

Vyom-Yadav · 2025-01-17T07:14:59Z

internal/reminder/reminder.go

+	if r.metrics != nil {
+		r.metrics.RecordBatch(ctx, int64(len(repos)))
+	}


I could change this to a direct call, but have kept it as it is for now.

I think the only advantage of using an indirect call is that you could hide the if r.metrics != nil inside the RecordBatch call if you use a pointer receiver.

Another option is to decide that metrics will simply always be on in Reminder -- I don't know if that will simplify the code, but I've always found the "maybe you don't want metrics" to be a strange notion in Minder that was there before I got here.

Vyom-Yadav · 2025-01-17T07:16:36Z

internal/reminder/reminder.go

+		if r.metrics != nil {
+			sendDelay := time.Since(repoToLastUpdated[repo.ID]) - r.cfg.RecurrenceConfig.MinElapsed

-	// TODO: Collect Metrics
-	// Potential metrics:
-	// - Gauge: Number of reminders in the current batch
-	// - UpDownCounter: Average reminders sent per batch
-	// - Histogram: reminder_last_sent time distribution
+			recorder := r.metrics.SendDelay
+			if !repo.ReminderLastSent.Valid {
+				recorder = r.metrics.NewSendDelay
+			}
+			recorder.Record(ctx, sendDelay.Seconds())
+		}
+	}


Now this all brings back the question that how useful is ReminderLastSent metric that we store in the main DB? Is there any other application for it? Right now it's just being used as a boolean for checking whether a repo has been reconciled or not (through reminder)

Ah, you mean the column in the database? Given where we're at now, maybe we don't need it?

Vyom-Yadav · 2025-01-17T07:20:39Z

@evankanderson Did you see this:

Because we reused eventer code in a6d9b22, which created pubWithMetrics using minder namespace, there are some non-reminder metrics being outputted by reminder (e.g. minder_eventer_publish_time_seconds_bucket).

I think this can cause problems, but I assume them to use the same infra, so not sure about how big of a problem that is. (Is this something we want to tackle in this PR?)

evankanderson · 2025-01-17T17:39:37Z

@evankanderson Did you see this:

Because we reused eventer code in a6d9b22, which created pubWithMetrics using minder namespace, there are some non-reminder metrics being outputted by reminder (e.g. minder_eventer_publish_time_seconds_bucket).
I think this can cause problems, but I assume them to use the same infra, so not sure about how big of a problem that is. (Is this something we want to tackle in this PR?)

This isn't actually a big problem, because the pod / job labels (and the OTel MetricProvider Resource, if you're using the Otel binary protocol) will be different. In fact, using the same metric name across different components makes it easier to build a dashboard that lets you "zoom out" and "zoom in" on how different components are handling common concerns (like queue lengths, latencies, etc).

evankanderson · 2025-01-17T17:42:14Z

internal/reminder/metrics/metrics.go

+// RecordBatch records the metrics for a batch of reminders
+func (m *Metrics) RecordBatch(ctx context.Context, size int64) {
+	m.BatchSize.Record(ctx, size)
+}


This is a super-minor nit, but it feels like you could remove this method, and just have callers invoke m.BatchSize.Record(ctx, size).

evankanderson · 2025-01-17T17:52:05Z

internal/reminder/metrics_server.go

+		logger.Err(err).Msg("error shutting down metrics provider")
+	}
+
+	return server.Shutdown(shutdownCtx)


Oh, now I see the other concern -- you may want to shut down the HTTP server before the metrics provider. Generally, I follow a LIFO order on teardown, with the idea that items constructed earlier probably don't depend on later items, but the other way may happen.

evankanderson · 2025-01-17T17:54:23Z

internal/reminder/reminder.go

+			if err := r.startMetricServer(ctx, metricsProviderReady); err != nil {
+				logger.Fatal().Err(err).Msg("failed to start metrics server")
+			}


Since startMetricServer blocks, this seems reasonable. The other option would be to have startMetricServer return a chan error and execute asynchronously within the function (i.e. move 68-86 into the goroutine started by startMetricServer).

evankanderson · 2025-01-17T17:55:59Z

internal/reminder/reminder.go

+		select {
+		case <-metricsProviderReady:
+			var err error
+			r.metrics, err = metrics.NewMetrics(otel.Meter("reminder"))
+			if err != nil {
+				return err
+			}
+		case <-ctx.Done():
+			logger.Info().Msg("reminder stopped")
+			return nil
+		}


I do think you're being overly conscientious for shutdown from main, but I'll allow it.

evankanderson · 2025-01-17T17:57:36Z

internal/reminder/reminder.go

+			if r.cfg.MetricsConfig.Enabled {
+				<-metricsServerDone
+			}


Here, you might more easily do the following:

if r.cfg.MetricsConfig.Enabled { ... } else { close(metricsServerDone) }

And then you won't need the conditional here.

evankanderson · 2025-01-17T17:58:47Z

internal/reminder/reminder.go

+	if r.metrics != nil {
+		r.metrics.RecordBatch(ctx, int64(len(repos)))
+	}


I think the only advantage of using an indirect call is that you could hide the if r.metrics != nil inside the RecordBatch call if you use a pointer receiver.

Another option is to decide that metrics will simply always be on in Reminder -- I don't know if that will simplify the code, but I've always found the "maybe you don't want metrics" to be a strange notion in Minder that was there before I got here.

evankanderson · 2025-01-17T17:59:21Z

internal/reminder/reminder.go

+		if r.metrics != nil {
+			sendDelay := time.Since(repoToLastUpdated[repo.ID]) - r.cfg.RecurrenceConfig.MinElapsed

-	// TODO: Collect Metrics
-	// Potential metrics:
-	// - Gauge: Number of reminders in the current batch
-	// - UpDownCounter: Average reminders sent per batch
-	// - Histogram: reminder_last_sent time distribution
+			recorder := r.metrics.SendDelay
+			if !repo.ReminderLastSent.Valid {
+				recorder = r.metrics.NewSendDelay
+			}
+			recorder.Record(ctx, sendDelay.Seconds())
+		}
+	}


Ah, you mean the column in the database? Given where we're at now, maybe we don't need it?

evankanderson reviewed Oct 28, 2024

View reviewed changes

github-actions bot added the Stale label Nov 28, 2024

github-actions bot closed this Dec 29, 2024

Vyom-Yadav reopened this Jan 3, 2025

Vyom-Yadav removed the Stale label Jan 3, 2025

Vyom-Yadav force-pushed the add-metrics-to-reminder branch from 20f38c1 to b92b0de Compare January 3, 2025 12:27

Vyom-Yadav force-pushed the add-metrics-to-reminder branch 2 times, most recently from ccde7be to 6dc6aa3 Compare January 4, 2025 17:24

Vyom-Yadav commented Jan 4, 2025

View reviewed changes

Vyom-Yadav marked this pull request as ready for review January 4, 2025 17:32

Vyom-Yadav requested a review from a team as a code owner January 4, 2025 17:32

Vyom-Yadav requested a review from evankanderson January 4, 2025 17:32

evankanderson reviewed Jan 7, 2025

View reviewed changes

Vyom-Yadav force-pushed the add-metrics-to-reminder branch from 6dc6aa3 to 31446e3 Compare January 17, 2025 07:08

Record Metrics for Reminder

eb5c7d0

Signed-off-by: Vyom Yadav <[email protected]>

Vyom-Yadav force-pushed the add-metrics-to-reminder branch from 31446e3 to eb5c7d0 Compare January 17, 2025 07:12

Vyom-Yadav commented Jan 17, 2025

View reviewed changes

evankanderson reviewed Jan 17, 2025

View reviewed changes

		// Current number of reminders in the batch
		BatchSize metric.Int64Gauge

	meter := mp.Meter("reminder-service")
	meter := mp.Meter("reminder")

Record Metrics for Reminder #4831

Are you sure you want to change the base?

Record Metrics for Reminder #4831

Conversation

Vyom-Yadav commented Oct 27, 2024 • edited Loading

Summary

Change Type

Testing

Review Checklist:

evankanderson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 28, 2024

Vyom-Yadav commented Jan 3, 2025

coveralls commented Jan 3, 2025 • edited Loading

Vyom-Yadav left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evankanderson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vyom-Yadav Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vyom-Yadav commented Jan 17, 2025

evankanderson commented Jan 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vyom-Yadav commented Oct 27, 2024 •

edited

Loading

coveralls commented Jan 3, 2025 •

edited

Loading

Vyom-Yadav left a comment •

edited

Loading

Vyom-Yadav Jan 17, 2025 •

edited

Loading