feat(batch-exports): Backfill UI #28146

rossgray · 2025-01-31T16:11:46Z

Problem

Currently the only way to view batch export backfills is via the API. It should be possible to view backfills in the UI.

Closes #27788

Changes

Frontend

Add a 'Backfills` tab to the Data pipeline -> Destinations page.

Move the 'Backfill batch export' button + modal from the Runs scene to Backfills. On the runs scene the button is still there is there are no runs. Therefore the backfill modal logic has been extracted out of batchExportRunsLogic.tsx and into its own batchExportBackfillsLogic.tsx.

There is a button to cancel an in-progress backfill

Backend

Add a DRF ModelViewSet for backfills
- You can fetch a list of backfills, fetch an individual backfill
- Creation of backfills already existed
Previously we were never setting the value for the backfill finished_at column, so have made sure we do this going forwards

Does this work well for both Cloud and self-hosted?

Should do.

How did you test this code?

Added automated tests.

Have also tested manually.

github-actions · 2025-01-31T16:24:14Z

Size Change: +399 B (+0.03%)

Total Size: 1.17 MB

ℹ️ View Unchanged

Filename	Size	Change
`frontend/dist/toolbar.js`	1.17 MB	+399 B (+0.03%)

_{compressed-size-action}

greptile-apps

PR Summary

This PR adds a dedicated UI for managing batch export backfills, moving from API-only access to a full user interface. Here's a summary of the key changes:

Added new 'Backfills' tab in Data Pipeline -> Destinations page with a table view showing backfill status, timestamps, and controls
Extracted backfill modal logic from batchExportRunsLogic into dedicated batchExportBackfillModalLogic for better code organization
Added backend API endpoints under /api/projects/{team_id}/batch_exports/{batch_export_id}/backfills/ with proper access controls
Fixed issue where backfill finished_at timestamp wasn't being set properly
Added ability to cancel in-progress backfills through the UI

Key implementation points:

New BatchExportBackfills component displays backfills with status indicators and cancel controls
batchExportBackfillModalLogic handles form validation with timezone-aware date handling
Backend changes include proper team/organization access controls and data partitioning
Added comprehensive test coverage for backfill operations including cancellation and status updates
Maintained backward compatibility while improving code organization

_{30 file(s) reviewed, 24 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-01-31T17:45:26Z

frontend/src/scenes/pipeline/BatchExportBackfills.tsx

+    cancelBackfill,
+}: {
+    backfill: BatchExportBackfill
+    cancelBackfill: any


logic: cancelBackfill should have a proper type definition instead of 'any'

Suggested change

cancelBackfill: any

cancelBackfill: (backfill: BatchExportBackfill) => void

greptile-apps · 2025-01-31T17:45:26Z

frontend/src/scenes/pipeline/BatchExportBackfills.tsx

+                            const color = colorForStatus(status)
+                            return (
+                                <span
+                                    className={`h-6 p-2 border-2 flex items-center justify-center rounded-full font-semibold text-xs border-${color} text-${color}-dark select-none`}


style: string interpolation in className may not work with CSS modules/purging - consider using a mapping object or utility classes

greptile-apps · 2025-01-31T17:45:27Z

frontend/src/scenes/pipeline/BatchExportBackfills.tsx

+                tooltip="Cancel backfill"
+                onClick={() =>
+                    LemonDialog.open({
+                        title: 'Cancel run?',


logic: dialog title 'Cancel run?' should be 'Cancel backfill?' to match the action being performed

greptile-apps · 2025-01-31T17:45:27Z

frontend/src/scenes/pipeline/BatchExportBackfills.tsx

+                footer={
+                    hasMoreBackfillsToLoad && (
+                        <div className="flex items-center m-2">
+                            <LemonButton center fullWidth onClick={loadOlderBackfills} loading={loading}>


logic: loading prop is used while loadOlderBackfills is in progress, but loading state is shared with other operations which could cause UI issues

greptile-apps · 2025-01-31T17:46:34Z

frontend/src/scenes/pipeline/batchExportBackfillModalLogic.tsx

+                        (start_at?.minute() !== undefined && !(start_at?.minute() % 5 === 0)) ||
+                        (end_at?.minute() !== undefined && !(end_at?.minute() % 5 === 0))


logic: Potential edge case: start_at?.minute() could be undefined but end_at?.minute() defined or vice versa. Consider handling these cases separately for clearer error messages.

greptile-apps · 2025-01-31T17:50:29Z

posthog/api/test/batch_exports/test_pause.py

-def test_connot_pause_and_unpause_batch_exports_of_other_organizations(client: HttpClient):
-    temporal = sync_connect()
-
+def test_connot_pause_and_unpause_batch_exports_of_other_organizations(client: HttpClient, temporal):


syntax: Typo in function name: 'connot' should be 'cannot'

Suggested change

def test_connot_pause_and_unpause_batch_exports_of_other_organizations(client: HttpClient, temporal):

def test_cannot_pause_and_unpause_batch_exports_of_other_organizations(client: HttpClient, temporal):

greptile-apps · 2025-01-31T17:50:35Z

posthog/api/test/batch_exports/test_runs.py

 @pytest.mark.django_db(transaction=True)
-def test_cancelling_a_batch_export_run(client: HttpClient):
+def test_cancelling_a_batch_export_run(client: HttpClient, temporal):


style: The transaction=True decorator is redundant since the test is already marked with pytestmark django_db

greptile-apps · 2025-01-31T17:51:29Z

posthog/batch_exports/models.py

+        if not self.start_at or not self.end_at:
+            # just return None in this case since we don't have enough information to calculate the total runs
+            return None
+        return (self.end_at - self.start_at) // self.batch_export.interval_time_delta


style: The integer division (//) may truncate partial intervals. Consider rounding up with math.ceil() to ensure all partial intervals are counted.

greptile-apps · 2025-01-31T17:51:53Z

posthog/batch_exports/service.py

+def update_batch_export_backfill_status(
+    backfill_id: UUID, status: str, finished_at: dt.datetime | None = None
+) -> BatchExportBackfill:


logic: finished_at parameter added but not used in async version aupdate_batch_export_backfill_status. Should be consistent between sync/async versions.

Suggested change

def update_batch_export_backfill_status(

backfill_id: UUID, status: str, finished_at: dt.datetime | None = None

) -> BatchExportBackfill:

async def aupdate_batch_export_backfill_status(backfill_id: UUID, status: str, finished_at: dt.datetime | None = None) -> BatchExportBackfill:

"""Update the status of an BatchExportBackfill with given id.

Arguments:

id: The id of the BatchExportBackfill to update.

status: The new status to assign to the BatchExportBackfill.

finished_at: The time the BatchExportBackfill finished.

"""

model = BatchExportBackfill.objects.filter(id=backfill_id)

updated = await model.aupdate(status=status, finished_at=finished_at)

if not updated:

raise ValueError(f"BatchExportBackfill with id {backfill_id} not found.")

return await model.aget()

greptile-apps · 2025-01-31T17:52:32Z

posthog/temporal/batch_exports/batch_exports.py

 async def update_batch_export_backfill_model_status(inputs: UpdateBatchExportBackfillStatusInputs) -> None:
    """Activity that updates the status of an BatchExportRun."""
    backfill = await database_sync_to_async(update_batch_export_backfill_status)(
-        backfill_id=uuid.UUID(inputs.id), status=inputs.status
+        backfill_id=uuid.UUID(inputs.id),
+        status=inputs.status,
+        # we currently only call this once the backfill is finished, so we can set the finished_at here
+        finished_at=dt.datetime.now(dt.UTC),
    )


logic: Setting finished_at here assumes the activity is only called when the backfill is finished. Consider adding a parameter to explicitly control when to set finished_at to avoid potential bugs if the activity is called for intermediate status updates.

rossgray requested a review from tomasfarias January 31, 2025 17:05

rossgray added 16 commits January 31, 2025 17:43

Set backfill finished_at

537b814

Refactor backfill modal logic into separate logic

6e55620

Backfill frontend WIP

bb952b7

Refactor existing tests

36f0553

Create failing test

4fb1be1

Refactor use of temporal fixture

7823837

Initial version of list endpoint

5a7e1d4

Add more tests

c6f15e8

Add tests for get endpoint

2eb62e5

Add backfills table on frontend

11f4671

Add refresh button

b097e42

Add total_runs

aa8e8ae

Fix existing cancellation tests

1cf5bb6

Implement cancellation of backfill

71ddcca

Polish

20a4464

Fix tests

d0d3403

rossgray force-pushed the be-backfill-ui branch from b084583 to d0d3403 Compare January 31, 2025 17:43

rossgray marked this pull request as ready for review January 31, 2025 17:44

greptile-apps bot reviewed Jan 31, 2025

View reviewed changes

Update query snapshots

f71a7ea

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(batch-exports): Backfill UI #28146

feat(batch-exports): Backfill UI #28146

rossgray commented Jan 31, 2025 •

edited

Loading

github-actions bot commented Jan 31, 2025 •

edited

Loading

greptile-apps bot left a comment

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

greptile-apps bot Jan 31, 2025

	cancelBackfill: any
	cancelBackfill: (backfill: BatchExportBackfill) => void

		(start_at?.minute() !== undefined && !(start_at?.minute() % 5 === 0)) \|\|
		(end_at?.minute() !== undefined && !(end_at?.minute() % 5 === 0))

	def test_connot_pause_and_unpause_batch_exports_of_other_organizations(client: HttpClient, temporal):
	def test_cannot_pause_and_unpause_batch_exports_of_other_organizations(client: HttpClient, temporal):

-def update_batch_export_backfill_status(
-    backfill_id: UUID, status: str, finished_at: dt.datetime | None = None
-) -> BatchExportBackfill:
+async def aupdate_batch_export_backfill_status(backfill_id: UUID, status: str, finished_at: dt.datetime | None = None) -> BatchExportBackfill:
+    """Update the status of an BatchExportBackfill with given id.
+    Arguments:
+        id: The id of the BatchExportBackfill to update.
+        status: The new status to assign to the BatchExportBackfill.
+        finished_at: The time the BatchExportBackfill finished.
+    """
+    model = BatchExportBackfill.objects.filter(id=backfill_id)
+    updated = await model.aupdate(status=status, finished_at=finished_at)
+    if not updated:
+        raise ValueError(f"BatchExportBackfill with id {backfill_id} not found.")
+    return await model.aget()

feat(batch-exports): Backfill UI #28146

Are you sure you want to change the base?

feat(batch-exports): Backfill UI #28146

Conversation

rossgray commented Jan 31, 2025 • edited Loading

Problem

Changes

Frontend

Backend

Does this work well for both Cloud and self-hosted?

How did you test this code?

github-actions bot commented Jan 31, 2025 • edited Loading

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

greptile-apps bot Jan 31, 2025

Choose a reason for hiding this comment

rossgray commented Jan 31, 2025 •

edited

Loading

github-actions bot commented Jan 31, 2025 •

edited

Loading