Skip to content

Let sensors request asset partition range backfills - #34096

Draft
carl-distill wants to merge 3 commits into
dagster-io:masterfrom
carl-distill:cpp/multi-partition-backfill-requests
Draft

Let sensors request asset partition range backfills#34096
carl-distill wants to merge 3 commits into
dagster-io:masterfrom
carl-distill:cpp/multi-partition-backfill-requests

Conversation

@carl-distill

@carl-distill carl-distill commented Aug 10, 2026

Copy link
Copy Markdown

Codex:

Why

Sensors can request one run per partition, but publicly requesting a multi-partition asset backfill required constructing the private AssetGraphSubset type. A shared range must also resolve unambiguously across every selected partitioned asset.

A sensor run key must suppress duplicate backfills when submissions use multiple workers. The initial check-then-write sequence could race within one tick, allowing two workers to persist backfills for the same key.

What

  • Add RunRequest.for_asset_partition_range(...) for an inclusive PartitionKeyRange.
  • Resolve and validate the range after sensor evaluation, including dynamic partition changes requested by the same sensor result.
  • Require every selected partitioned asset to use the same partitions definition, both range endpoints to be valid partition keys, and the range to contain at least two partitions.
  • Reject missing or non-materializable assets and partial selections of non-subsettable execution sets.
  • Preserve run keys, config, and tags on the resulting backfill, with sensor-scoped run-key idempotency across evaluations.
  • Serialize keyed backfill checks and persistence within a submission batch, so concurrent duplicate requests produce one backfill and one skipped result.
  • Leave execution fanout to BackfillPolicy: the default policy launches one run per partition, while single_run() launches one range run.

Follow-up to #34095; the code changes are independent.

Verification

  • 35 passed across the changed run-request and sensor-daemon suites.
  • The barrier-backed two-worker regression reproduced two persisted backfills before the fix and passed 10 consecutive runs after it.
  • Ruff check and format checks pass across the seven-file PR diff.
  • Diff-aware ty check: 0 errors and 0 warnings.

@carl-distill
carl-distill force-pushed the cpp/multi-partition-backfill-requests branch from debaf88 to c544504 Compare August 10, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant