One area that I haven't seen clearly documented in Microsoft Fabric Data Factory is how pipeline run id and state behave across activity retries and pipeline reruns. Tbh I haven't looked closely either (please let me know if this is clearly documented somewhere!).
Anyway, after testing a parent pipeline triggering a worker pipeline with multiple notebook activities, the following behaviour emerged.
Curious if anyone can confirm this, have made the same observations, or have found docs about this:
THE CORE DISTINCTION
The most important distinction is that an activity retry and a pipeline rerun are two different things.
Activity Retry
A retry is simply another attempt at the activity within the existing pipeline execution.
A retry can ONLY be configured by setting the activity's retry option to >0. I don't think there's a way to manually trigger retry of a pipeline activity.
For example:
Pipeline Run A
Both attempts have the same RunId, GroupId and TriggerTime.
Pipeline Rerun
A rerun creates a NEW pipeline run.
This applies to both:
In both cases:
For example:
Original run:
RunId = A
GroupId = A
TriggerTime = T1
Rerun from failed activity:
RunId = B
GroupId = A
TriggerTime = T2
So even though "Rerun from failed activity" feels like continuing the original run from the failure point, technically it is a NEW PIPELINE RUN.
RunId vs. GroupId
Think of them as:
Observed behaviour:
Initial run
Rerun #1
Rerun #2
So:
RunId is therefore not suitable as a stable identifier across reruns.
GroupId appears to be the built-in pipeline identity that remains stable across manual reruns.
PIPELINE VARIABLES ARE RE-EVALUATED
Pipeline variables should not be treated as persistent state across reruns.
For example, a variable assigned from:
@pipeline().RunId
or:
@string(rand(1,1000000))
will be evaluated again when a rerun starts, including when using "Rerun from failed activity".
This (somewhat surprisingly!) includes the case when the Set Variable activity had already run successfully before the failed activity. When clicking "Rerun from failed activity", the Set Variable will be evaluated again.
As a result, variables reflect the new pipeline execution rather than the original one.
This is an important consequence of the fact that a rerun is a new pipeline run.
PRESERVING STABLE VALUES ACROSS RERUNS
When a stable value must survive reruns, there are three useful approaches.
If the goal is simply to identify all reruns as belonging to the same logical execution, use:
This remains constant across reruns in my testing.
- 2. Pass a Parameter from the Parent Pipeline
A parent pipeline can generate a value once and pass it to the worker pipeline as a parameter.
For example:
- ExecutionId = @pipeline().GroupId
or:
- ExecutionId = @pipeline().RunId
from the parent pipeline's original execution.
Because parameters are passed explicitly, downstream activities can continue using the same value regardless of how many times the worker pipeline is rerun.
- 3. Use Activity Outputs / Notebook Exit Values
When using "Rerun from failed activity", Fabric preserves outputs from activities that already succeeded and are skipped during the rerun.
This means a notebook can generate an identifier once:
- notebookutils.notebook.exit(execution_id)
Downstream activities can then continue consuming that same output during reruns, even though:
The preserved activity output therefore provides a way of propagating upstream state through the remainder of the pipeline.
PRACTICAL GUIDANCE
Requirement: Identify a specific pipeline execution
- Recommended approach: RunId
Requirement: Identify a logical execution across reruns
- Recommended approach: GroupId
Requirement: Pass a stable value from orchestrator to worker pipelines
- Recommended approach: Invoke Pipeline parameter
Requirement: Preserve state produced by an already-successful activity
- Recommended approach: Activity output / notebook exit value
Requirement: Store values that should be re-evaluated on rerun
- Recommended approach: Pipeline variables
THE MENTAL MODEL
The key mental model is:
Logical execution: GroupId A
Run A
Rerun #1
Rerun #2
Whereas an activity retry is simply another attempt within the same run:
Run A
In one sentence:
An activity retry continues the same pipeline execution, while both "Rerun from failed activity" and "Rerun entire pipeline" create a new pipeline execution that remains linked to the original through GroupId. RunId, TriggerTime, and expression-based pipeline variables belong to the new execution, whereas GroupId and preserved outputs from previously successful activities provide continuity across reruns.
This is purely based on my observations, but I haven't found particularly clear documentation explaining all of these details.
Has anyone else tested this, or found Microsoft documentation that confirms this?
