Restart-Resilient Backfill Design in Date-Partitioned Data Pipelines

Whether a day in the interval [d, d + 1) is eligible for processing depends not only on the end of the calendar day but also on the source data closure policy. Given daily delay tolerance L, current time t, and the end of day end(d), the condition is:

eligible(d, t) = t >= end(d) + L

Accordingly, the latest closed day is:

Dclosed(t) = max { d | eligible(d, t) }

The existence of a daily output file is reliable state information only when it conclusively represents complete production. If the target file becomes visible during writing, if interrupted execution leaves partial content, or if progress is retained only in memory, it becomes impossible to determine which days were actually completed after a restart.

Data closure boundary and processing time

The end of a calendar day does not mean that its data are ready to be processed. Transactions in the source system may complete with delay, final records may arrive after midnight, or daily data may become stable only at a specified operational time.

For example, if the finalization boundary is 01:00 on the following day, then at 00:40 on 4 August, 3 August is not yet considered closed; the latest safe day is 2 August. After 01:00, 3 August becomes eligible for processing.

This boundary serves a function similar to the watermark concept in stream processing systems. In Apache Beam, a watermark is a progress indicator for the extent to which input data are complete for a given time window. The closure time in a daily batch task may instead be a deterministic time boundary if permitted by the source system's operating contract. If late data can arrive after closure, the boundary represents completeness accepted by business policy rather than absolute completeness.

Processing time and data time must be separated. Running the task on 4 August does not necessarily mean that it must process 4 August data. The date to be processed is derived from the data closure rule, not from the calendar day on which the task runs.

Deriving state from committed outputs

Retaining the last processed date in a cursor may initially seem sufficient:

cursorDate = 2026-07-31

However, the cursor and the actual output state can diverge:

1. Output is written, but the process stops before the cursor is updated.
2. The cursor is updated, but the process stops before output becomes durable.
3. A new category is added; the cursor is ahead, but historical outputs for the new category do not exist.
4. A file is deleted externally; the cursor does not represent this absence.
5. Configuration changes; the old cursor no longer represents the new output space.

The actual state is not the date at which processing stopped, but which elements of the expected output set have been committed. An output partition can be defined by the following coordinates:

P = (date, unit, categoryType, valueType)

The expected set of partitions for a date is:

Expected(date) = Units(date) × Queries(date)

Here, Queries includes logical output dimensions such as category and value type. A partition is complete under the following condition:

complete(P) = committedArtifactExists(path(P))

At every startup, the process regenerates expected partitions from the current configuration, compares them with committed outputs in the file system, and processes only missing partitions. Thus, when a new category is added, Expected(date) expands; because outputs for the new coordinates are absent on earlier dates, backfill begins automatically.

If the earliest meaningful date of each query is denoted by E(q), the processing range is:

[max(globalStart, E(q)), Dclosed]

This prevents scanning meaningless years for categories added later when no historical source data exist.

Deterministic scanning that prioritizes the newest day

Traditional backfill proceeds from the oldest date to the newest. When there is a large historical gap, this can delay current data. Where the operational requirement is both to publish the latest closed day quickly and to close historical gaps over time, scanning can proceed in reverse chronological order.

1. Determine the latest closed day.
2. Inspect all expected partitions for that day.
3. Complete missing partitions to guarantee the most current output.
4. Move to the preceding day.
5. Continue backwards to the earliest permitted date.
6. Stop the loop when a resource, duration, or workload limit is reached.
7. Repeat the same calculation on the next run.

The basic flow is as follows:

closedDate = latestClosedDate(now)
expected = loadCurrentDefinitions()

for date = closedDate downto earliestRelevantDate:
    for partition in deterministicOrder(expected, date):
        target = path(date, partition)

        if committed(target):
            continue

        data = query(date, partition)
        publishAtomically(target, data)

This algorithm does not update a date or category cursor. On every restart, scanning begins again from the current latest closed date. Previously completed partitions are skipped through an inexpensive existence check, while the first incomplete partition is regenerated.

Scanning may be bounded by the number of days, the number of partitions, total query duration, or the quantity of files generated in a run. Such a limit does not change correctness; it only divides the work into cycles. Because state is derived from outputs, this division requires no additional checkpoint.

Atomic publication and the commit indicator

For file existence to count as a commit indicator, publication should use a temporary file rather than write directly to the target file:

generate data
write to a temporary file
close the stream
atomically move the temporary file to the target name

The fundamental Java pattern is:

final Path temporary = target.resolveSibling(target.getFileName() + ".tmp");
writeResult(temporary, result);
Files.move(temporary, target, StandardCopyOption.ATOMIC_MOVE);

ATOMIC_MOVE requests that the file be moved as an atomic file system operation. The Java API throws AtomicMoveNotSupportedException if the operation is unsupported. Under supported conditions, the POSIX rename operation also requires name replacement to be atomic. The target file is therefore either not visible at all or visible in its completed form; readers cannot observe an intermediate writing state.

The temporary file should be created on the same file system as the target, preferably in the same directory. Moving across file systems may become a copy-and-delete operation. Silently falling back to a non-atomic move in an environment that does not support atomic moves violates the assumption of an algorithm that uses file existence as a commit indicator.

There are two safe options in this case:

• Treat the operation as failed if atomic publication is unsupported.
• Use a commit marker separate from the data file and create it only after writing is complete.

In the second model, data.json alone is insufficient; a partition is complete only when data.json.commit also exists. The commit marker itself must also be published safely.

Under this protocol, failure cases produce two stable outcomes:

• If failure occurs before or while writing the temporary file, the target does not exist; the partition is processed again.
• If failure occurs after writing completes but before the move, the temporary file is cleaned up or overwritten.
• If failure occurs after the atomic move, the target exists; the partition is considered complete.
• If failure occurs while advancing to the next partition, the completed partition is skipped and processing resumes from the missing partition.

Rerunnability and idempotent effects

A restarted task may execute the same database query more than once. This is not itself an error; what matters is that the durable, externally observable effect occurs once.

Consistent with the distinction in Apache Flink documentation, exactly-once processing does not mean that every record is physically processed only once. After a failure, a source stream may be rewound and replayed; the guarantee is that each event affects managed state exactly once. End-to-end exactly-once requires a replayable source and a transactional or idempotent sink.

The same distinction applies to date-partitioned file generation:

exactly-once processing ≠ exactly-once observable effect

A query may execute twice, but deterministic output produces only one atomically committed file at the same target coordinate. The appropriate characterization is therefore rerunnable and idempotent publication.

If file generation is accompanied by database writes, the file system move and database transaction are not atomic under a single local transaction. Committing the file first or committing the database first leaves an intermediate failure window in both orders. One target must safely tolerate retries of the other. An idempotent insert using a natural or synthetic partition key can be used in the database; atomic publication and a reconciliation mechanism can be used for files.

state = committed outputs

time boundary = data closure policy

progress = deterministic scanning of missing partitions

resilience = idempotent generation + atomic publication

An in-memory cursor may be a performance optimization, but it should not be the sole source of correctness. Configuration defines the expected output set, the file system defines the realized output set, and the closure rule defines the time range permitted for processing. The system progresses by processing the difference between these sets. When such a task can rerun its normal algorithm without requiring a special recovery scenario, fault tolerance becomes a natural property of its processing model.