Atomic Publication in File Processing

Writing a file under a temporary name and then moving it to the target name is a common publication method that prevents readers from observing incomplete content in live data processing:

generate the data write it to a temporary file close the file move the temporary file to the target name

On a local file system, this often appears sufficient. When the same pattern is used over NFS, three different guarantees can be conflated:

atomicity visibility durability

Atomicity means that the name change occurs without exposing an intermediate state. Visibility concerns when other clients observe the new file. Durability determines whether the data survives on storage after a server or client crash.

An atomic rename does not show that the file becomes visible on all NFS clients at the same instant or that it is certainly preserved after a power failure. Reliable file publication must establish these guarantees separately.

Transition from temporary file to target file

A process writing directly to the target file exposes the following intermediate states:

a zero-byte file partially written JSON an incomplete UTF-8 sequence an unclosed JSON object a partial overwrite of the old file

If a reader opens the file while it is being written, it finds a valid directory entry although the content is not complete. Checking file size is not sufficient either. Size changes during the write, and content that appears plausible at a particular moment can still be incomplete.

With a temporary file, the target name is published only after the content has been completed:

report.json.tmp ↓ report.json

On Linux, a successful rename changes the target name atomically. If the target already exists, there is no interval during which another process cannot find the target name. If source and target are on different mounted file systems, the operation can fail with EXDEV. Creating the temporary file in the same directory as the target is therefore the simplest way to keep both on the same file system.

The Java equivalent is:

final Path temporary = target.resolveSibling(target.getFileName() + ".tmp");

write(temporary, content);

Files.move( temporary, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING );

ATOMIC_MOVE requests that the file-system provider perform the move as one atomic operation. If the provider does not support it, AtomicMoveNotSupportedException is thrown. If a move without the atomic option fails, the final state of source and target can be undefined. The Java API therefore does not allow atomicity to be assumed silently.

A reliable application should not fall back directly to a normal move when ATOMIC_MOVE fails. Such a fallback breaks the contract that the target file represents completed output. If atomic movement is unsupported, publication can be treated as failed or a separate commit protocol can be used.

Atomicity is not durability

Moving a temporary file atomically to the target name ensures that the target appears consistently during the operation. It does not by itself guarantee that the data has reached persistent storage.

The operating system can retain written data in the page cache first:

application → operating-system cache → file system → storage controller → persistent medium

Completion of a write call does not mean that the end of this chain has been reached. close should not be interpreted as a persistent-storage guarantee in every environment either.

Linux fsync requests that modified file data and associated metadata be transferred to the storage device. Applying fsync to the file does not guarantee durability of the file's directory entry. After creating a new file or performing rename, the parent directory must also be synchronized to protect the directory entry.

A strong publication sequence for a local file system is:

1. Create the temporary file
2. Write the content
3. fsync the temporary file
4. Atomically move it to the target name
5. fsync the parent directory

The third step protects file content. The fifth reduces the risk that the new target name or changed directory entry disappears after a crash.

Java FileChannel.force(true) can issue a synchronization request for file content and metadata. Directory synchronization is not a portable operation with identical behavior across all platforms in the standard Java API. On Linux, methods such as opening the directory through FileChannel can be used, but their validity depends on the file-system provider and operating system. The required guarantee must be verified against actual target-platform behavior rather than inferred from Java code alone.

Durability over NFS contains additional layers. The NFS protocol separately addresses whether writes between client and server have reached stable storage. NFSv3 permits UNSTABLE writes to be committed to stable storage later through a COMMIT operation. In NFSv4, FILE_SYNC4 requires the server not to return success before data and required metadata have reached stable storage.

Applications usually do not manage these protocol details directly. The NFS client, kernel, mount options, and server act together. An fsync call propagates a durability request through these layers, but the actual guarantee also depends on the server implementation and storage infrastructure.

If server write cache, RAID controller, or a virtual-storage layer acknowledges data as persistent while retaining it only in volatile memory, application-level calls cannot by themselves protect against power failure. The durability contract must also be validated at infrastructure level.

Visibility on NFS clients

After one client publishes the target file, the time at which another client observes it depends on more than atomic rename. NFS clients cache data and file attributes locally.

The default Linux NFS behavior is known as close-to-open consistency. When the writer closes a file, pending changes are sent to the server. When another client opens the file, it has an opportunity to validate whether cached attributes are current. This model aims for a file closed by one client to be observed by another on a subsequent open. Client attribute caches are nevertheless not perfectly synchronous, and stale data can appear under some conditions.

NFSv4 requires RENAME to be atomic from the client's perspective. A reader may observe the old name or the new name, but should not observe a partial name change at protocol level. This does not mean that directory caches on different clients are refreshed simultaneously.

A reader that keeps the file open continuously should not be expected to switch automatically to the new target content. rename associates the existing filename with a new inode or file object. A file descriptor opened earlier can continue reading the old object. A reader that needs the new version must reopen the file.

A safe consumer sequence is:

detect a directory or manifest change reopen the target file read the content from beginning to end close the file

Keeping the file continuously open and observing new publications through the same descriptor does not match the atomic-replacement model.

Directory listing can exhibit separate cache behavior. If a client recently looked up a nonexistent file, the negative lookup result may remain cached. Reducing attribute-cache duration can improve visibility, but increases metadata calls sent to the server. Options such as noac or actimeo=0 have performance costs and do not solve every consistency problem generally. Linux NFS documentation explicitly describes the attribute cache as a tradeoff between performance and freshness.

It is therefore safer to correct the application protocol before modifying mount settings. If the reader can tolerate a small delay before a completed file becomes visible, default caching provides higher throughput. If millisecond-level cross-client visibility is required, polling an NFS directory may not be an appropriate notification mechanism.

Using a commit file or manifest

If the existence of one data file represents a completed publication, atomic movement can be a sufficient commit point:

daily.json.tmp → daily.json

When one publication consists of several files, the problem changes:

schema.json data.json summary.json

Even if each file is moved atomically, a reader can observe different versions together:

new schema.json old data.json new summary.json

Per-file atomicity does not provide a transaction for a set of files.

The solution is a versioned directory or manifest:

/releases/104/schema.json /releases/104/data.json /releases/104/summary.json /current.json

The writer first creates every file belonging to version 104. After completion, it publishes a small manifest:

{ "version": 104, "path": "/releases/104" }

The reader first opens current.json, then reads the files in the referenced version directory. The commit point is only the atomic replacement of the manifest.

This model has two benefits. First, a multi-file publication becomes consistent through one version key. Second, the old version does not need to be deleted immediately. Clients that read the previous manifest can complete their reads from the old directory.

Safe cleanup of old versions requires a waiting period or reader-reference model:

publish replace the manifest wait for reader transition time clean the old version

A separate .commit file can also be used:

data.json data.json.commit

The reader considers the data complete only if both files exist. Creation of the commit file is still subject to the same atomicity and visibility rules. For multi-file structures, a manifest provides a clearer version contract than a separate commit marker for every file.

Crash scenarios

A publication protocol should be evaluated through interruptions at each step rather than only through the normal path.

If the process terminates while writing the temporary file, the target remains unchanged. The remaining .tmp file can be removed or overwritten on the next startup.

If the process terminates after the temporary file is complete but before the move, the old target remains valid. The new data has not been published.

If the process terminates after the atomic move, readers observe the new target. If file and directory durability were not established, whether the new name or content can disappear after a server crash depends on the file system and NFS server behavior.

If data files are deleted after changing the manifest, publication is corrupted. Every file in the new version must therefore be prepared before the manifest, and the old version must not be removed immediately after the manifest changes.

When several writers publish to the same target, atomic rename does not eliminate the data race. The last writer to move wins the target name:

writer A → version 104 writer B → version 105 writer A rename writer B rename

The result may be 105. If timing is reversed, the older 104 can overwrite the new version. Atomicity guarantees only that each name change is indivisible. It does not guarantee version ordering.

With multiple writers, publication must be serialized through one of the following:

• A single active producer
• An external lock or lease
• Increasing versions and a compare-and-set-like manifest update
• Separate version directories for each producer with a centralized publication decision

NFS file locks can be used, but lock loss, client crashes, and network partitions must be addressed separately. Atomically moving the target file alone does not provide multi-writer coordination.

A practical publication contract

A simple contract for a single-writer service publishing daily JSON or statistics files over NFS can be:

1. Create the temporary file in the same directory as the target
2. Write the complete content
3. Check write and close errors
4. Synchronize the file according to the required durability level
5. Move to the target name with ATOMIC_MOVE
6. Treat publication as failed if atomic movement is unsupported
7. Require readers to reopen the file for each version
8. Use a versioned manifest for multi-file publication
9. Clean old versions after a delay
10. Reconcile temporary files on restart

If file existence is used as a marker for a processed partition, the target name must appear only after a successful commit. Presence of the .tmp file must not be treated as completed output.

A parse error on the reader side should not automatically be interpreted as the writer having published a partial file. Stale client cache, an incorrect manifest version, invalid character encoding, or corrupted source data can produce the same symptom. Recording publication version, file size, and content digest makes the source easier to distinguish:

version size sha256 publishedAt schemaVersion

A hash may not be required for security validation, but it provides a stable integrity measure for detecting transfer or storage corruption.

Atomic file publication over NFS is not merely one Files.move call. rename solves the problem of not exposing a partial file to the reader. fsync and NFS commit semantics concern post-crash durability. Cache policy determines when another client sees the new version.

When these three guarantees are defined separately, a file system can become a reliable publication interface. When they are not, the same file can be complete for one client, invisible to another, and lost after a server crash.