Operating Systems: Processes, Memory, Files and I/O

These operating-systems notes treat process management, memory and I/O as parts of one resource-management and isolation problem rather than unrelated topics. Threads, virtual memory, file systems, drivers and protection mechanisms are connected through that shared model, while later implementation terminology is kept distinct from the historical course core.

Unit 1: Introduction to Operating Systems

Fundamental role of an operating system

An operating system mediates between applications and hardware. It provides abstractions that make hardware usable while controlling access to shared resources.

Its responsibilities typically include:

• CPU/process scheduling,
• memory management,
• device and I/O management,
• file and storage management,
• protection and security,
• interprocess communication,
• error and resource accounting.

Resource management

CPU time, memory, devices, files and communication channels are finite resources. The operating system allocates them according to policy and enforces the mechanisms that make those policies meaningful.

The distinction between mechanism and policy is useful. A context-switch mechanism describes how one task is replaced by another; the scheduler's policy determines which task should run next.

Basic computer-system model

A simplified system can be viewed as:

applications
    ↓
system-call interface
    ↓
operating-system kernel
    ↓
device controllers / MMU / CPU
    ↓
hardware

This is an abstraction. Modern systems may include hypervisors, firmware, user-space service layers and hardware acceleration, but the boundary remains useful for reasoning about privilege and resource ownership.

Instruction execution and privilege

Processors execute ordinary instructions and privileged instructions. The operating system kernel runs with elevated privilege so it can configure page tables, interrupt controllers and devices. Applications normally run in a restricted mode.

A controlled transition to the kernel occurs through mechanisms such as system calls, traps and interrupts.

Job, program, task and process

A program is passive code/data. A process is an executing program together with its execution state, address space, resources and operating-system metadata.

Historical batch systems often described submitted work as a job. The word task is used more generally and may mean a process, thread or schedulable unit depending on the system.

Single-programming and multiprogramming

In a single-job system, CPU time can be wasted while the active job waits for I/O. Multiprogramming keeps several jobs/processes available so another can run when one blocks.

This requires:

• memory protection,
• scheduling,
• interrupt handling,
• resource accounting,
• controlled I/O.

Multitasking and interactive systems

Time-sharing gives interactive tasks short CPU intervals, creating the impression of simultaneous execution. Responsiveness becomes important in addition to throughput.

Batch processing

Batch processing groups work for non-interactive execution. It is still conceptually relevant for queues, scheduled analytics and large offline jobs even though the historical card/job-control environment has changed.

Real-time processing

Real-time correctness depends on both result and timing. A hard real-time system treats missed deadlines as system failures; a soft real-time system tolerates occasional deadline misses with degraded quality.

Low average latency alone does not guarantee real-time behavior. Worst-case latency, scheduling interference, interrupt latency and resource contention matter.

Kernel and user space

The kernel owns privileged mechanisms and global resource management. User-space processes execute with restricted access. Protection boundaries prevent an ordinary process from directly modifying another process's memory or device-control state.

System calls

A system call is the controlled interface by which user code requests kernel services such as:

• file operations,
• process creation,
• memory mapping,
• communication,
• device access.

The exact ABI differs across operating systems. A library function is not necessarily a system call; many library operations execute entirely in user space or wrap one or more kernel calls.

Evolution of operating systems

The historical progression from manual operation to batch systems, multiprogramming, time-sharing, personal computing, network/distributed systems and virtualization reflects the same pressure: improve utilization while increasing abstraction, isolation and usability.

Unit 2: Input/Output System

Purpose of I/O management

Devices differ in speed, transfer granularity, command model and failure behavior. The operating system provides a uniform enough interface while preserving device-specific control in drivers.

Device, controller and interface

A physical device may be managed through a controller that exposes registers, queues or descriptors. The CPU communicates with the controller through an I/O interface such as memory-mapped registers, bus transactions or architecture-specific I/O instructions.

Serial and parallel communication

Serial communication transfers bits sequentially over fewer signal lines. Parallel communication transfers several bits simultaneously. Higher pin count and skew limit the scalability of traditional parallel links; many modern high-speed interconnects are serial at the physical level.

Synchronous and asynchronous communication

Synchronous communication shares or recovers timing so symbols are interpreted relative to a clock relationship. Asynchronous serial formats may frame each character with start/stop information. The terms are protocol-level concepts rather than simply statements about whether operations block a thread.

Polling

In polling, software repeatedly reads device status:

while device_not_ready:
    check status

Polling can waste CPU time for slow events, but it is sometimes appropriate when event frequency is high, latency is critical and the expected wait is shorter than interrupt/context overhead.

Interrupts

An interrupt causes the processor to transfer control to an interrupt handler. The hardware/OS preserves enough context to resume execution according to the architecture.

Important concepts include:

• interrupt enable/mask,
• interrupt vector,
• priority,
• nesting,
• service routine,
• deferred/bottom-half processing.

Handlers should generally perform bounded urgent work and defer lengthy processing when the operating system provides such a mechanism.

Direct memory access

DMA allows a controller to transfer data between device and memory with limited per-byte CPU involvement. The CPU configures the transfer and handles completion/error events.

DMA introduces issues including:

• physical/IOMMU addressability,
• cache coherence,
• buffer ownership,
• alignment,
• synchronization with the device.

I/O channels and processors

Historically, channel processors and I/O processors offloaded complex device command sequences. Modern controllers, storage devices and network adapters similarly execute substantial firmware/hardware processing, although the architecture differs from classical textbook channels.

Unit 3: Process Management

Process definition

A process contains execution context such as:

• program counter,
• registers,
• address-space mappings,
• open resources,
• scheduling state,
• credentials and accounting information.

The operating system represents this through a process-control structure.

Context switching

A context switch saves the execution state of one schedulable entity and restores another. It is necessary for multitasking but has cost: kernel work, cache/TLB disruption and scheduler overhead.

Process states

A simple model includes:

new -> ready -> running -> waiting/blocked -> ready -> terminated

A running process that exhausts its time slice can return to ready; one waiting for I/O becomes ready when the event completes.

Queues

Schedulers maintain ready queues and wait queues. Queue position and selection policy determine who runs and when.

Process creation and termination

Process creation establishes a new execution context and resources. Unix-like systems traditionally separate fork-style creation from exec-style program replacement; other systems provide different primitives.

Termination releases resources, but parent/child bookkeeping can persist until exit status is collected. The exact model is operating-system specific.

CPU scheduling

Scheduling objectives can include:

• throughput,
• CPU utilization,
• response time,
• turnaround time,
• fairness,
• deadline compliance.

No single policy optimizes all objectives.

First-Come First-Served

FCFS is simple and non-preemptive. A long CPU-bound task can delay many short tasks, producing the convoy effect.

Shortest Job First and SRTF

SJF selects the shortest predicted CPU burst and minimizes average waiting time under ideal knowledge. Because future burst length is unknown, practical systems can only estimate.

Shortest Remaining Time First is the preemptive form: a newly ready task with shorter remaining time can preempt the current task.

Priority scheduling

Priority scheduling selects according to assigned priority. Low-priority tasks can starve. Aging or dynamic priority adjustment can reduce starvation.

Round Robin

Round Robin gives each task a time quantum. A very large quantum approaches FCFS; a very small quantum improves responsiveness but increases switching overhead.

Multilevel queues

Tasks may be separated into classes with different policies. Multilevel feedback queues allow tasks to move among queues based on behavior, approximating favorable treatment of interactive/short jobs without knowing future burst lengths.

Long-, medium- and short-term scheduling

Classical texts distinguish:

• long-term admission of jobs,
• medium-term swapping/suspension decisions,
• short-term CPU selection.

Modern systems do not always implement these as separate named schedulers, but the conceptual time scales remain useful.

Threads

Threads share process resources such as address space and open files while maintaining separate execution contexts, including registers and stacks.

Threads reduce the cost of sharing memory compared with separate processes, but shared address space creates synchronization obligations. A data race can corrupt invariants even when each individual instruction is valid.

Scheduling trade-offs beyond the average

A scheduling policy should be evaluated against the workload and the service objective. Average waiting time can hide starvation or very poor tail latency. Interactive workloads care about response time, batch workloads care about throughput, and real-time workloads care about deadline behavior.

FCFS can suffer from the convoy effect when a long CPU-bound job delays many short jobs. Round-robin improves responsiveness but introduces context-switch overhead and depends on the time quantum. Priority scheduling can starve low-priority work unless aging or another fairness mechanism is used.

The scheduler therefore optimizes a policy, not a universal notion of speed.

Synchronization and visibility

A mutex protects a critical section, but synchronization also establishes ordering and visibility between threads. Without a proper synchronization edge, two threads may observe memory operations in different orders even if the source code appears sequential.

Semaphores are useful when the resource count is larger than one. Condition variables or monitors express waiting for a state transition. Spin locks trade CPU time for avoiding a sleep/wakeup transition and are appropriate only for carefully bounded critical sections.

Correctness requires defining the invariant protected by each synchronization primitive, not merely placing a lock around code that sometimes races.

Unit 4: Cooperating Processes and Concurrency

Parallel/concurrent execution

Tasks that overlap in time may access shared state. The result must not depend on uncontrolled instruction interleaving when the operation is intended to be atomic at a higher level.

Race condition

A race condition occurs when correctness depends on timing/order between concurrent operations.

For example, counter++ is conceptually read-modify-write and is not automatically atomic:

read counter
add one
write counter

Two threads can lose an update.

Critical section

A critical section accesses shared state that must satisfy an invariant. Mutual exclusion ensures only the permitted number of threads enter that region simultaneously.

Disabling interrupts

On a uniprocessor kernel, disabling interrupts can protect very short kernel regions from interrupt-driven concurrency. It is not a general user-space mutual-exclusion method and is insufficient by itself on multiprocessor systems because another CPU can still run concurrently.

Software mutual exclusion

Algorithms such as Peterson's illustrate the logical requirements for mutual exclusion under specific memory assumptions. On modern multiprocessors, real synchronization must use architecture/language primitives with defined atomicity and memory ordering.

Atomic operations

Hardware atomic instructions support read-modify-write primitives such as compare-and-swap or exchange. Higher-level mutexes, semaphores and lock-free algorithms are built on such mechanisms plus memory-ordering rules.

Spinlocks

A spinlock waits actively:

while lock unavailable:
    spin

It is suitable only when hold times are expected to be very short and sleeping would cost more, or when sleeping is not allowed in the execution context. Long waits waste CPU time.

Semaphores

A semaphore maintains a count changed by atomic wait/signal operations. A binary semaphore can resemble a mutex, but a mutex normally has ownership semantics: the thread that acquires it is expected to release it. Counting semaphores naturally represent a set of interchangeable resources.

Monitors

A monitor groups shared state with mutually exclusive procedures and condition synchronization. Language/runtime implementations vary, but the conceptual advantage is that the synchronization discipline is attached to the protected abstraction.

Message passing

Processes can coordinate by sending messages rather than sharing memory directly. Message passing simplifies some ownership boundaries but introduces queueing, copying/serialization, ordering and failure semantics.

Deadlock

The classic necessary conditions are:

1. mutual exclusion,
2. hold and wait,
3. no preemption,
4. circular wait.

Removing at least one condition prevents deadlock in the modeled resource system.

Resource ordering

Assigning a global order to lock acquisition prevents circular wait when all code follows the order.

Banker's algorithm

The Banker's algorithm grants a resource request only if the resulting state remains safe, meaning there exists some completion order for all processes given declared maximum demands. It is theoretically important, but many real systems cannot know maximum future demands precisely enough to apply it directly.

Unit 5: Main Memory Management

Purpose

Memory management maps process-visible addresses to physical storage, allocates memory and isolates processes.

Logical and physical addresses

A program issues virtual/logical addresses. Hardware plus the operating system translate them to physical memory or generate a fault when the mapping is absent/invalid.

Fixed and variable partitions

Early contiguous-allocation systems divided memory into fixed or variable partitions. Fixed partitions cause internal fragmentation; variable partitions can cause external fragmentation.

Placement algorithms

For variable free regions:

• first fit chooses the first adequate hole,
• best fit chooses the smallest adequate hole,
• worst fit chooses the largest.

Each trades search cost and fragmentation behavior; none eliminates external fragmentation.

Compaction and relocation

Compaction moves allocated regions to combine free space, requiring relocatable addresses and potentially high copy cost. Paging largely removes the need for external compaction of process address spaces.

Swapping

Swapping moves process memory between RAM and backing storage. Classical whole-process swapping differs from page-level demand paging, though both use secondary storage to extend effective memory capacity.

Paging

Paging divides virtual memory into pages and physical memory into frames of equal size. A virtual address can be separated into:

virtual page number + page offset

The page table maps the virtual page number to a physical frame and protection metadata.

Translation Lookaside Buffer

A TLB caches recent address translations. A TLB hit avoids a page-table walk. Context switches and address-space changes require architecture-specific handling such as tags, identifiers or invalidation.

Multilevel page tables

A flat page table can be large. Multilevel structures allocate lower-level tables only for populated regions of the virtual address space. Modern architectures may use several levels.

Protection

Page-table entries can enforce permissions such as read, write, execute and user/kernel access. Memory protection is one of the foundations of process isolation.

Segmentation

Segmentation describes variable-sized logical regions with base/limit-like information. Some historical architectures exposed segmentation directly to applications. Contemporary mainstream 64-bit systems rely primarily on paging, though segmentation concepts remain useful for understanding logical regions and protection history.

Virtual memory and page faults

Demand-paged virtual memory permits a page to be absent from physical memory until accessed. An access to a valid but nonresident page causes a page fault. The operating system locates or creates the page contents, selects a frame, updates mappings and resumes the instruction.

A fault for an invalid/prohibited address is a protection error rather than ordinary demand paging.

Page replacement

When no free frame is available, the system may evict a page.

FIFO removes the page loaded earliest and can exhibit Belady's anomaly.

LRU would remove the page not used for the longest time, but exact implementation can be expensive; practical systems approximate recency.

Optimal replacement removes the page whose next use is farthest in the future. It is not implementable online because it requires future knowledge, but it is a useful theoretical baseline.

Locality

Temporal locality means recently used data is likely to be reused. Spatial locality means nearby addresses are likely to be used. Caches and virtual-memory systems exploit these tendencies.

Working sets, replacement, and memory pressure

Paging separates virtual address space from physical frames, but performance depends on locality. A process whose active working set does not fit in available memory generates frequent page faults and can enter thrashing.

Replacement policies such as FIFO, LRU approximations, clock algorithms, and working-set-oriented policies make different assumptions about locality. Their practical behavior also depends on dirty pages, writeback, shared pages, memory-mapped files, and the operating system's page-cache policy.

A high page-fault count is therefore a symptom to interpret together with residency, reclaim, I/O, and workload phase.

Durability boundary of file operations

A successful application write does not always imply stable persistence on physical media. Data may still exist in a user-space buffer, kernel page cache, controller cache, or device-internal queue. Correct durability depends on the operating-system API, filesystem semantics, ordering barriers, and the storage device.

Applications that require crash consistency must define exactly which state must survive a process crash, kernel crash, or power loss and then use the corresponding persistence contract.

Unit 6: File and Storage Management

File abstraction

A file is a named persistent byte/record sequence or object as defined by the operating system. Metadata can include size, timestamps, ownership, permissions and storage mapping.

Directories

Directories map names to filesystem objects and create a hierarchical namespace. Path interpretation, mount points and links determine how names resolve to objects.

File operations

Typical operations include create, open, read, write, seek, close, rename and remove. Open-file state can include current offset, access mode and references to filesystem metadata.

Storage and filesystem layers

A filesystem maps logical files to storage blocks. Internally it may maintain allocation metadata, free-space structures, directories, journals and caches.

FAT

The File Allocation Table family represents a file's cluster chain through entries in a central table. It is simple and widely interoperable but has scalability and robustness limits compared with more modern filesystems.

Inodes

Unix-style inode-based filesystems store file metadata and block mappings in inode structures. Directory entries map names to inode numbers/objects. The filename is therefore not inherently stored in the inode itself.

Hard and symbolic links

A hard link creates another directory entry for the same filesystem object/inode, subject to filesystem restrictions. A symbolic link stores a path-like reference to another name and can cross some boundaries that hard links cannot.

Allocation methods

Contiguous allocation gives excellent sequential/random access but complicates growth and external fragmentation.

Linked allocation makes growth easy but random access poor unless indexing structures are added.

Indexed allocation stores block references in index structures, supporting direct access at the cost of metadata.

Modern filesystems use more sophisticated extent/tree variants, but these classical models explain the tradeoffs.

File cache and write strategies

Operating systems cache filesystem data in memory. Writes may be buffered and acknowledged before durable storage unless an explicit durability mechanism is used. Application-visible write completion and physical persistence are therefore distinct guarantees.

Consistency

A crash can interrupt metadata updates. Journaling, copy-on-write or log-structured techniques provide different consistency/recovery strategies. They do not automatically guarantee application-level transactional consistency for arbitrary sequences of file operations.

RAID

RAID combines multiple storage devices for performance and/or redundancy. Examples:

• RAID 0 stripes without redundancy,
• RAID 1 mirrors,
• parity-based RAID distributes data/parity according to level.

RAID is not a backup: it does not protect against deletion, corruption, malware or every multi-device failure.

Unit 7: Security and Protection

Security versus protection

Protection controls access among subjects and objects inside the system. Security is broader and includes authentication, attack resistance, cryptography, auditing and operational threats.

Authentication and authorization

Authentication establishes an identity or credential claim. Authorization decides what that identity may do. Conflating them leads to designs where being logged in is incorrectly treated as permission for every resource.

Passwords

Passwords should not be stored as plaintext or fast unsalted hashes. Modern systems use salted password-hashing/KDF schemes with appropriate work factors. This is a later security correction to older textbook descriptions of password tables.

Access matrix

An access matrix conceptually maps subjects to objects and permitted operations. It is usually sparse and therefore implemented through structures such as access-control lists or capabilities.

ACLs

An ACL attaches permissions to an object, listing which subjects/groups may perform which actions.

Capabilities

A capability is an unforgeable reference/token carrying authority to an object. Capability-based designs place emphasis on possession and delegation of authority rather than repeatedly consulting an object's ACL.

Protection rings and least privilege

Hardware privilege levels create boundaries for code with different authority. The principle of least privilege gives each component only the permissions required for its task and only for the required duration.

Cryptographic primitives

Symmetric encryption uses shared secret keys and is efficient for bulk confidentiality. Asymmetric cryptography uses key pairs and supports mechanisms such as key establishment and digital signatures. Hash functions map arbitrary input to fixed-size digests and are used in integrity constructions, signatures and many protocols.

These tools solve different problems; a hash is not encryption, and encryption without authentication is not automatically secure communication.

Malware

A virus typically attaches to host code/data and propagates when the host executes. A worm propagates through systems/networks without requiring the same host-file attachment model. Modern malware often combines several techniques, so these are conceptual categories rather than mutually exclusive boxes.

Unit 8: Device Drivers

Role

A device driver translates the operating system's generic I/O model into the commands, registers, descriptors and interrupts of a specific device/controller.

Kernel position

Many drivers execute in kernel space and therefore have broad authority. Some operating systems move selected driver components to user space for isolation. The exact boundary is an architectural choice.

Device classes and device files

Character devices expose stream-like or operation-based access; block devices expose addressable storage blocks. Unix-like systems may represent device endpoints in the filesystem namespace, but the device file is an interface object rather than ordinary stored file data.

Resource mapping

Drivers manage MMIO regions, interrupts, DMA buffers, clocks, buses and power state. Resource ownership must be acquired/released in a defined lifecycle, especially across hot-plug and failure paths.

MMIO

Memory-mapped I/O maps device registers into an address space. Reads/writes may have side effects and ordering requirements. Ordinary cached-memory assumptions cannot be applied blindly to device registers.

Interrupt service and deferred work

Interrupt handlers acknowledge or capture urgent device state. Longer processing is commonly deferred to kernel worker mechanisms to limit interrupt latency.

DMA and synchronization

DMA buffers require synchronization between CPU and device ownership. On non-coherent systems, cache maintenance can be necessary. Even on coherent systems, memory-ordering rules and descriptor publication must be correct.

Character and block drivers

Character drivers commonly implement operations resembling open/read/write/ioctl/poll according to the OS. Block drivers integrate with storage request queues, caching and filesystem layers.

Driver concurrency can arise from user threads, interrupts, DMA completion and kernel workers. Locks and atomic operations must match the execution context; sleeping locks cannot be used in every interrupt context.

Unit 9: Distributed Processing and Network Communication

Distributed processing

A distributed system coordinates components across independent failure and timing domains. Unlike threads in one address space, remote peers can fail, messages can be delayed/reordered/lost, and network partitions can make state uncertain.

TCP/IP layers

A practical layered view includes link, internet, transport and application protocols. Layering does not eliminate cross-layer performance effects, but it separates responsibilities.

IP

IP provides packet addressing and routing across networks. Delivery is best effort: packets can be lost, duplicated, delayed or reordered.

TCP

TCP provides an ordered reliable byte stream between endpoints. It uses sequence numbers, acknowledgments, retransmission, flow control and congestion control.

TCP does not preserve application message boundaries. An application protocol must frame records itself.

UDP

UDP provides datagrams with ports and checksum protection but no built-in delivery, ordering, retransmission or congestion-control semantics equivalent to TCP. Applications that need those properties must implement the required protocol behavior.

Ports and sockets

A port identifies a transport endpoint within a host/protocol context. A socket API object represents local state associated with network communication.

A TCP server conceptually:

1. creates a socket,
2. binds a local address,
3. listens,
4. accepts connections,
5. exchanges bytes,
6. closes resources.

A client creates a socket and connects to the server endpoint before exchange.

Concurrent servers

A server can handle multiple clients through processes, threads, event loops, async I/O or combinations. Concurrency architecture should be chosen according to connection count, work type, latency requirements and failure isolation.

DNS

DNS resolves hierarchical names to records such as addresses and service information. Resolution can involve caches and multiple servers. Applications must tolerate expiration, multiple answers and lookup failure.

Network failure model

A timeout does not prove that the remote operation failed. A request may have executed while its response was lost. This uncertainty is fundamental to distributed systems and is why retries require idempotency or deduplication semantics.

NFS and VFS

NFS exposes remote files through a filesystem-style interface. The semantics of caching, locking and failure differ from a local disk.

A Virtual File System layer gives the kernel a common interface across different filesystem implementations. Local and remote filesystems can therefore participate in one namespace while retaining implementation-specific behavior.

Unit 10: Real-Time Scheduling and the Modern I/O Path

An operating-system scheduler does not optimize the same objective for every workload. General-purpose systems often emphasize fairness and average responsiveness, whereas real-time systems care about whether a task completes before its deadline.

Priority inversion

If a high-priority task waits for a lock held by a low-priority task, while medium-priority tasks repeatedly preempt the lock holder, the high-priority task is indirectly delayed by lower-priority work. This is priority inversion.

L: acquires lock
H: waits for the same lock
M: preempts L
H: is delayed by M as well

Protocols such as priority inheritance temporarily raise the effective priority of the lock holder so that the blocking interval can be bounded more tightly. They do not repair an unbounded critical section or a flawed concurrency design; they address a specific scheduling pathology.

Deadlines and WCET

Average execution time is not sufficient for real-time reasoning. A task's estimated worst-case execution time, period, and deadline must be considered together. A system may appear fast under ordinary load yet still violate a deadline because of a rare long pause.

release -> execution -> completion
              |
              +---- completion <= deadline ?

Tail latency, scheduler delay, interrupt latency, page faults, and I/O wait therefore belong to the same end-to-end timing budget.

The modern I/O path

A file read is not a single direct operation between an application and a storage device. A typical path is:

application
  ↓
system call / async submission
  ↓
VFS and file system
  ↓
page cache
  ↓
block layer / driver
  ↓
storage device

The page cache can reduce apparent read and write latency, while dirty-page writeback moves physical persistence to a later time. Durability operations such as fsync therefore represent a persistence boundary rather than merely another function call.

Asynchronous I/O interfaces can reduce the need to block one thread per operation. They do not increase the physical bandwidth of the device; they reorganize how waiting work and execution resources are managed.

Queueing and backpressure

If the I/O queue is allowed to grow without a bound, throughput may initially look better while queueing delay and memory use rise sharply. Reliable systems combine admission control, bounded queues, and backpressure.

The operating-system boundary is clearer when three responsibilities remain distinct: the scheduler allocates CPU time, the I/O stack arbitrates device access, and the application controls how much work it admits. Latency failures are often misdiagnosed when these layers are treated as one mechanism.

General Conceptual Framework

The course topics can be connected as one resource-flow model:

CPU scheduling
      ↕
process/thread state
      ↕
virtual memory
      ↕
files and I/O
      ↕
devices and network

Protection cuts across every layer. Concurrency appears at every layer. Failure handling appears at every layer. The operating system is therefore not merely a collection of APIs; it is the component that makes controlled sharing, isolation and progress possible while hardware and applications compete for the same physical resources.

Historical Process and Scheduling Experiments

My early experiments with process priority and CPU affinity are preserved in the Köker Optimizer project, while Köker Zamanla! provides a historical example of timers, external-process execution and a command-line contract. Their original assumptions are now documented with current operating-system limits.

From OS Mechanisms to Linux Operations

This course note treats processes, memory, file systems and I/O at the operating-system level. For service management, SELinux, NetworkManager, firewalld, DNF, LVM/XFS, Podman and operational troubleshooting, see Red Hat Enterprise Linux System Administration.

Resource exhaustion, backpressure, and fair sharing

An operating system schedules more than CPU and memory. File descriptors, sockets, process-table entries, page cache, and I/O queues are finite resources as well.

An unbounded queue often appears first as a latency problem. If producers remain faster than consumers, backpressure must exist somewhere; otherwise memory growth, timeout cascades, and retry storms can destabilize the system.

A useful design question is what happens when a resource limit is reached: reject, queue, shed older work, or reduce quality. These are application-level service contracts built on operating-system mechanisms.

Verifying operating-system claims at the right layer

POSIX, the Linux kernel, Windows NT, and individual file systems can implement similar abstractions differently. Before making a general claim, distinguish a standards-level contract from a kernel implementation detail or version-specific behavior.

Resource investigations may combine CPU use, context switches, major/minor page faults, storage queues, and file-descriptor consumption. Observation conditions and sampling intervals should be recorded because diagnostic tools also have cost.

Failure paths matter as much as normal execution: full disks, descriptor exhaustion, OOM, interrupted system calls, and partial I/O should have defined handling. Reliability claims are incomplete if they cover only the successful path.

The Operating-System Dimension of AI Workloads

An AI model is still a user-space workload, but model size, accelerator access, parallel execution, and sustained data movement can put unusual pressure on operating-system resources. Mathematical model quality and production execution behavior must therefore be evaluated at different layers.

A simplified inference path is:

request
↓
input preparation
↓
model runtime
↓
CPU / accelerator execution
↓
post-processing
↓
response

Each edge may involve a queue, memory allocation, file access, network I/O, or a device operation. Loading weights, allocating tensor workspaces, reading datasets, submitting work to an accelerator, and moving results between processes all depend on process, virtual-memory, and I/O mechanisms.

Memory pressure is a central concern. Weights, workspaces, activations, and the application heap compete for physical memory. Once paging begins, tail latency can deteriorate even when nominal compute capacity is high. An OOM condition is not necessarily explained by model size alone; concurrency, runtime buffers, page cache, and neighboring processes must be considered together.

NUMA placement can also become visible. If execution threads and frequently accessed memory reside on different NUMA nodes, remote access increases latency. Accelerator topology, CPU affinity, and interrupt placement can likewise matter in high-throughput services. The operating system abstracts hardware resources; it does not erase their physical cost.

Concurrency introduces another common error: more threads or processes do not necessarily produce more throughput. If the accelerator, memory bandwidth, or runtime is already saturated, additional work increases queue length. Average latency may remain deceptively acceptable while high percentiles grow. Admission control and backpressure are therefore part of inference reliability.

Isolation mechanisms such as cgroups can bound CPU and memory use; namespaces and permissions establish execution boundaries. Resource limits, however, do not create capacity. A model that cannot operate within a given memory budget will simply fail more deterministically under that limit.

When a GPU or NPU is present, the driver stack, device files, DMA, and user-kernel transitions enter the path. A single utilization percentage cannot prove where the bottleneck is. CPU preprocessing, copies, synchronization, or I/O may produce the same visible symptom.

Operating-systems theory does not explain how a model learns. It explains how a learned model becomes process, memory, I/O, and device work on finite hardware. Once deployed, throughput, latency, resource exhaustion, and recovery behavior become as real as model-level metrics.

Separate the mechanism from the observed outcome

Operating-system questions become clearer when concepts are kept at the correct layer. A process owns an address-space and resource context, while threads inside one process may share memory. Sharing makes communication cheap but creates data-race and synchronization risks. Mutexes, semaphores, and condition variables are not interchangeable; the ownership and waiting semantics of the problem determine the appropriate primitive.

Virtual memory is not a synonym for physical memory. Access to a virtual address may involve the TLB, page tables, and sometimes a page fault before a physical page is reached. A page fault is not necessarily an error; demand paging can make it part of normal execution. Excessive page replacement, however, can turn useful computation into continual memory movement.

The classic deadlock conditions are mutual exclusion, hold-and-wait, no forced preemption, and circular wait. Prevention deliberately breaks at least one of these conditions. Starvation is different: the system as a whole may continue to make progress while a particular thread is repeatedly denied a resource.

Scheduling questions also require precise metrics. Turnaround time, waiting time, and response time answer different questions. Interactive workloads care strongly about first response; batch workloads may care more about completion time. High CPU utilisation is not proof of useful throughput either: the processor may be busy with lock contention, page-fault handling, or excessive system activity.

A useful first step is to locate the boundary of the problem: inter-process communication, threads within one process, virtual memory, file-system semantics, or device/I/O behaviour. Optimising the wrong layer often moves the symptom without removing the cause.

The Systems-Programming Layer: From System Calls to Resource Ownership

An operating-systems course explains kernel mechanisms; systems programming studies the contracts through which user-space programs access those mechanisms. Some computer-engineering programs teach these as separate courses. The distinction is useful: understanding how a scheduler works is not the same skill as using the error, lifetime, and ownership rules of fork(), execve(), read(), or mmap() correctly.

The system-call boundary

A library function is not necessarily a system call, and a system call is not always visible directly in application code. User-space libraries may prepare parameters, buffer data, or provide compatibility behavior; the kernel becomes involved through the system-call boundary when privileged resources are required.

application
  ↓
standard library / runtime
  ↓
system-call interface
  ↓
kernel
  ↓
file system / network / process / device

The distinction also matters in performance analysis. A printf() call may remain in a user-space buffer; a write() may enter the kernel; data accepted by the kernel may still not be durable on the physical device at that instant.

File descriptors as common resource handles

On POSIX systems, a file descriptor is not limited to ordinary files. Pipes, sockets, terminals, and several kernel objects can be exposed through related read/write contracts. This unified model provides powerful composition in systems programming.

open/socket/pipe
      ↓
   fd = 3
      ↓
read / write / poll
      ↓
    close

The important property is not the numeric fd value but resource ownership and lifetime. Closing too early creates use errors; failing to close can leak resources for the lifetime of the process; after fork(), descriptors in different processes can refer to the same open-file description, which has consequences for file offsets and close behavior.

fork, exec, and wait are separate operations

In the Unix process model, launching a program should not be treated as one opaque “create process” operation. fork() creates a new process context from the current process; the exec family replaces a process image with another program; the wait family allows the parent to collect child-process status.

parent
  │
  ├─ fork() ──→ child
  │              ↓
  │           exec(...)
  │              ↓
  │          new program
  │
  └─ wait(...) ← exit status

This separation is fundamental to shell pipelines, process supervision, and daemon or service managers.

Pipes, sockets, and shared memory are different forms of IPC

A pipe provides a byte stream and is especially simple for related processes and shell pipelines. A Unix-domain socket or network socket can provide bidirectional communication and different communication topologies. Shared memory can avoid copying payload through the kernel for each transfer by exposing a common memory region, but it increases synchronization responsibility. Message queues and semaphores provide still different communication and coordination contracts.

An IPC mechanism should not be selected only for throughput. Message boundaries, copying cost, failure isolation, process lifetime, backpressure, and cleanup after failure all matter.

Blocking, non-blocking, and readiness notification

When read() waits until data is available, the descriptor exhibits blocking I/O behavior. With O_NONBLOCK, the operation can return an appropriate status instead of waiting on an unready resource. select() and poll() monitor readiness across several descriptors; Linux epoll provides different scaling properties for large descriptor sets.

Readiness notification is not identical to true asynchronous I/O. A readiness model asks “which operation can proceed without blocking now?” A completion-based asynchronous model asks “when has the operation I started actually completed?”

Partial reads, partial writes, and interruptions are normal cases

Systems code must not assume that one read() or write() call processes every requested byte. On sockets, pipes, and several device types, partial progress is normal. A call may also be interrupted by a signal. Reliable code explicitly manages the remaining byte count, the reported error, and whether retrying is semantically safe.

requested: 8192 bytes
write():    3072 bytes
remaining:  5120 bytes

These details connect operating-system theory to production code. Knowledge of processes, virtual memory, and file systems becomes operationally useful when user-space software obeys kernel contracts without leaking resources or creating unexpected blocking behavior.

References

• Abraham Silberschatz; Peter B. Galvin; Greg Gagne. Operating System Concepts. Wiley, 2012.

• Andrew S. Tanenbaum; Herbert Bos. Modern Operating Systems. Pearson, 2014.

• David A. Patterson, John L. Hennessy. Computer Organization and Design: The Hardware/Software Interface, 6th ed. Morgan Kaufmann, 2020.

• IEEE; The Open Group. POSIX.1-2008. IEEE / The Open Group, 2008.