Operating Systems: Processes, Memory, Files and I/O

Process management, memory, and I/O are treated as parts of one resource-management and isolation problem rather than unrelated topics. Threads, virtual memory, file systems, drivers, and protection mechanisms are connected through that shared model, while later implementation terminology remains distinct from the historical course core.

Unit 1: Introduction to Operating Systems

1.1. Fundamental Role of an Operating System

An operating system mediates between applications and hardware. It provides abstractions that make hardware usable while controlling access to shared resources.

Its responsibilities typically include:

• CPU/process scheduling,
• memory management,
• device and I/O management,
• file and storage management,
• protection and security,
• interprocess communication,
• error and resource accounting.

1.2. Resource Management

CPU time, memory, devices, files and communication channels are finite resources. The operating system allocates them according to policy and enforces the mechanisms that make those policies meaningful.

The distinction between mechanism and policy is useful. A context-switch mechanism describes how one task is replaced by another; the scheduler's policy determines which task should run next.

1.3. Basic computer-system Model

A simplified system can be viewed as:

applications
    ↓
system-call interface
    ↓
operating-system kernel
    ↓
device controllers / MMU / CPU
    ↓
hardware

This is an abstraction. Modern systems may include hypervisors, firmware, user-space service layers and hardware acceleration, but the boundary remains useful for reasoning about privilege and resource ownership.

1.4. Instruction Execution and Privilege

Processors execute ordinary instructions and privileged instructions. The operating system kernel runs with elevated privilege so it can configure page tables, interrupt controllers and devices. Applications normally run in a restricted mode.

A controlled transition to the kernel occurs through mechanisms such as system calls, traps and interrupts.

1.5. Job, Program, Task and Process

A program is passive code/data. A process is an executing program together with its execution state, address space, resources and operating-system metadata.

Historical batch systems often described submitted work as a job. The word task is used more generally and may mean a process, thread or schedulable unit depending on the system.

1.6. Single-programming and Multiprogramming

In a single-job system, CPU time can be wasted while the active job waits for I/O. Multiprogramming keeps several jobs/processes available so another can run when one blocks.

This requires:

• memory protection,
• scheduling,
• interrupt handling,
• resource accounting,
• controlled I/O.

1.7. Multitasking and Interactive Systems

Time-sharing gives interactive tasks short CPU intervals, creating the impression of simultaneous execution. Responsiveness becomes important in addition to throughput.

1.8. Batch Processing

Batch processing groups work for non-interactive execution. It is still conceptually relevant for queues, scheduled analytics and large offline jobs even though the historical card/job-control environment has changed.

1.9. Real-time Processing

Real-time correctness depends on both result and timing. A hard real-time system treats missed deadlines as system failures; a soft real-time system tolerates occasional deadline misses with degraded quality.

Low average latency alone does not guarantee real-time behavior. Worst-case latency, scheduling interference, interrupt latency and resource contention matter.

1.10. Kernel and User Space

The kernel owns privileged mechanisms and global resource management. User-space processes execute with restricted access. Protection boundaries prevent an ordinary process from directly modifying another process's memory or device-control state.

1.11. System Calls

A system call is the controlled interface by which user code requests kernel services such as:

• file operations,
• process creation,
• memory mapping,
• communication,
• device access.

The exact ABI differs across operating systems. A library function is not necessarily a system call; many library operations execute entirely in user space or wrap one or more kernel calls.

1.12. Evolution of Operating Systems

The historical progression from manual operation to batch systems, multiprogramming, time-sharing, personal computing, network/distributed systems and virtualization reflects the same pressure: improve utilization while increasing abstraction, isolation and usability.

Unit 2: Input/Output System

2.1. Purpose of I/O Management

Devices differ in speed, transfer granularity, command model and failure behavior. The operating system provides a uniform enough interface while preserving device-specific control in drivers.

2.2. Device, Controller and Interface

A physical device may be managed through a controller that exposes registers, queues or descriptors. The CPU communicates with the controller through an I/O interface such as memory-mapped registers, bus transactions or architecture-specific I/O instructions.

2.3. Serial and Parallel Communication

Serial communication transfers bits sequentially over fewer signal lines. Parallel communication transfers several bits simultaneously. Higher pin count and skew limit the scalability of traditional parallel links; many modern high-speed interconnects are serial at the physical level.

2.4. Synchronous and Asynchronous Communication

Synchronous communication shares or recovers timing so symbols are interpreted relative to a clock relationship. Asynchronous serial formats may frame each character with start/stop information. The terms are protocol-level concepts rather than simply statements about whether operations block a thread.

2.5. Polling

In polling, software repeatedly reads device status:

while device_not_ready:
    check status

Polling can waste CPU time for slow events, but it is sometimes appropriate when event frequency is high, latency is critical and the expected wait is shorter than interrupt/context overhead.

2.6. Interrupts

An interrupt causes the processor to transfer control to an interrupt handler. The hardware/OS preserves enough context to resume execution according to the architecture.

Important concepts include:

• interrupt enable/mask,
• interrupt vector,
• priority,
• nesting,
• service routine,
• deferred/bottom-half processing.

Handlers should generally perform bounded urgent work and defer lengthy processing when the operating system provides such a mechanism.

2.7. Direct Memory Access

DMA allows a controller to transfer data between device and memory with limited per-byte CPU involvement. The CPU configures the transfer and handles completion/error events.

DMA introduces issues including:

• physical/IOMMU addressability,
• cache coherence,
• buffer ownership,
• alignment,
• synchronization with the device.

2.8. I/O Channels and Processors

Historically, channel processors and I/O processors offloaded complex device command sequences. Modern controllers, storage devices and network adapters similarly execute substantial firmware/hardware processing, although the architecture differs from classical textbook channels.

Unit 3: Process Management

To evaluate process scheduling, first describe workloads by arrival times, execution times, and priority requirements. Compute waiting and response times for the selected scheduler, but low averages do not establish acceptable worst-case latency. Account separately for blocking caused by shared locks and interrupts. In real-time workloads, the decision must be based on demonstrable deadline satisfaction rather than CPU utilization alone.

3.1. Process Definition

A process contains execution context such as:

• program counter,
• registers,
• address-space mappings,
• open resources,
• scheduling state,
• credentials and accounting information.

The operating system represents this through a process-control structure.

3.2. Context Switching

A context switch saves the execution state of one schedulable entity and restores another. It is necessary for multitasking but has cost: kernel work, cache/TLB disruption and scheduler overhead.

3.3. Process States

A simple model includes:

new -> ready -> running -> waiting/blocked -> ready -> terminated

A running process that exhausts its time slice can return to ready; one waiting for I/O becomes ready when the event completes.

3.4. Queues

Schedulers maintain ready queues and wait queues. Queue position and selection policy determine who runs and when.

3.5. Process Creation and Termination

Process creation establishes a new execution context and resources. Unix-like systems traditionally separate fork-style creation from exec-style program replacement; other systems provide different primitives.

Termination releases resources, but parent/child bookkeeping can persist until exit status is collected. The exact model is operating-system specific.

3.6. CPU Scheduling

Scheduling objectives can include:

• throughput,
• CPU utilization,
• response time,
• turnaround time,
• fairness,
• deadline compliance.

No single policy optimizes all objectives.

3.7. First-Come First-Served

FCFS is simple and non-preemptive. A long CPU-bound task can delay many short tasks, producing the convoy effect.

3.8. Shortest Job First and SRTF

SJF selects the shortest predicted CPU burst and minimizes average waiting time under ideal knowledge. Because future burst length is unknown, practical systems can only estimate.

Shortest Remaining Time First is the preemptive form: a newly ready task with shorter remaining time can preempt the current task.

3.9. Priority Scheduling

Priority scheduling selects according to assigned priority. Low-priority tasks can starve. Aging or dynamic priority adjustment can reduce starvation.

3.10. Round Robin

Round Robin gives each task a time quantum. A very large quantum approaches FCFS; a very small quantum improves responsiveness but increases switching overhead.

3.11. Multilevel Queues

Tasks may be separated into classes with different policies. Multilevel feedback queues allow tasks to move among queues based on behavior, approximating favorable treatment of interactive/short jobs without knowing future burst lengths.

3.12. Long-, medium- and short-term Scheduling

Classical texts distinguish:

• long-term admission of jobs,
• medium-term swapping/suspension decisions,
• short-term CPU selection.

Modern systems do not always implement these as separate named schedulers, but the conceptual time scales remain useful.

3.13. Threads

Threads share process resources such as address space and open files while maintaining separate execution contexts, including registers and stacks.

Threads reduce the cost of sharing memory compared with separate processes, but shared address space creates synchronization obligations. A data race can corrupt invariants even when each individual instruction is valid.

3.14. Scheduling trade-offs Beyond the Average

A scheduling policy should be evaluated against the workload and the service objective. Average waiting time can hide starvation or very poor tail latency. Interactive workloads care about response time, batch workloads care about throughput, and real-time workloads care about deadline behavior.

FCFS can suffer from the convoy effect when a long CPU-bound job delays many short jobs. Round-robin improves responsiveness but introduces context-switch overhead and depends on the time quantum. Priority scheduling can starve low-priority work unless aging or another fairness mechanism is used.

The scheduler therefore optimizes a policy, not a universal notion of speed.

3.15. Synchronization and Visibility

A mutex protects a critical section, but synchronization also establishes ordering and visibility between threads. Without a proper synchronization edge, two threads may observe memory operations in different orders even if the source code appears sequential.

Semaphores are useful when the resource count is larger than one. Condition variables or monitors express waiting for a state transition. Spin locks trade CPU time for avoiding a sleep/wakeup transition and are appropriate only for carefully bounded critical sections.

Correctness requires defining the invariant protected by each synchronization primitive, not merely placing a lock around code that sometimes races.

Unit 4: Cooperating Processes and Concurrency

To analyze a concurrency defect, list shared variables and the access order of each task. If the failure occurs only under particular interleavings, identify the minimal operation that must remain atomic. Then inspect lock order, waiting cycles, and wake-up conditions. When repeated successful runs prove little, deliberately explore alternative schedules and check state invariants.

4.1. Parallel/concurrent Execution

Tasks that overlap in time may access shared state. The result must not depend on uncontrolled instruction interleaving when the operation is intended to be atomic at a higher level.

4.2. Race Condition

A race condition occurs when correctness depends on timing/order between concurrent operations.

For example, counter++ is conceptually read-modify-write and is not automatically atomic:

read counter
add one
write counter

Two threads can lose an update.

4.3. Critical Section

A critical section accesses shared state that must satisfy an invariant. Mutual exclusion ensures only the permitted number of threads enter that region simultaneously.

4.4. Disabling Interrupts

On a uniprocessor kernel, disabling interrupts can protect very short kernel regions from interrupt-driven concurrency. It is not a general user-space mutual-exclusion method and is insufficient by itself on multiprocessor systems because another CPU can still run concurrently.

4.5. Software Mutual Exclusion

Algorithms such as Peterson's illustrate the logical requirements for mutual exclusion under specific memory assumptions. On modern multiprocessors, real synchronization must use architecture/language primitives with defined atomicity and memory ordering.

4.6. Atomic Operations

Hardware atomic instructions support read-modify-write primitives such as compare-and-swap or exchange. Higher-level mutexes, semaphores and lock-free algorithms are built on such mechanisms plus memory-ordering rules.

4.7. Spinlocks

A spinlock waits actively:

while lock unavailable:
    spin

It is suitable only when hold times are expected to be very short and sleeping would cost more, or when sleeping is not allowed in the execution context. Long waits waste CPU time.

4.8. Semaphores

A semaphore maintains a count changed by atomic wait/signal operations. A binary semaphore can resemble a mutex, but a mutex normally has ownership semantics: the thread that acquires it is expected to release it. Counting semaphores naturally represent a set of interchangeable resources.

4.9. Monitors

A monitor groups shared state with mutually exclusive procedures and condition synchronization. Language/runtime implementations vary, but the conceptual advantage is that the synchronization discipline is attached to the protected abstraction.

A condition variable is not a counter that preserves the history of events; it coordinates rechecking a shared-state predicate. Waiting must occur in a loop while the protected condition is false. A consumer awakened when an item becomes available may find that another consumer has already removed it; notification does not guarantee the item remains. The predicate check and state update must obey the same locking protocol, or missed notifications and data races can occur.

4.10. Message Passing

Processes can coordinate by sending messages rather than sharing memory directly. Message passing simplifies some ownership boundaries but introduces queueing, copying/serialization, ordering and failure semantics.

4.11. Deadlock

The classic necessary conditions are:

1. mutual exclusion,
2. hold and wait,
3. no preemption,
4. circular wait.

Removing at least one condition prevents deadlock in the modeled resource system.

4.12. Resource Ordering

Assigning a global order to lock acquisition prevents circular wait when all code follows the order.

4.13. Banker's Algorithm

The Banker's algorithm grants a resource request only if the resulting state remains safe, meaning there exists some completion order for all processes given declared maximum demands. It is theoretically important, but many real systems cannot know maximum future demands precisely enough to apply it directly.

Unit 5: Main Memory Management

5.1. Purpose

Memory management maps process-visible addresses to physical storage, allocates memory and isolates processes.

5.2. Logical and Physical Addresses

A program issues virtual/logical addresses. Hardware plus the operating system translate them to physical memory or generate a fault when the mapping is absent/invalid.

5.3. Fixed and Variable Partitions

Early contiguous-allocation systems divided memory into fixed or variable partitions. Fixed partitions cause internal fragmentation; variable partitions can cause external fragmentation.

5.4. Placement Algorithms

For variable free regions:

• first fit chooses the first adequate hole,
• best fit chooses the smallest adequate hole,
• worst fit chooses the largest.

Each trades search cost and fragmentation behavior; none eliminates external fragmentation.

5.5. Compaction and Relocation

Compaction moves allocated regions to combine free space, requiring relocatable addresses and potentially high copy cost. Paging largely removes the need for external compaction of process address spaces.

5.6. Swapping

Swapping moves process memory between RAM and backing storage. Classical whole-process swapping differs from page-level demand paging, though both use secondary storage to extend effective memory capacity.

5.7. Paging

Paging divides virtual memory into pages and physical memory into frames of equal size. A virtual address can be separated into:

virtual page number + page offset

The page table maps the virtual page number to a physical frame and protection metadata.

5.8. Translation Lookaside Buffer

A TLB caches recent address translations. A TLB hit avoids a page-table walk. Context switches and address-space changes require architecture-specific handling such as tags, identifiers or invalidation.

5.9. Multilevel Page Tables

A flat page table can be large. Multilevel structures allocate lower-level tables only for populated regions of the virtual address space. Modern architectures may use several levels.

5.10. Protection

Page-table entries can enforce permissions such as read, write, execute and user/kernel access. Memory protection is one of the foundations of process isolation.

5.11. Segmentation

Segmentation describes variable-sized logical regions with base/limit-like information. Some historical architectures exposed segmentation directly to applications. Contemporary mainstream 64-bit systems rely primarily on paging, though segmentation concepts remain useful for understanding logical regions and protection history.

5.12. Virtual Memory and Page Faults

Demand-paged virtual memory permits a page to be absent from physical memory until accessed. An access to a valid but nonresident page causes a page fault. The operating system locates or creates the page contents, selects a frame, updates mappings and resumes the instruction.

A fault for an invalid/prohibited address is a protection error rather than ordinary demand paging.

Copy-on-write process creation can initially share physical pages between distinct virtual address spaces, allocating a separate page on the first write. Duplicating a process with a large virtual address space therefore need not copy its entire memory immediately. If parent and child both modify a large working set, physical memory consumption can nevertheless rise sharply. Virtual allocation, resident pages, and shared pages should be measured separately; address-space size alone does not quantify memory pressure.

5.13. Page Replacement

When no free frame is available, the system may evict a page.

FIFO removes the page loaded earliest and can exhibit Belady's anomaly.

LRU would remove the page not used for the longest time, but exact implementation can be expensive; practical systems approximate recency.

Optimal replacement removes the page whose next use is farthest in the future. It is not implementable online because it requires future knowledge, but it is a useful theoretical baseline.

5.14. Locality

Temporal locality means recently used data is likely to be reused. Spatial locality means nearby addresses are likely to be used. Caches and virtual-memory systems exploit these tendencies.

5.15. Working Sets, Replacement, and Memory Pressure

Paging separates virtual address space from physical frames, but performance depends on locality. A process whose active working set does not fit in available memory generates frequent page faults and can enter thrashing.

Replacement policies such as FIFO, LRU approximations, clock algorithms, and working-set-oriented policies make different assumptions about locality. Their practical behavior also depends on dirty pages, writeback, shared pages, memory-mapped files, and the operating system's page-cache policy.

A high page-fault count is therefore a symptom to interpret together with residency, reclaim, I/O, and workload phase.

5.16. Durability Boundary of File Operations

A successful application write does not always imply stable persistence on physical media. Data may still exist in a user-space buffer, kernel page cache, controller cache, or device-internal queue. Correct durability depends on the operating-system API, filesystem semantics, ordering barriers, and the storage device.

Applications that require crash consistency must define exactly which state must survive a process crash, kernel crash, or power loss and then use the corresponding persistence contract.

5.17. Paging and Segmentation Solve Different Problems

Paging and segmentation both divide an address space, but they provide different abstractions.

Paging divides virtual memory into fixed-size pages and physical memory into equally sized frames. Fixed-size placement avoids external fragmentation, although unused space in a final page can produce internal fragmentation. Page tables and the TLB are central to translating virtual page numbers to physical frames.

Segmentation divides a program into logical, variable-sized regions. Code, stack, heap, or data regions can be modeled as separate segments. Variable-size placement can leave scattered free regions and therefore create external fragmentation; historical systems could use compaction to recover large contiguous holes.

The distinction can be summarized as:

page    -> fixed size; suited to virtual-memory placement and mapping
segment -> variable size; suited to expressing logical regions and protection/sharing boundaries

The mechanisms can also be combined. A segment can select a page table, while the page number within that segment selects the physical frame. Such a design combines logical protection boundaries with the placement flexibility of paging, at the cost of additional translation and management complexity.

Modern general-purpose 64-bit systems rely primarily on paging for virtual memory, but segmentation remains important for understanding variable-size allocation, protection domains, and historical architecture design.

A process memory footprint differs from a managed runtime's heap. A Java heap limit is not a process-memory ceiling. Native allocations, thread stacks, memory mappings, direct buffers, and runtime metadata all affect the address space and memory accounted by the operating system. A container can reach a cgroup limit such as memory.max and be OOM-killed while significant free space remains inside the managed heap.

Do not conflate virtual address space (VSZ), resident set (RSS), major/minor page faults, swap pressure, and cgroup counters. Under memory pressure, page-fault activity can amplify latency without a corresponding increase in application CPU use. Correlate JVM GC observations with OS memory pressure and I/O wait on a common timeline.

Unit 6: File and Storage Management

6.1. File Abstraction

A file is a named persistent byte/record sequence or object as defined by the operating system. Metadata can include size, timestamps, ownership, permissions and storage mapping.

6.2. Directories

Directories map names to filesystem objects and create a hierarchical namespace. Path interpretation, mount points and links determine how names resolve to objects.

6.3. File Operations

Typical operations include create, open, read, write, seek, close, rename and remove. Open-file state can include current offset, access mode and references to filesystem metadata.

6.4. Storage and Filesystem Layers

A filesystem maps logical files to storage blocks. Internally it may maintain allocation metadata, free-space structures, directories, journals and caches.

6.5. FAT

The File Allocation Table family represents a file's cluster chain through entries in a central table. It is simple and widely interoperable but has scalability and robustness limits compared with more modern filesystems.

6.6. Inodes

Unix-style inode-based filesystems store file metadata and block mappings in inode structures. Directory entries map names to inode numbers/objects. The filename is therefore not inherently stored in the inode itself.

6.7. Hard and Symbolic Links

A hard link creates another directory entry for the same filesystem object/inode, subject to filesystem restrictions. A symbolic link stores a path-like reference to another name and can cross some boundaries that hard links cannot.

6.8. Allocation Methods

Contiguous allocation gives excellent sequential/random access but complicates growth and external fragmentation.

Linked allocation makes growth easy but random access poor unless indexing structures are added.

Indexed allocation stores block references in index structures, supporting direct access at the cost of metadata.

Modern filesystems use more sophisticated extent/tree variants, but these classical models explain the tradeoffs.

6.9. File Cache and Write Strategies

Operating systems cache filesystem data in memory. Writes may be buffered and acknowledged before durable storage unless an explicit durability mechanism is used. Application-visible write completion and physical persistence are therefore distinct guarantees.

6.10. Consistency

A crash can interrupt metadata updates. Journaling, copy-on-write or log-structured techniques provide different consistency/recovery strategies. They do not automatically guarantee application-level transactional consistency for arbitrary sequences of file operations.

6.11. RAID

RAID combines multiple storage devices for performance and/or redundancy. Examples:

• RAID 0 stripes without redundancy,
• RAID 1 mirrors,
• parity-based RAID distributes data/parity according to level.

RAID is not a backup: it does not protect against deletion, corruption, malware or every multi-device failure.

6.12. Atomic Replacement and Crash Durability of Files

Writing a temporary file and renaming it over a destination on the same file system may provide atomic visibility, but this does not guarantee persistence of the new data across a power failure. Data can remain in the page cache, while file content and directory entries depend on distinct synchronization steps. On POSIX-like local file systems, a common robust sequence is to write the temporary file, synchronize the file with fsync, rename it to the destination, and synchronize the parent directory with fsync. The guarantees remain constrained by the file system and storage-device flush semantics.

Even after fsync succeeds, remote storage, an NFS server, a RAID controller, and device caches may not share identical durability contracts. Journaling can protect metadata consistency without supplying application-level atomic transactions across multiple files. Experimental validation distinguishes a process crash from a machine power loss and verifies names, contents, and integrity constraints after recovery.

Unit 7: Security and Protection

7.1. Security versus Protection

Protection controls access among subjects and objects inside the system. Security is broader and includes authentication, attack resistance, cryptography, auditing and operational threats.

7.2. Authentication and Authorization

Authentication establishes an identity or credential claim. Authorization decides what that identity may do. Conflating them leads to designs where being logged in is incorrectly treated as permission for every resource.

7.3. Passwords

Passwords should not be stored as plaintext or fast unsalted hashes. Modern systems use salted password-hashing/KDF schemes with appropriate work factors. This is a later security correction to older textbook descriptions of password tables.

7.4. Access Matrix

An access matrix conceptually maps subjects to objects and permitted operations. It is usually sparse and therefore implemented through structures such as access-control lists or capabilities.

7.5. ACLs

An ACL attaches permissions to an object, listing which subjects/groups may perform which actions.

7.6. Capabilities

A capability is an unforgeable reference/token carrying authority to an object. Capability-based designs place emphasis on possession and delegation of authority rather than repeatedly consulting an object's ACL.

7.7. Protection Rings and Least Privilege

Hardware privilege levels create boundaries for code with different authority. The principle of least privilege gives each component only the permissions required for its task and only for the required duration.

7.8. Cryptographic Primitives

Symmetric encryption uses shared secret keys and is efficient for bulk confidentiality. Asymmetric cryptography uses key pairs and supports mechanisms such as key establishment and digital signatures. Hash functions map arbitrary input to fixed-size digests and are used in integrity constructions, signatures and many protocols.

These tools solve different problems; a hash is not encryption, and encryption without authentication is not automatically secure communication.

7.9. Malware

A virus typically attaches to host code/data and propagates when the host executes. A worm propagates through systems/networks without requiring the same host-file attachment model. Modern malware often combines several techniques, so these are conceptual categories rather than mutually exclusive boxes.

7.10. Lifecycle of Secrets in Process Memory

Encrypting a key at rest does not automatically protect copies present in process memory during use. Copies in objects, swap, crash dumps, and diagnostic logs must be considered. A compiler may optimize away a naive erasure call, and a managed-language runtime may retain relocated copies. Verified libraries, controlled lifetimes, and least-privilege process design are therefore complementary.

When consuming the operating system's CSPRNG, boot conditions, restarts, virtual-machine cloning, and entropy-source failures belong to the threat model. Memory controls do not replace a cryptographic security definition; the assumptions are developed in Cryptology.

Unit 8: Device Drivers

8.1. Role

A device driver translates the operating system's generic I/O model into the commands, registers, descriptors and interrupts of a specific device/controller.

8.2. Kernel Position

Many drivers execute in kernel space and therefore have broad authority. Some operating systems move selected driver components to user space for isolation. The exact boundary is an architectural choice.

8.3. Device Classes and Device Files

Character devices expose stream-like or operation-based access; block devices expose addressable storage blocks. Unix-like systems may represent device endpoints in the filesystem namespace, but the device file is an interface object rather than ordinary stored file data.

8.4. Resource Mapping

Drivers manage MMIO regions, interrupts, DMA buffers, clocks, buses and power state. Resource ownership must be acquired/released in a defined lifecycle, especially across hot-plug and failure paths.

8.5. MMIO

Memory-mapped I/O maps device registers into an address space. Reads/writes may have side effects and ordering requirements. Ordinary cached-memory assumptions cannot be applied blindly to device registers.

8.6. Interrupt Service and Deferred Work

Interrupt handlers acknowledge or capture urgent device state. Longer processing is commonly deferred to kernel worker mechanisms to limit interrupt latency.

8.7. DMA and Synchronization

DMA buffers require synchronization between CPU and device ownership. On non-coherent systems, cache maintenance can be necessary. Even on coherent systems, memory-ordering rules and descriptor publication must be correct.

8.8. Character and Block Drivers

Character drivers commonly implement operations resembling open/read/write/ioctl/poll according to the OS. Block drivers integrate with storage request queues, caching and filesystem layers.

Driver concurrency can arise from user threads, interrupts, DMA completion and kernel workers. Locks and atomic operations must match the execution context; sleeping locks cannot be used in every interrupt context.

Unit 9: Distributed Processing and Network Communication

9.1. Distributed Processing

A distributed system coordinates components across independent failure and timing domains. Unlike threads in one address space, remote peers can fail, messages can be delayed/reordered/lost, and network partitions can make state uncertain.

9.2. TCP/IP Layers

A practical layered view includes link, internet, transport and application protocols. Layering does not eliminate cross-layer performance effects, but it separates responsibilities.

9.3. IP

IP provides packet addressing and routing across networks. Delivery is best effort: packets can be lost, duplicated, delayed or reordered.

9.4. TCP

TCP provides an ordered reliable byte stream between endpoints. It uses sequence numbers, acknowledgments, retransmission, flow control and congestion control.

TCP does not preserve application message boundaries. An application protocol must frame records itself.

9.5. UDP

UDP provides datagrams with ports and checksum protection but no built-in delivery, ordering, retransmission or congestion-control semantics equivalent to TCP. Applications that need those properties must implement the required protocol behavior.

9.6. Ports and Sockets

A port identifies a transport endpoint within a host/protocol context. A socket API object represents local state associated with network communication.

A TCP server conceptually:

1. creates a socket,
2. binds a local address,
3. listens,
4. accepts connections,
5. exchanges bytes,
6. closes resources.

A client creates a socket and connects to the server endpoint before exchange.

9.7. Concurrent Servers

A server can handle multiple clients through processes, threads, event loops, async I/O or combinations. Concurrency architecture should be chosen according to connection count, work type, latency requirements and failure isolation.

9.8. DNS

DNS resolves hierarchical names to records such as addresses and service information. Resolution can involve caches and multiple servers. Applications must tolerate expiration, multiple answers and lookup failure.

9.9. Network Failure Model

A timeout does not prove that the remote operation failed. A request may have executed while its response was lost. This uncertainty is fundamental to distributed systems and is why retries require idempotency or deduplication semantics.

9.10. NFS and VFS

NFS exposes remote files through a filesystem-style interface. The semantics of caching, locking and failure differ from a local disk.

A Virtual File System layer gives the kernel a common interface across different filesystem implementations. Local and remote filesystems can therefore participate in one namespace while retaining implementation-specific behavior.

Unit 10: Real-Time Scheduling and the Modern I/O Path

An operating-system scheduler does not optimize the same objective for every workload. General-purpose systems often emphasize fairness and average responsiveness, whereas real-time systems care about whether a task completes before its deadline.

10.1. Priority Inversion

If a high-priority task waits for a lock held by a low-priority task, while medium-priority tasks repeatedly preempt the lock holder, the high-priority task is indirectly delayed by lower-priority work. This is priority inversion.

L: acquires lock
H: waits for the same lock
M: preempts L
H: is delayed by M as well

Protocols such as priority inheritance temporarily raise the effective priority of the lock holder so that the blocking interval can be bounded more tightly. They do not repair an unbounded critical section or a flawed concurrency design; they address a specific scheduling pathology.

10.2. Deadlines and WCET

Average execution time is not sufficient for real-time reasoning. A task's estimated worst-case execution time, period, and deadline must be considered together. A system may appear fast under ordinary load yet still violate a deadline because of a rare long pause.

release -> execution -> completion
              |
              +---- completion <= deadline ?

Tail latency, scheduler delay, interrupt latency, page faults, and I/O wait therefore belong to the same end-to-end timing budget.

10.3. The Modern I/O Path

A file read is not a single direct operation between an application and a storage device. A typical path is:

application
  ↓
system call / async submission
  ↓
VFS and file system
  ↓
page cache
  ↓
block layer / driver
  ↓
storage device

The page cache can reduce apparent read and write latency, while dirty-page writeback moves physical persistence to a later time. Durability operations such as fsync therefore represent a persistence boundary rather than merely another function call.

Asynchronous I/O interfaces can reduce the need to block one thread per operation. They do not increase the physical bandwidth of the device; they reorganize how waiting work and execution resources are managed.

10.4. Queueing and Backpressure

If the I/O queue is allowed to grow without a bound, throughput may initially look better while queueing delay and memory use rise sharply. Reliable systems combine admission control, bounded queues, and backpressure.

The operating-system boundary is clearer when three responsibilities remain distinct: the scheduler allocates CPU time, the I/O stack arbitrates device access, and the application controls how much work it admits. Latency failures are often misdiagnosed when these layers are treated as one mechanism.

10.5. Reading the Operating System Through Resource Flow, Isolation, and Progress Guarantees

The course topics can be connected as one resource-flow model:

CPU scheduling
      ↕
process/thread state
      ↕
virtual memory
      ↕
files and I/O
      ↕
devices and network

Protection cuts across every layer. Concurrency appears at every layer. Failure handling appears at every layer. The operating system is therefore not merely a collection of APIs; it is the component that makes controlled sharing, isolation and progress possible while hardware and applications compete for the same physical resources.

10.6. Historical Process and Scheduling Experiments

My early experiments with process priority and CPU affinity are preserved in the Köker Optimizer project, while Köker Zamanla! provides a historical example of timers, external-process execution and a command-line contract. Their original assumptions are now documented with current operating-system limits.

10.7. From OS Mechanisms to Linux Operations

Processes, memory, file systems, and I/O are treated here at the operating-system level. For service management, SELinux, NetworkManager, firewalld, DNF, LVM/XFS, Podman, and operational troubleshooting, see Red Hat Enterprise Linux System Administration.

10.8. Resource Exhaustion, Backpressure, and Fair Sharing

An operating system schedules more than CPU and memory. File descriptors, sockets, process-table entries, page cache, and I/O queues are finite resources as well.

An unbounded queue often appears first as a latency problem. If producers remain faster than consumers, backpressure must exist somewhere; otherwise memory growth, timeout cascades, and retry storms can destabilize the system.

A useful design question is what happens when a resource limit is reached: reject, queue, shed older work, or reduce quality. These are application-level service contracts built on operating-system mechanisms.

10.9. Verifying operating-system Claims at the Right Layer

POSIX, the Linux kernel, Windows NT, and individual file systems can implement similar abstractions differently. Before making a general claim, distinguish a standards-level contract from a kernel implementation detail or version-specific behavior.

Resource investigations may combine CPU use, context switches, major/minor page faults, storage queues, and file-descriptor consumption. Observation conditions and sampling intervals should be recorded because diagnostic tools also have cost.

Failure paths matter as much as normal execution: full disks, descriptor exhaustion, OOM, interrupted system calls, and partial I/O should have defined handling. Reliability claims are incomplete if they cover only the successful path.

10.10. The Operating-System Dimension of AI Workloads

An AI model is still a user-space workload, but model size, accelerator access, parallel execution, and sustained data movement can put unusual pressure on operating-system resources. Mathematical model quality and production execution behavior must therefore be evaluated at different layers.

A simplified inference path is:

request
↓
input preparation
↓
model runtime
↓
CPU / accelerator execution
↓
post-processing
↓
response

Each edge may involve a queue, memory allocation, file access, network I/O, or a device operation. Loading weights, allocating tensor workspaces, reading datasets, submitting work to an accelerator, and moving results between processes all depend on process, virtual-memory, and I/O mechanisms.

Memory pressure is a central concern. Weights, workspaces, activations, and the application heap compete for physical memory. Once paging begins, tail latency can deteriorate even when nominal compute capacity is high. An OOM condition is not necessarily explained by model size alone; concurrency, runtime buffers, page cache, and neighboring processes must be considered together.

NUMA placement can also become visible. If execution threads and frequently accessed memory reside on different NUMA nodes, remote access increases latency. Accelerator topology, CPU affinity, and interrupt placement can likewise matter in high-throughput services. The operating system abstracts hardware resources; it does not erase their physical cost.

Concurrency introduces another common error: more threads or processes do not necessarily produce more throughput. If the accelerator, memory bandwidth, or runtime is already saturated, additional work increases queue length. Average latency may remain deceptively acceptable while high percentiles grow. Admission control and backpressure are therefore part of inference reliability.

Isolation mechanisms such as cgroups can bound CPU and memory use; namespaces and permissions establish execution boundaries. Resource limits, however, do not create capacity. A model that cannot operate within a given memory budget will simply fail more deterministically under that limit.

When a GPU or NPU is present, the driver stack, device files, DMA, and user-kernel transitions enter the path. A single utilization percentage cannot prove where the bottleneck is. CPU preprocessing, copies, synchronization, or I/O may produce the same visible symptom.

Operating-systems theory does not explain how a model learns. It explains how a learned model becomes process, memory, I/O, and device work on finite hardware. Once deployed, throughput, latency, resource exhaustion, and recovery behavior become as real as model-level metrics.

10.11. Separate the Mechanism from the Observed Outcome

Operating-system questions become clearer when concepts are kept at the correct layer. A process owns an address-space and resource context, while threads inside one process may share memory. Sharing makes communication cheap but creates data-race and synchronization risks. Mutexes, semaphores, and condition variables are not interchangeable; the ownership and waiting semantics of the problem determine the appropriate primitive.

Virtual memory is not a synonym for physical memory. Access to a virtual address may involve the TLB, page tables, and sometimes a page fault before a physical page is reached. A page fault is not necessarily an error; demand paging can make it part of normal execution. Excessive page replacement, however, can turn useful computation into continual memory movement.

The classic deadlock conditions are mutual exclusion, hold-and-wait, no forced preemption, and circular wait. Prevention deliberately breaks at least one of these conditions. Starvation is different: the system as a whole may continue to make progress while a particular thread is repeatedly denied a resource.

Scheduling questions also require precise metrics. Turnaround time, waiting time, and response time answer different questions. Interactive workloads care strongly about first response; batch workloads may care more about completion time. High CPU utilisation is not proof of useful throughput either: the processor may be busy with lock contention, page-fault handling, or excessive system activity.

A useful first step is to locate the boundary of the problem: inter-process communication, threads within one process, virtual memory, file-system semantics, or device/I/O behaviour. Optimising the wrong layer often moves the symptom without removing the cause.

10.12. The Systems-Programming Layer: from System Calls to Resource Ownership

An operating-systems course explains kernel mechanisms; systems programming studies the contracts through which user-space programs access those mechanisms. Some computer-engineering programs teach these as separate courses. The distinction is useful: understanding how a scheduler works is not the same skill as using the error, lifetime, and ownership rules of fork(), execve(), read(), or mmap() correctly.

10.12.1. The system-call Boundary

A library function is not necessarily a system call, and a system call is not always visible directly in application code. User-space libraries may prepare parameters, buffer data, or provide compatibility behavior; the kernel becomes involved through the system-call boundary when privileged resources are required.

application
  ↓
standard library / runtime
  ↓
system-call interface
  ↓
kernel
  ↓
file system / network / process / device

The distinction also matters in performance analysis. A printf() call may remain in a user-space buffer; a write() may enter the kernel; data accepted by the kernel may still not be durable on the physical device at that instant.

10.12.2. File Descriptors as Common Resource Handles

On POSIX systems, a file descriptor is not limited to ordinary files. Pipes, sockets, terminals, and several kernel objects can be exposed through related read/write contracts. This unified model provides powerful composition in systems programming.

open/socket/pipe
      ↓
   fd = 3
      ↓
read / write / poll
      ↓
    close

The important property is not the numeric fd value but resource ownership and lifetime. Closing too early creates use errors; failing to close can leak resources for the lifetime of the process; after fork(), descriptors in different processes can refer to the same open-file description, which has consequences for file offsets and close behavior.

10.12.3. fork, exec, And wait Are Separate Operations

In the Unix process model, launching a program should not be treated as one opaque “create process” operation. fork() creates a new process context from the current process; the exec family replaces a process image with another program; the wait family allows the parent to collect child-process status.

parent
  │
  ├─ fork() ──→ child
  │              ↓
  │           exec(...)
  │              ↓
  │          new program
  │
  └─ wait(...) ← exit status

This separation is fundamental to shell pipelines, process supervision, and daemon or service managers.

10.12.4. Pipes, Sockets, and Shared Memory Are Different Forms of IPC

A pipe provides a byte stream and is especially simple for related processes and shell pipelines. A Unix-domain socket or network socket can provide bidirectional communication and different communication topologies. Shared memory can avoid copying payload through the kernel for each transfer by exposing a common memory region, but it increases synchronization responsibility. Message queues and semaphores provide still different communication and coordination contracts.

An IPC mechanism should not be selected only for throughput. Message boundaries, copying cost, failure isolation, process lifetime, backpressure, and cleanup after failure all matter.

10.12.5. Blocking, non-blocking, and Readiness Notification

When read() waits until data is available, the descriptor exhibits blocking I/O behavior. With O_NONBLOCK, the operation can return an appropriate status instead of waiting on an unready resource. select() and poll() monitor readiness across several descriptors; Linux epoll provides different scaling properties for large descriptor sets.

Readiness notification is not identical to true asynchronous I/O. A readiness model asks “which operation can proceed without blocking now?” A completion-based asynchronous model asks “when has the operation I started actually completed?”

10.12.6. Partial Reads, Partial Writes, and Interruptions Are Normal Cases

Systems code must not assume that one read() or write() call processes every requested byte. On sockets, pipes, and several device types, partial progress is normal. A call may also be interrupted by a signal. Reliable code explicitly manages the remaining byte count, the reported error, and whether retrying is semantically safe.

requested: 8192 bytes
write():    3072 bytes
remaining:  5120 bytes

These details connect operating-system theory to production code. Knowledge of processes, virtual memory, and file systems becomes operationally useful when user-space software obeys kernel contracts without leaking resources or creating unexpected blocking behavior.

10.13. Utilization, Saturation, and Queues in the Operating System

High CPU utilization alone does not identify the operating-system bottleneck. Growth in the runnable queue indicates processor contention, while page faults, memory reclaim, and block-I/O waits reveal different resource pressures. Process, kernel-thread, and device observations should be correlated over the same interval to locate actual waiting points. Provisioning decisions based solely on averages may conceal the source of saturation.

• Track run-queue growth separately from CPU utilization percentage.
• Separate page-fault and I/O wait from application CPU time.
• Measure scheduler policy effects on p95/p99 latency, not only on the mean.

CPU utilization is not a complete measure of runnable capacity. Under container CPU quotas, a service can be throttled for the remainder of a quota period even while physical cores elsewhere are idle. A CPU-percentage graph alone may therefore fail to explain high tail latency. Inspect run-queue length, context-switch patterns, cgroup cpu.stat throttling counters, and application p99 on a shared timeline.

High utilization is not identical to saturation: saturation means additional work must wait for a resource. CPU runnable queues, blocked disk I/O, network socket buffers, and application database-connection queues are distinct service centers. Without the arrival rate, service capacity, and waiting point for each center, a single “system is slow” metric cannot establish root cause.

10.14. Priority Inversion and Blocking Time

In fixed-priority real-time scheduling, the inability of a high-priority task to run is not necessarily caused by CPU saturation. A low-priority task may hold a shared lock needed by a high-priority task; intervening medium-priority tasks can delay the lock holder and prolong the high-priority task's response. This is priority inversion. Priority inheritance and priority-ceiling protocols can bound blocking under specified resource models, but they do not repair incorrectly scoped critical sections.

For periodic tasks on a uniprocessor, a common response-time model is the fixed-point relation \(R_i=C_i+B_i+\sum_{j\in hp(i)}\lceil R_i/T_j\rceil C_j\). Here \(C_i\) is worst-case execution time, \(B_i\) denotes blocking by lower-priority work, and \(T_j\) is the period of a higher-priority task. The result is conditional on the scheduling and locking assumptions. Interrupt handling, shared caches, and non-real-time device drivers require separate treatment rather than being silently absorbed into the equation.

A latency experiment should report observed worst-case response time measured from release to completion, blocking intervals, and missed deadlines rather than execution-time averages alone. An observed maximum is not a proven upper bound. A path that did not appear during a stress test cannot be regarded as excluded from a critical system's timing analysis.

Unit 11: Classical Concurrency Problems and Runtime Distinctions

Concurrency is not primarily a question of memorizing APIs. The central task is to separate shared-state protection, condition waiting, and progress guarantees.

11.1. Mode Switch Is Not the Same as Context Switch

A system call or interrupt can move execution from user mode to kernel mode without necessarily replacing the running thread.

A context switch changes the active execution context and requires saving and restoring processor and scheduler state.

mode switch ≠ mandatory context switch
context switch -> broader execution-state replacement

This distinction matters when reasoning about operating-system overhead.

11.2. Producer-consumer Contains Two Synchronization Problems

A bounded producer-consumer buffer requires:

1. mutual exclusion so concurrent access cannot corrupt the data structure,
2. condition synchronization so producers wait when full and consumers wait when empty.

A mutex alone can protect the container while still failing to define correct full/empty waiting behavior.

Semaphores or condition variables therefore represent a different responsibility from critical-section ownership.

11.3. Capacity and Shutdown Semantics of a Bounded Queue

Let C be queue capacity and q the number of occupied slots. A producer waits for q < C before insertion, while a consumer waits for q > 0 before removal. Element access and counter updates must follow the same synchronization contract; an atomic count alone does not make the underlying array or linked structure safe. The full-queue policy is part of the algorithm: block the producer, reject after a deadline, or apply an explicitly permitted drop rule. These choices provide different correctness guarantees.

A consumer waiting on a condition variable must recheck the predicate in a while loop after awakening. Another consumer may already have removed the last element, and wakeups need not establish that the condition is true. Shutdown also requires a defined protocol: stop accepting new work, wake blocked participants, distinguish an empty-but-open queue from a closed one, and decide whether admitted work is drained or canceled. Capacity control without such a protocol can still end in indefinite waits or silent work loss.

11.4. Readers-writers Requires an Explicit Policy

Readers can often proceed concurrently while a writer requires exclusive access.

There is no universally correct preference policy:

reader preference
writer preference
fair / bounded waiting

A design can be free of data races and still suffer starvation. Correctness therefore includes:

safety
+ liveness
+ fairness

11.5. Dining Philosophers and Resource Ordering

Dining philosophers models circular wait caused by acquiring multiple resources in inconsistent order.

A general mitigation is stable resource ordering:

lock(lower-id resource)
lock(higher-id resource)

The same rule applies to accounts, files, rows, or mutexes; it is not specific to the textbook metaphor.

11.6. Thread versus Process Isolation

Threads share one process address space. This can provide low-cost communication and shared caches, but failures in mutable memory can affect the whole process.

Separate processes provide stronger address-space and fault isolation at the cost of IPC and additional runtime resources.

The choice is therefore a trade-off among communication cost, fault containment, deployment, and state sharing.

11.7. Spin versus Block

A waiting thread may actively spin or be parked by the runtime/operating system.

spin:
  consumes CPU
  can be effective for very short waits

block / park:
  involves scheduler/runtime work
  releases CPU for longer waits

Practical synchronization primitives can combine both strategies.

11.8. Concurrency Review Checklist

Ask:

1. What mutable state is shared?
2. Which operation must be atomic?
3. Is there a condition that requires waiting?
4. Is lock acquisition ordered?
5. Can one participant starve?
6. Is a lock held across I/O?
7. Are locks released on cancellation and failure?
8. Would a process boundary provide necessary fault isolation?

This turns interview-style primitive selection into a systems reasoning problem.

11.9. Sleeping Barber: Rendezvous with a Bounded Waiting Room

The sleeping-barber problem tests more than mutual exclusion. A correct design must coordinate a bounded queue, the number of waiting jobs, an idle service resource, and wake-up ordering.

In the classical model there is one barber, one service chair, and N waiting chairs:

barber idle + customer present -> service starts
barber busy + waiting chair free -> customer waits
barber busy + waiting room full -> customer leaves
no customer -> barber sleeps

This is directly analogous to bounded queues in production systems. A basic invariant is:

0 <= waiting <= N

If the counter update and the sleep/wake protocol are not synchronized correctly, a lost wake-up can occur: work is available but the worker remains asleep, or a thread proceeds without a corresponding resource.

A semaphore-based formulation commonly uses separate signals for available customers and an available barber, while a mutex protects the waiting count. Correct semaphore counts alone are not enough; three properties must be considered together:

safety   -> can two customers use the same service resource concurrently?
liveness -> does the system progress when both work and capacity exist?
fairness -> can continuing arrivals postpone an older customer indefinitely?

The problem is therefore a compact model of bounded admission + rendezvous + service discipline, not merely an exercise in memorizing semaphore operations. Thread pools, connection pools, and bounded worker queues reproduce the same structure under different names.

A full queue is a policy decision as much as a capacity signal. Once a queue has a bound, define its behavior on reaching the limit: blocking, early rejection, work coalescing, and priority shedding have different semantics. A larger queue can absorb a burst, but cannot cure sustained arrivals above service capacity. Little's law, L = λW, applies to a stable system and a consistently defined population of work items.

Shutdown semantics matter as well. Producers must not remain blocked forever after consumers stop; accepted items need an explicit durability policy; and restarts need a documented approach to replay. Multiplying workers without adding processing capacity does not solve those protocol requirements.

11.10. Resource Ownership and Result Validity After Cancellation

A cancellation request does not prove that work has stopped immediately. A worker may continue computing, a network request may complete, or a callback may arrive after its resource has been closed. Cancellation therefore has two distinct contracts: eventual release of resources and invalidation of stale results. Generation-based validation is stronger for the second guarantee than relying on a cancellation flag alone.

An explicit ownership model binds resource acquisition, use, disposal and post-disposal notifications to one lifecycle. Listeners, timers, workers, file handles and network operations each need an identified owner. Double closure, partially initialized resources, and races between cancellation and natural completion are separate edge cases.

Deterministic tests use controlled barriers rather than arbitrary sleeps. Completion, cancellation and restart are exercised in different orders; resource counts should return to baseline and an old callback must not change the state of a new owner.

11.11. Shutdown and Failure Isolation in Bounded Queues

When a bounded queue is full, producer behavior must be specified in advance: block, wait for a bounded interval or reject in a controlled way. An empty queue and a stopped consumer are not equivalent states. A shutdown protocol stops new admissions, drains or cancels accepted work according to policy, and then releases resources. A poison-pill approach can deadlock when worker count and available queue capacity are ignored.

Failure of an optional stage must not unnecessarily halt independent work in a shared queue, provided their data and authorization contracts are genuinely independent. Waiting time, cancellations, lost work and shutdown duration should be assessed together.

11.12. Cancellation, Shutdown, and Resource Lifecycles

A cancellation signal does not guarantee that a task has terminated. Ownership of file descriptors, sockets, and locks must be defined together with rollback boundaries and the recognition of completed side effects. Shutdown has separate phases: stopping admission, draining or cancelling accepted work, and releasing resources.

Without bounded queues, backpressure may become unbounded memory growth. Rejecting, waiting, and shedding work have different latency and loss semantics. Queue depth, residence time, and active tasks must be measured separately.

References

• Abraham Silberschatz; Peter B. Galvin; Greg Gagne. Operating System Concepts. Wiley, 2012.

• Andrew S. Tanenbaum; Herbert Bos. Modern Operating Systems. Pearson, 2014.

• Brendan Gregg. Systems Performance: Enterprise and the Cloud. Pearson Education, 2014.

• David A. Patterson, John L. Hennessy. Computer Organization and Design: The Hardware/Software Interface, 6th ed. Morgan Kaufmann, 2020.

• IEEE; The Open Group. POSIX.1-2008. IEEE / The Open Group, 2008.

• Mor Harchol-Balter. Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press, 2013.
• IETF. RFC 9106, Argon2, 2021. https://www.rfc-editor.org/info/rfc9106