Computer Architecture: Instruction Sets, Pipelining and Memory Systems

The core sequence follows the 2015 computer architecture and organization coursework: the CISC/RISC distinction, instruction classes, program control, pipelining, vector processing, computer arithmetic, I/O, memory, and multiprocessor organization. RISC-V, modern out-of-order execution, virtual memory, NUMA, GPUs and accelerators, chiplets, CXL, and UCIe are treated as contemporary extensions of that foundation.

The main thread of the note is the path from a high-level program to machine instructions and then through registers, caches, main memory and I/O devices. Reading ISA, pipeline and memory decisions together with their effects on performance, power, reliability and software makes the architecture easier to reason about than memorizing one processor family.

I discuss the practical consequences of these concepts for AltiVec/VSX, NUMA, and CPU-GPU connectivity separately in my work on the IBM POWER9 AC922.

Unit 1: Computer Architecture, Performance, and Design Limits

Architecture and organization

Computer architecture covers both the contract visible to software and the hardware organization used to implement it.

The instruction set architecture (ISA) is the boundary visible to programmers and compilers. Instructions, registers, data types, addressing modes, privilege levels, the memory model, and exception behavior are part of that contract.

The microarchitecture is the internal implementation of the same ISA. Pipeline depth, issue width, cache organization, branch prediction, reorder buffers, the number of physical registers, and prefetchers are microarchitectural choices.

Two processors implementing the same ISA can have very different performance and power behavior. ISA compatibility does not imply an identical microarchitecture.

The basic components

It is more useful to view a computer as a dataflow and state-transition system than as a checklist of parts. Input brings information into the machine, memory holds instructions and data, processing transforms them, and output exposes the result. Inside the processor, two responsibilities are especially useful to separate:

datapath   → moves and transforms values
control    → decides which operation occurs and when

The datapath includes the ALU, register file, multiplexers, and interconnect. Control logic maps the meaning of an instruction to signals that drive those resources.

A contemporary SoC scales this model out: CPU cores, GPUs or NPUs, cache levels, memory controllers, a PCI Express root complex, DMA engines, and an on-chip network may coexist. Three questions remain useful at every scale: where is the data now, which unit will operate on it, and what state changes afterward?

Von Neumann, Harvard, and modified Harvard

A Von Neumann machine keeps instructions and data in a shared memory system and address space. This is flexible, but instruction and data traffic can compete for the same path, producing the classical Von Neumann bottleneck.

A Harvard machine separates instruction and data memories.

Modern general-purpose processors usually use a modified Harvard organization: main memory is unified, while L1 instruction and data caches may be separate and lower levels can be unified again.

From source code to hardware

Source code does not pass directly into a processor. Several layers, each with its own contract, sit in between:

C/C++ source
 ↓ compilation and optimization
assembly / intermediate representation
 ↓ assembler
object file
 ↓ linker
executable
 ↓ loader + operating system
virtual address space
 ↓ ISA
microarchitectural execution

The compiler is more than a syntax translator. Register allocation, constant propagation, dead-code elimination, inlining, loop unrolling, vectorization, and instruction scheduling can change both the dynamic instruction stream and its memory-access pattern.

There is therefore no one-to-one relationship between a line of source code and the work seen by the core. Architectural analysis may require reading assembly, object layout, virtual-memory placement, and microarchitectural behavior together.

Fetch-decode-execute

The simplest instruction cycle is:

fetch at PC
↓
decode
↓
read operands
↓
execute
↓
access memory if needed
↓
write result
↓
update PC

Modern cores overlap these steps across many instructions, can execute independent work out of order, and can speculate along control paths that have not yet been resolved. Architecturally retired state must still obey the ISA contract.

Chip manufacturing and yield

A microarchitecture ultimately has to fit inside a manufacturable and economical device. Wafer area, defect density, die size, test, packaging, cooling, and yield place physical limits on what can be integrated.

A larger die can hold more cache or cores, but it reduces the number of dies obtained from a wafer and exposes a larger area to defects. That trade-off influences architecture as well as manufacturing cost.

Chiplets are therefore more than a packaging fashion. Functions can be produced on different process technologies, smaller dies can improve yield, and product families can be assembled modularly. The price is additional die-to-die latency, power, package complexity, and coherence work.

A process-node label is likewise not a literal transistor dimension. Density, frequency, power, SRAM behavior, analog capability, yield, and cost must be considered together.

Moore's Law, Dennard scaling, and the power wall

Moore's Law summarized the long-term increase in transistor count. During the Dennard-scaling era, smaller transistors also enabled frequency growth without proportional power-density growth.

When that relationship broke down, increasing frequency indefinitely became impractical. The power wall helped drive multicore processors, heterogeneous computing, and domain-specific accelerators.

Dynamic CMOS power is approximately related to:

Pdynamic ≈ α C V² f

where α is switching activity, C the switched capacitance, V supply voltage, and f frequency. Voltage is especially important because of its squared term.

Static leakage power also matters and becomes increasingly important in advanced processes.

Power management

Common mechanisms include:

• dynamic voltage and frequency scaling,
• clock gating,
• power gating,
• core and accelerator sleep states,
• heterogeneous core selection,
• thermal throttling.

Maximum clock frequency is not automatically the best performance-per-watt point.

The CPU performance equation

A useful relationship is:

CPU time = instruction count × CPI × cycle time

or:

CPU time = IC × CPI / clock rate

Optimizing one term can worsen another. A more complex instruction may reduce instruction count while increasing cycle time or CPI. A deeper pipeline can increase frequency while increasing branch-misprediction cost.

Latency and throughput

Latency is the time for one operation to finish. Throughput is the amount of work completed per unit time.

A pipeline can raise throughput without reducing the end-to-end latency of one instruction by the same factor. GPUs are designed primarily for throughput; CPUs invest more aggressively in low latency for individual threads.

MIPS, FLOPS, and benchmark traps

MIPS counts millions of instructions per second. It is unreliable across different ISAs because the same task can require different instruction counts.

FLOPS is useful for floating-point workloads, but peak FLOPS says little about a memory-bound application.

A benchmark must resemble the target workload. SPEC CPU is useful for general-purpose CPU comparison; MLPerf standardizes important machine-learning scenarios. Results must be read together with compiler settings, memory configuration, power limits, and benchmark rules.

Geometric means are useful when combining normalized ratios because they avoid some distortions of arithmetic averaging across different reference scales.

Amdahl and Gustafson

If fraction P is accelerated by factor S, Amdahl's Law gives:

Speedup = 1 / ((1-P) + P/S)

Even a small non-accelerated portion can dominate total speedup.

Gustafson's Law emphasizes that available compute capacity can be used to solve a larger problem rather than only finishing a fixed problem sooner.

The memory wall

Processor execution capability improved much faster than DRAM latency. A cache miss can therefore cost hundreds of core cycles.

There is no single remedy:

• multilevel caches,
• prefetch,
• memory-level parallelism,
• wider memory channels,
• HBM,
• locality-aware software,
• NUMA-aware placement,
• accelerators that reduce data movement.

Reliability

Three terms should be separated:

• fault: the physical or logical cause,
• error: incorrect internal state,
• failure: externally visible violation of the specification.

Faults may be transient, intermittent, or permanent.

Parity detects classes of errors. ECC can do more; common server-memory SECDED codes correct a single-bit error and detect a double-bit error.

A reliable architecture includes detection, containment, reporting, and recovery, not merely an assumption that faults never occur.

Unit 2: Instruction Sets, CISC/RISC, and RISC-V

CISC and RISC

CISC and RISC are best treated as historical ISA design tendencies, not two immutable processor categories. CISC designs emphasized code density and doing more work per architectural instruction; RISC designs emphasized regular decode, explicit load/store operations, and simpler datapaths.

| Property | CISC tendency | RISC tendency | | --- | --- | --- | | Encoding | may be variable | generally regular | | Memory operands | common in many operations | separated into loads/stores | | Decode | can be complex | usually more regular | | Code density | can be high | can improve through compressed forms | | Microarchitecture | may be highly aggressive | may be highly aggressive |

The distinction no longer predicts performance by itself. Modern x86 cores can translate architectural instructions into internal micro-operations; RISC-V and AArch64 have substantial vector, atomic, cryptographic, and bit-manipulation facilities. Front-end design, execution resources, caches, branch prediction, the memory system, and the power budget matter more than the historical label.

x86, Arm, Power, MIPS, SPARC, and RISC-V

x86-64 combines strong backward compatibility with variable-length encoding.

AArch64 provides a regular 64-bit RISC ISA across mobile, embedded, and server systems.

Power ISA remains important in servers and high-performance computing and provides substantial vector and numerical capabilities.

MIPS is historically important to RISC development and computer-architecture education.

SPARC is well known for register windows.

RISC-V is an open-standard, modular, extensible ISA family.

A serious ISA comparison includes ABI, software ecosystem, privilege architecture, vector and atomic facilities, virtualization, code density, and implementation freedom—not only instruction count.

Register windows

SPARC-style register windows overlap part of a caller's output registers with a callee's input registers:

caller:  global | local | out
                         |||
callee:           in | local | out

The goal is to reduce register spill/fill traffic on procedure calls.

Windows are finite; deep call chains can still require spilling. RISC-V and AArch64 normally rely on ABI-defined caller-saved and callee-saved registers instead.

Why RISC-V matters

What distinguishes RISC-V is not simply a small instruction count. Its important property is the explicit separation between a base architectural contract and optional standardized extensions. An RV32I or RV64I foundation can be extended with multiplication and division, atomics, floating point, compressed encodings, bit manipulation, vector operations, or cryptography according to the target system.

That lets the same ISA family span very small controllers and high-performance processors without requiring every implementation to carry every feature.

An open ISA is not the same thing as an open core. A vendor can build a proprietary RISC-V microarchitecture, and different implementations can expose the same software-visible contract while using very different pipelines, caches, and execution engines.

Registers

RV32I/RV64I define x0 through x31.

x0 always reads as zero and discards writes. This simplifies several common operations and pseudo-instructions.

ABI names provide usage conventions:

x0       zero
x1       ra
x2       sp
x5-x7    t0-t2
x8       s0/fp
x10-x17  a0-a7

The architectural register number and ABI role are different layers: the ISA defines the register, the ABI defines its conventional role.

Why no general flag register?

RISC-V base integer code does not rely on a general arithmetic flags register. Conditional branches directly compare register operands.

This reduces hidden state dependencies and can simplify dependence tracking. Other ISAs make different, equally legitimate tradeoffs.

Instruction formats

Base RISC-V uses R, I, S, B, U, and J formats:

R  register-register
I  immediate/load/jalr
S  store
B  conditional branch
U  upper immediate
J  jump

Regular field placement simplifies decode and immediate generation.

The base encoding is regular, but compressed instruction extensions can improve code density. Modern RISC should not be reduced to the slogan that every instruction always has the same length.

Instruction classes

Common classes include:

• loads and stores,
• integer arithmetic,
• logic and bit manipulation,
• shifts,
• comparisons,
• control transfer,
• atomics and synchronization,
• floating point,
• vector/SIMD,
• system and privileged operations,
• cryptographic operations.

Addressing modes

Common addressing forms include immediate, register, direct, register indirect, base plus offset, indexed, and PC-relative forms.

RISC-V loads and stores primarily use base-plus-offset:

lw t0, 16(sp)

with:

EA = sp + 16

PC-relative addressing is important for branches, relocation, and position-independent code.

Alignment and endianness

Alignment places objects at boundaries suited to their natural width. Misaligned access can be slower, trapped, or handled in hardware depending on the architecture and platform.

Little-endian places the least-significant byte at the lowest address; big-endian places the most-significant byte there. Endianness changes byte layout in memory, not the mathematical value held in a register.

Program control

Control-transfer instructions update the program counter:

• conditional branches,
• unconditional jumps,
• calls,
• returns,
• traps/system calls.

RISC-V jal writes a return address and jumps. jalr obtains a target from a register plus an immediate.

Pseudo-instructions

An assembler can translate convenient syntax into one or more real instructions. A line such as:

mv a0, a1

can become an appropriate addi operation.

Performance analysis must distinguish source assembly lines from real machine instructions.

Status flags and processor status

Some ISAs generate arithmetic condition flags:

Z  zero
N/S sign
C  carry
V  signed overflow

They can drive conditional branches, multi-precision arithmetic, or privileged state. Base RISC-V integer code does not depend on a general condition-code register, while x86 and many historical ISAs use flags extensively.

The classical Processor Status Word (PSW) concept groups condition, privilege, interrupt-mask, and control state. Modern ISAs can distribute the same responsibilities across several control/status registers.

Procedures, stack frames, and ABI

A call normally:

1. places arguments,
2. preserves a return address,
3. saves required registers,
4. allocates a stack frame,
5. executes the callee,
6. restores state and returns.

Recursion works naturally because each invocation can have a separate frame.

The ABI defines argument registers, caller/callee saved sets, return values, stack alignment, data layout, and calling conventions.

Compilation, assembly, linking, loading

Object files contain more than instructions. They carry code, data, symbols, and relocation information.

Typical sections include:

.text
.rodata
.data
.bss

The linker resolves symbols and relocations. The loader creates a virtual address space and maps the program before transferring control to its entry point.

Interrupts, exceptions, and privilege

An interrupt is generally asynchronous and externally triggered. An exception is synchronous with the executing instruction. A trap can refer to a broader exception class or an intentional software transition.

RISC-V privilege levels include:

U  User
S  Supervisor
M  Machine

with optional hypervisor facilities.

ecall requests a service from more privileged software. CSRs such as trap vectors, saved PCs, and cause registers support trap handling.

Privilege is a hardware foundation of operating-system isolation.

Unit 3: RISC-V Programming, Compilers, and Data Layout

Reading assembly

Writing assembly is useful; reading compiler output is more broadly useful.

A typical investigation can compare:

gcc -O0
gcc -O2
objdump -d

-O0 tends to preserve source structure. Optimized builds may apply constant folding, inlining, register allocation, loop transformations, vectorization, and dead-code elimination.

Register arithmetic and memory access

RISC-V arithmetic generally operates on registers:

add  t0, t1, t2
addi t0, t1, 16
sub  t3, t4, t5

Loads and stores select width and signedness:

lb   t0, 0(a0)
lbu  t0, 0(a0)
lh   t0, 0(a0)
lw   t0, 0(a0)
sw   t0, 0(a0)

Signed loads sign-extend; unsigned loads zero-extend.

Arrays and locality

Sequential traversal:

A[0], A[1], A[2], ...

uses cache lines well.

Large matrices can behave very differently depending on whether the program walks rows or columns. The arithmetic operation count can be identical while cache and TLB behavior differ dramatically.

Structures and padding

C structures can contain padding inserted for alignment. The sum of field sizes is therefore not guaranteed to equal sizeof(struct).

Field order affects cache footprint, memory bandwidth, ABI compatibility, and false sharing.

Linked lists and pointer chasing

A linked list does not guarantee contiguous placement. Pointer chasing can reduce spatial locality, defeat prefetching, and serialize memory latency.

Two algorithms with similar asymptotic complexity can behave very differently on real memory hierarchies.

Recursion and the stack

Each recursive call can allocate a frame containing local state and return information. Deep recursion therefore consumes stack and creates call/return overhead.

Tail-call optimization can eliminate some frames when language, ABI, and compiler conditions permit it.

Loop unrolling

Loop unrolling reduces loop-control overhead and can expose more instruction-level parallelism.

It can also increase code size, register pressure, and instruction-cache pressure. More unrolling is not automatically better.

Auto-vectorization

Compilers can transform independent loop iterations into SIMD/vector operations.

Vectorization can be blocked by alias uncertainty, loop-carried dependencies, irregular memory access, unknown trip counts, or required exception semantics.

Compiler optimization reports are valuable because they explain why a loop was or was not vectorized.

Memory-mapped I/O

MMIO maps device registers into the normal address space:

load/store
   ↓
address decode
   ↓
RAM or device

Device mappings can have side effects and special caching/ordering rules.

Compiler volatile semantics and architectural memory barriers solve different problems. volatile is not a replacement for multicore synchronization.

UART

A simple UART driver can poll a status register and write a transmit-data register when ready, or use interrupts.

This small example connects CPU instructions, MMIO, device timing, and interrupt handling.

Toolchain

Useful RISC-V tools include assemblers, linkers, objdump, debuggers, teaching simulators such as RARS, system emulators such as QEMU, HDL tools, and FPGA flows.

An ISA interpreter teaches semantics; a pipeline simulator exposes hazards; a cache simulator makes locality and mapping behavior measurable.

Unit 4: Digital Logic, Datapaths, and Processor Control

From gates to a processor

At the lowest level, processors are built from components such as:

• AND/OR/XOR/NOT and universal gates,
• multiplexers and demultiplexers,
• encoders and decoders,
• flip-flops,
• registers,
• counters.

Boolean algebra, De Morgan transformations, Karnaugh maps, and Quine-McCluskey illustrate how logically equivalent functions can be implemented with different area and delay costs.

Combinational and sequential logic

A combinational circuit depends only on current inputs. A sequential circuit also carries state.

combinational: y = f(x)
sequential:    next_state = f(state, x)

Flip-flops are basic state elements; registers combine them into wider state.

Mealy and Moore machines

A Moore output depends only on state. A Mealy output depends on both state and current input.

Processor control units, cache controllers, and bus/device protocols can all be modeled as finite-state machines.

Datapath

A simple RISC-V datapath contains:

PC
↓
instruction memory
↓
decode / register file
↓
immediate generator
↓
ALU
↓
data memory
↓
write-back mux
↓
register file

Branches and jumps create multiple candidate next-PC values selected by control logic.

Register file

A simple integer pipeline often needs two simultaneous source reads and one destination write. A multiported register file supports that behavior, but port count increases area, wiring, power, and access time.

Wide superscalar cores therefore require much more complex physical register structures than the architectural register set suggests.

ALU

The ALU performs integer addition, subtraction, logic, comparison, and shifts. A comparison can use dedicated logic or be derived from subtraction, depending on timing and area tradeoffs.

Single-cycle processor

A single-cycle processor forces every architectural state transition into one clock interval. This makes the relationship between datapath and control unusually clear, which is why the model is useful for teaching. Timing, however, is dominated by the longest instruction path.

A register-register add may need only register access and an ALU, while a load traverses instruction fetch, decode, address generation, data memory, and write-back. If the clock is sized for the load path, simpler instructions inherit the same long period.

The design is therefore a reference model rather than a realistic organization for a modern high-performance core.

Critical path

A synchronous timing constraint is approximately:

Tclock ≥ Tcq + Tlogic,max + Tsetup + clock uncertainty

The longest register-to-register path, rather than gate count alone, constrains clock frequency.

Multi-cycle processor

A multi-cycle design divides one instruction into several shorter state transitions. The same ALU or memory interface can be reused at different times, and an instruction need not pass through work that it does not require.

Arithmetic operations can finish without a data-memory phase, while a store has no register write-back. This can shorten the clock and reduce replicated hardware.

The cost is control complexity: the processor must respond not only to the opcode but also to which execution state it currently occupies. Intermediate registers and a finite-state controller become part of the datapath organization.

Hardwired and microprogrammed control

Hardwired control generates signals directly from opcode fields and state. It is fast but becomes harder to manage as behavior grows.

Microprogrammed control maps an architectural instruction to a sequence of microinstructions stored in a control store.

Horizontal microcode uses a wide control word with substantial explicit parallelism. Vertical microcode uses denser encoding and additional decoding.

Modern x86 processors can still use microcode for complex or uncommon instruction behavior. Microcode updates can correct selected processor behaviors after manufacture.

ISA versus implementation

The ISA may state only:

rd = rs1 + rs2

That same operation can be implemented in a single-cycle teaching core, a five-stage pipeline, or a wide out-of-order machine. The contract stays the same while the implementation changes.

Unit 5: Computer Arithmetic and Numerical Datapaths

Integer representations

An n-bit unsigned value ranges from:

0 ... 2^n - 1

Two's-complement signed range is:

-2^(n-1) ... 2^(n-1)-1

Sign-magnitude and one's complement are historically important but contain two representations of zero. Two's complement provides a regular arithmetic structure.

Carry and overflow

Carry describes unsigned overflow beyond the bit width.

Signed overflow occurs when a two's-complement result falls outside the signed representable range.

They are not the same condition.

Extension

Sign extension replicates the sign bit of a narrower signed value. Zero extension fills high bits with zero for unsigned values.

This distinction appears directly in signed and unsigned load/compare instructions.

Ripple-carry, carry-lookahead, and carry-select

A ripple-carry adder chains full adders. It is compact, but carry delay grows with width.

Carry-lookahead derives generate/propagate information:

Gi = Ai Bi
Pi = Ai xor Bi
Ci+1 = Gi + Pi Ci

and computes carries more parallelly.

Carry-select precomputes results for both possible carry-in values and selects once the real carry arrives.

Parallel-prefix families such as Kogge-Stone distribute carry information in tree structures. They trade wiring, area, and power for lower delay.

Subtraction and shifts

Two's-complement subtraction can reuse an adder:

A - B = A + (~B + 1)

Logical shifts insert zeros; arithmetic right shift preserves the sign bit. Barrel shifters can perform large shifts in one combinational stage using wider mux networks.

Multiplication

Shift-and-add multiplication accumulates shifted partial products. It can be implemented iteratively with little hardware.

Booth recoding reduces work for signed operands containing runs of ones. Modified radix-4 Booth reduces the number of partial products further.

Array multipliers offer a regular physical layout. Wallace/Dadda-style trees compress many partial products in parallel before a final carry-propagate addition.

Division

Restoring division restores the previous remainder after an unsuccessful subtraction.

Non-restoring division combines that recovery with subsequent iterations.

SRT division uses redundant quotient digits and is useful in high-performance divider implementations.

Division is typically a longer-latency operation than addition and may not be fully pipelined.

Fixed point

Fixed-point formats attach an implied scale to integers. They offer predictable cost and compact storage for DSP, embedded systems, and quantized inference.

The scale is part of the software/hardware contract and must be managed explicitly.

IEEE 754

Floating point separates:

sign | exponent | significand

and defines normal values, subnormals, positive and negative zero, infinities, and NaNs.

Floating-point numbers are a finite approximation to real numbers; many decimal values are not exactly representable in binary.

Floating-point operations and rounding

Addition aligns exponents, adds/subtracts significands, normalizes, and rounds.

Multiplication conceptually uses:

sign = signA xor signB
exponent = exponentA + exponentB - bias
significand = significandA × significandB

followed by normalization and rounding.

Round-to-nearest, ties-to-even is a common default. Guard, round, and sticky information support correct rounding.

Floating-point addition is not associative; different reduction orders can produce different low-order results.

FMA

Fused multiply-add computes:

a × b + c

with one combined rounding rather than independently rounding the product first. It improves accuracy and is central to modern vector, DSP, GPU, and matrix hardware.

FP16, BF16, TF32, FP8, and mixed precision

FP16 reduces storage and compute cost.

BF16 preserves an FP32-like exponent range with fewer significand bits, making it useful for training workloads.

TF32 is a computation format used in NVIDIA tensor operations and should not be confused with a general software storage type identical to FP32.

FP8 families such as E4M3 and E5M2 provide still greater density but require explicit scaling and range management.

Mixed-precision training can use narrow matrix operations, wider accumulation, and high-precision master state to balance throughput and numerical stability.

INT8/INT4 quantization can further reduce model size and memory traffic for inference, at the cost of scale, calibration, and saturation management.

Unit 6: Pipelining, Instruction-Level Parallelism, and Out-of-Order Execution

Flynn's taxonomy

The classical classification by instruction and data streams is:

| Class | Instruction streams | Data streams | | --- | --- | --- | | SISD | one | one | | SIMD | one | many | | MISD | many | one | | MIMD | many | many |

SIMD describes an important form of data parallelism; MIMD is a useful model for multicore and multiprocessor systems. MISD is uncommon in general-purpose computing. Modern GPUs and heterogeneous systems do not always fit perfectly into one box, but the taxonomy remains useful.

Arithmetic pipelines

A floating-point adder can itself be pipelined:

compare exponents
→ align significands
→ add/subtract
→ normalize
→ round/pack

One operand pair can be normalizing while later pairs occupy earlier stages. Functional-unit latency can span several cycles while the initiation interval is one cycle.

Pipeline

Pipelining is a temporal partitioning of the datapath: different instructions occupy different pieces of the machine at the same time. The familiar teaching model uses five stages:

IF → ID → EX → MEM → WB

IF fetches, ID decodes and prepares operands, EX performs arithmetic/address/branch work, MEM accesses data memory when needed, and WB commits a register result.

The main benefit is not that one instruction suddenly has no latency. The benefit is overlap: once the pipeline is filled, more work can complete per unit time. Pipelining is therefore primarily a throughput technique.

Ideal execution time

For k stages, n instructions, and stage time t:

Tpipe = (k+n-1)t

For a long stream, ideal throughput approaches one completed instruction per cycle in a single-issue pipeline.

A useful model is:

CPI ≈ ideal CPI + stall cycles / retired instructions

Pipeline depth

More stages can reduce the logic per stage and permit a shorter clock period.

Costs include:

• pipeline-register overhead,
• clock distribution,
• bypass complexity,
• longer branch-misprediction recovery,
• more wakeup/select and control complexity.

Frequency must therefore be traded against energy and IPC.

Structural hazards

A structural hazard occurs when concurrent instructions need the same physical resource.

Solutions include separate instruction/data caches, multiple ports, resource replication, scheduling, or stalls.

RAW, WAR, and WAW

RAW is a true data dependence.

WAR and WAW are name dependences.

A simple in-order pipeline is dominated by RAW hazards. An out-of-order design also has to eliminate false name dependences so that independent instructions can proceed.

Forwarding and stalls

A newly produced value can be forwarded from an execution or memory stage to a consumer before it reaches the architectural register file.

producer result → consumer input

This removes many RAW stalls.

A load-use dependence can still require a bubble because cache data arrives later than a normal ALU result.

Control hazards and branch prediction

A conditional branch makes the next PC uncertain.

Static policies include always-not-taken, always-taken, and backward-taken/forward-not-taken.

A two-bit saturating predictor has strong/weak taken and not-taken states.

Local predictors learn behavior of one branch. Global predictors use the history of other recent branches. gshare can XOR global history with branch-address bits to index a prediction table.

A tournament predictor combines multiple predictors and learns which one is more reliable for a given branch.

BTB and return prediction

Even with a correct branch direction, target calculation can delay fetch.

A branch target buffer caches likely targets. A return-address stack predicts nested call/return targets.

Superscalar execution

A superscalar core can issue and execute multiple independent operations in the same cycle.

A nominal four-wide machine cannot guarantee four retired instructions every cycle. IPC is constrained by dependencies, cache misses, branches, front-end bandwidth, execution ports, and retirement width.

The front end

A modern front end can include:

branch prediction
→ fetch
→ instruction cache
→ predecode/decode
→ micro-op generation/cache
→ rename

If the front end cannot supply operations, execution units remain idle no matter how wide the back end is.

Micro-operations

Complex x86 instructions can be translated into one or more internal micro-operations.

A micro-op cache can reduce repeated decode work. This internal representation is a microarchitectural choice, not an architectural requirement.

Register renaming

The architectural name and physical storage location are separated:

architectural R1 → physical P17

A later write to R1 can allocate another physical register. This removes WAR and WAW name dependences.

RAW cannot be removed because it represents a real flow of data.

Out-of-order execution

Instructions can be decoded and renamed in program order while independent ready operations execute before older blocked operations.

This exposes useful work behind cache misses and long-latency operations.

Typical structures include:

• rename maps,
• physical register files,
• issue queues/reservation stations,
• load/store queues,
• reorder buffers,
• multiple execution units.

Reorder buffer and retirement

Out-of-order completion must not expose an incorrect architectural state.

A reorder buffer tracks program order, completion, and exception state. Results normally become architecturally visible at retirement/commit.

This supports precise exceptions: when a fault is reported, all older instructions can appear complete and all younger architectural effects absent.

Speculation

A core may execute instructions beyond an unresolved branch.

If the prediction is right, useful latency is hidden.

If it is wrong, younger speculative work is squashed, rename/ROB state is recovered, and fetch restarts at the correct target.

Speculation must preserve architectural correctness, although its microarchitectural side effects matter for security.

Memory disambiguation

A younger load can be blocked by an older store whose address is not yet known.

High-performance cores can predict that the operations do not overlap and execute the load early. If the prediction is wrong, replay is required.

This improves memory-level parallelism.

Load/store queue

The LSQ tracks memory operations, addresses, store-to-load forwarding, ordering, and detected violations.

It is central to both high performance and enforcement of the ISA memory model.

Delayed branches and delayed loads

Older RISC designs such as classic MIPS and SPARC exposed some branch or load latency as architectural delay slots.

Compilers attempted to fill those slots with useful work.

Modern deep dynamic pipelines generally keep this latency inside the microarchitecture. Base RISC-V has no branch delay slot.

SMT

Simultaneous multithreading allows multiple hardware threads to share a core's execution resources.

One thread can use resources while another waits on memory.

Tradeoffs include shared-cache and TLB pressure, execution contention, variable single-thread performance, and security/isolation considerations.

Unit 7: Vector Processing, GPUs, and Domain-Specific Accelerators

Data-level parallelism

A loop such as:

C[i] = A[i] + B[i]

can operate independently on many elements.

This is the basis of SIMD and vector processing.

SIMD and vector-length-agnostic ISAs

Fixed-width SIMD uses registers such as 128, 256, or 512 bits wide.

A vector-length-agnostic architecture such as the RISC-V V extension allows software to adapt to an implementation's vector length instead of hardcoding one physical width.

This helps portable vector software survive different implementations.

Strip mining

A problem larger than the active vector length is processed in chunks:

while remaining > 0:
    VL = min(remaining, suitable_hardware_VL)
    vector_operation(VL)
    remaining -= VL

This is fundamental to vector-length-agnostic programming.

Chaining

If early results from one vector operation can feed the next vector unit before the entire vector completes, the operations are chained.

V1 = A + B
V2 = V1 × C

The consumer begins as soon as initial producer elements are available.

Memory banking and interleaving

A vector execution unit requires matching memory bandwidth.

Consecutive addresses can be spread across independent banks:

address 0 → bank 0
address 1 → bank 1
address 2 → bank 2
...

Independent banks permit overlapping accesses. Unfortunate relationships between access stride and bank count can create bank conflicts.

Masks, gather, and scatter

A vector mask selects active elements and can replace some short control-flow branches.

Contiguous vector access is the easy case.

Gather loads elements from different addresses. Scatter writes to different addresses. Irregular patterns can stress caches, TLBs, and memory systems.

SIMD versus SIMT

GPUs often use a SIMT execution model: many software threads are grouped into hardware execution groups.

NVIDIA uses the term warp; AMD uses related wavefront concepts.

GPUs invest in many arithmetic lanes, high memory bandwidth, and many resident threads to hide latency.

CPUs invest more aggressively in large caches, branch prediction, and out-of-order machinery to minimize the latency of individual threads.

Warp divergence

If threads in an execution group take different branch directions, paths may execute separately under masks.

Effective lane utilization falls when control flow diverges heavily.

Memory coalescing

Neighboring threads accessing neighboring addresses allow memory transactions to be combined.

Irregular access wastes bandwidth and can create more transactions. This is the GPU-scale expression of the same spatial-locality principle seen in CPU caches.

GPU memory hierarchy

Depending on the architecture, relevant storage can include:

• registers,
• shared/local scratchpad memory,
• L1,
• L2,
• global device memory,
• constant/read-only storage.

Software-managed scratchpads can provide predictable low latency when data reuse is explicit.

HBM

High Bandwidth Memory stacks DRAM close to a processor or accelerator through advanced packaging and very wide interfaces.

It targets aggregate bandwidth rather than only capacity.

In AI and HPC systems, HBM bandwidth and capacity can be as important as arithmetic-unit count.

Tensor cores and systolic arrays

Dense workloads are dominated by operations such as:

D = A × B + C

Tensor-style execution units perform many multiply-accumulate operations on small matrix tiles.

Systolic arrays move data rhythmically among neighboring processing elements to increase reuse and reduce global data movement.

NPU, FPGA, and ASIC

NPUs specialize in neural-network operators, tensor data movement, quantized arithmetic, and activation functions.

FPGAs configure a spatial datapath from LUTs, flip-flops, block RAM, DSP blocks, and programmable interconnect. They can provide deterministic pipelines and custom bit widths, at the cost of more specialized development.

ASICs maximize specialization and efficiency but carry high non-recurring engineering cost and long design cycles.

Roofline model

A kernel is constrained by either computational throughput or data movement.

Arithmetic intensity is:

operations / bytes transferred

Low intensity tends to be memory-bound; high intensity can become compute-bound.

Optimization should target the active ceiling instead of blindly adding arithmetic work or memory tweaks.

Supercomputers

An HPC machine is more than a fast CPU or GPU. It combines:

• compute nodes,
• DRAM/HBM,
• low-latency high-bandwidth interconnect,
• parallel storage,
• topology-aware scheduling,
• large-scale power and cooling.

Peak FLOPS does not capture communication, synchronization, and memory costs.

Unit 8: I/O, Interrupts, DMA, and High-Speed Devices

Why I/O is its own architecture problem

Device data rates differ by orders of magnitude. A keyboard, sensor, NIC, NVMe SSD, and GPU have very different timing, protocol, error, and bandwidth characteristics.

Controllers bridge these differences.

Device registers

A controller commonly exposes data, status, and command/control registers.

Status can indicate ready, busy, error, or interrupt-pending state.

System-bus concepts

The classical bus model separates address, data, and control signals.

Modern links such as PCI Express are packetized serial fabrics, but the conceptual distinction between what is addressed, what data is transferred, and how transactions are controlled remains useful.

Port-mapped and memory-mapped I/O

Port-mapped I/O uses a distinct I/O address space and special instructions.

Memory-mapped I/O places device registers in the memory address space and accesses them with normal loads and stores.

MMIO regions can have side effects, special caching attributes, and strict ordering requirements.

Serial and parallel communication

Parallel links transfer multiple bits simultaneously on separate wires.

Modern high-speed external and board-level interfaces are predominantly serial: PCI Express, SATA, USB, and Ethernet use high-speed serial signaling, clock recovery, encoding, and often multiple lanes.

FIFO buffering

A FIFO absorbs short-term rate differences between producer and consumer.

A bursty transmitter can fill the queue while a slower receiver drains it at its own rate.

Buffering smooths bursts but does not create infinite capacity; a full FIFO produces backpressure, dropped data, or stalls depending on the protocol.

System buses, strobes, and handshakes

A classical bus model separates address, data, and control paths. Modern PCIe-style interconnects are physically packetized serial fabrics, but those functional classes remain useful.

A simple asynchronous transfer can use a one-sided strobe to indicate valid data. It depends on timing assumptions.

A handshake is explicit in both directions:

source: data valid
sink  : accepted
source: deassert request
sink  : deassert acknowledge

This works more reliably across devices with different timing.

Synchronous and asynchronous serial communication

Synchronous transfer relies on a common clock or derived timing relationship.

UART-style asynchronous serial framing can use:

start
data bits
optional parity
stop

with both sides configured for a baud rate.

Modern high-speed external links are predominantly serial. PCIe, USB, SATA, and Ethernet use clock recovery, encoding, and multiple lanes to reach high bandwidth.

FIFO buffering

A FIFO absorbs short-term rate mismatch:

producer → FIFO → consumer

When it fills, the protocol must apply backpressure, stall, or drop data. Buffer capacity smooths bursts; it does not create throughput.

Character encoding

ASCII is a historical 7-bit code. Modern text systems depend on Unicode and encodings such as UTF-8.

UTF-8 preserves ASCII's first 128 code points but uses a variable number of bytes per character. Assuming one character equals one byte breaks I/O buffers, file formats, and string processing.

Daisy-chain and parallel priority

In a historical interrupt daisy chain, an acknowledge signal passes through devices in order, giving earlier devices fixed priority. It is simple and scales poorly.

Parallel-priority designs collect requests at centralized priority logic. Modern programmable interrupt controllers provide richer routing, affinity, and priority control.

Polling and interrupt-driven I/O

Polling repeatedly reads device status and is simple but can waste CPU time.

Interrupt-driven I/O lets the CPU perform other work until the device signals completion.

Polling is not inherently wrong. At very low device latency or extreme packet rates, interrupt overhead can exceed the cost of polling. High-performance network and storage systems often combine polling, interrupt coalescing, and hybrid techniques.

Interrupt priority

When multiple devices request service, the system needs priority, masking, and routing.

A historical daisy-chain design passes an acknowledge signal through devices in priority order.

A parallel priority design presents requests to centralized priority logic such as a priority encoder.

Modern programmable interrupt controllers provide much richer routing and affinity control.

DMA

Direct Memory Access allows devices to transfer blocks between device and memory without the CPU performing every word-sized copy.

A typical sequence is:

CPU prepares descriptors
↓
device/DMA engine transfers memory ↔ device
↓
completion or interrupt

The CPU still manages buffers, descriptors, policy, and completion.

Scatter-gather DMA and bus mastering

Scatter-gather descriptors describe multiple physical segments as one logical transfer.

Modern high-speed devices are bus masters: the device initiates memory transactions rather than asking the CPU to move each unit of data.

DMA addresses and IOMMU

CPU virtual addresses, CPU physical addresses, and device DMA addresses need not be identical.

An IOMMU translates device-visible I/O virtual addresses and enforces device isolation.

Benefits include virtualization, contiguous device address spaces over scattered physical pages, and containment of erroneous or malicious DMA.

DMA coherence

A DMA engine can update memory while a CPU retains an older cache copy.

Some platforms provide coherent I/O; others require explicit cache clean/invalidate operations.

Coherence and ordering remain different concerns. A coherent platform can still require barriers.

MSI and MSI-X

PCI Express devices can generate message-signaled interrupts through memory writes instead of dedicated physical interrupt pins.

MSI-X supports many vectors and is well suited to multiqueue devices that route different queues to different CPUs.

PCI Express

PCIe is a point-to-point packetized serial hierarchy built from root complexes, switches, endpoints, links, and lanes.

Lane counts such as x1, x4, x8, and x16 express link width.

As of August 2026, PCI Express 7.0 is the current approved base specification and defines 128 GT/s signaling. Signaling rate is not identical to application payload bandwidth because protocol overhead, encoding, link width, and traffic direction all matter.

PCIe transactions

Transaction-layer packets carry memory reads/writes, completions, configuration operations, and messages.

A posted write can proceed without an immediate completion; a read request requires returned data. This distinction is important to latency and queue-depth behavior.

SR-IOV

Single Root I/O Virtualization lets one physical PCIe function expose multiple virtual functions:

PF
├── VF0
├── VF1
└── VF2

VMs or workloads can obtain a more direct path to device queues.

The benefit is lower software datapath overhead. Costs include more complicated policy, migration, IOMMU configuration, and finite hardware resources.

NVMe

NVMe is designed around flash storage's low latency and high parallelism.

Submission and completion queues are placed in host memory, and doorbell registers notify the controller of new work.

Its many-queue model is fundamentally better suited to parallel SSDs than legacy single-queue storage interfaces.

NAND flash and FTL

NAND flash programs pages but erases larger blocks and has finite program/erase endurance.

The SSD controller's Flash Translation Layer handles logical-to-physical mapping, garbage collection, wear leveling, bad-block management, and over-provisioning.

SSD latency can therefore vary significantly even without mechanical movement.

TRIM and RAID

TRIM tells a flash device which logical blocks are no longer needed. It helps internal reclamation; it should not be interpreted as an immediate cryptographic erase guarantee.

RAID combines drives through striping, mirroring, or parity. RAID is an availability/performance mechanism, not a backup.

SmartNIC and DPU

Networking and storage platforms can offload virtual switching, encryption, storage protocols, packet classification, telemetry, RDMA, and tenant isolation.

A SmartNIC or DPU can contain its own CPU cores, memory, and accelerators.

Offload saves host CPU cycles but makes state ownership and debugging more complex.

Unit 9: Memory Hierarchy, Caches, DRAM, and Virtual Memory

Memory hierarchy and locality

A memory system is not one large uniform array. Each level occupies a different point in latency, capacity, energy, and cost:

registers → L1 → L2 → LLC/L3 → DRAM/HBM → persistent storage

Upper levels are small and close to execution; lower levels are larger and slower. The hierarchy does not make lower storage disappear. It exploits temporal and spatial locality so that a large fraction of accesses can be served near the core.

No single technology can simultaneously provide register-class latency, DRAM-scale capacity, and SSD-class cost per bit. The hierarchy follows from that physical trade-off.

CAM and associative memory

Content-addressable memory searches by content:

RAM: address → data
CAM: key → matching entry

Parallel comparisons are fast but expensive in area and power.

Useful applications include TLB/tag lookup, networking classification/routing structures, and selected cache or predictor lookup structures.

HDDs and magnetic tape

HDD access includes:

seek + rotational latency + transfer

so random access is much more expensive than sequential access.

Magnetic tape has poor random access but remains valuable for very high capacity, low cost per bit, and offline or long-term archives.

Working set, hit, miss, and AMAT

The working set is the actively used code and data over an interval.

A cache hit finds the requested line at the current level. A miss goes lower.

A simple average memory-access model is:

AMAT = hit time + miss rate × miss penalty

A small miss rate still matters when miss penalties are large.

Cache line

Caches transfer blocks rather than individual bytes.

A larger line can exploit spatial locality but also wastes bandwidth on unused data, increases pollution, and can worsen false sharing.

Direct-mapped, set-associative, and fully associative

A direct-mapped cache gives each block one location:

index = block_number mod number_of_sets

It is simple and fast but vulnerable to conflict misses.

An N-way set-associative cache lets a block occupy any of N ways within one selected set.

A fully associative structure permits placement anywhere and minimizes conflict misses but requires more expensive lookup.

Tag, index, offset

An address can be viewed as:

tag | set index | block offset

The offset identifies a byte within the line, the index selects a set, and the tag verifies identity.

CAM

Content-addressable memory searches by content rather than by address:

key → matching entry

Parallel comparison is fast but expensive in area and power.

CAM-like structures are useful in TLBs, networking tables, and selected high-speed lookup structures, not as general main memory.

Replacement and the 3C model

Replacement policies include LRU, pseudo-LRU, random, and other adaptive schemes.

Classical cache misses are divided into:

• compulsory,
• capacity,
• conflict.

Multicore systems add coherence-related effects.

Write policies

Write-through sends every cache write downward.

Write-back retains dirty data and writes it on eviction.

On a write miss, write-allocate fetches the line before modifying it; no-write-allocate can send the write to a lower level without allocating a line.

Write buffers allow stores to retire without waiting for every lower-level transaction.

Non-blocking caches and MSHRs

A non-blocking cache allows independent accesses to proceed while one or more misses are outstanding.

MSHR-like structures track outstanding miss addresses and waiting consumers.

This is a major source of memory-level parallelism.

Prefetch

Hardware prefetchers recognize sequential, stride, or more complex patterns and fetch lines before demand access.

Accurate prefetch hides latency. Bad prefetch consumes bandwidth and cache capacity and can hurt other cores.

Victim caches and cache inclusion

A small victim cache can retain recently evicted lines and reduce repeated conflict misses.

Multilevel hierarchies can be inclusive, exclusive, or non-inclusive/non-exclusive. Inclusion choices affect effective capacity and coherence/snoop design.

Separate instruction and data caches

Separate L1 I-cache and D-cache permit instruction fetch and data access in parallel.

Lower levels can be unified. This is a common modified-Harvard design.

SRAM and DRAM

SRAM is fast, does not require refresh, and uses more transistor area per bit; it is common in caches.

DRAM stores charge densely and requires refresh; it is common as main memory.

DRAM organization

A modern DRAM path can be viewed as:

memory controller
↓
channel
↓
DIMM/rank
↓
bank group/bank
↓
row
↓
column

Activating a row places it in a row buffer. A row hit can be cheaper than switching to another row.

Memory latency is therefore not one fixed number.

Memory controller and DDR

The memory controller schedules requests, handles read/write turnarounds, exploits bank parallelism, performs refresh coordination, and can implement QoS and ECC behavior.

DDR transfers on both clock edges. Higher transfer rates do not imply proportionally lower access latency.

HBM and ECC

HBM provides very wide interfaces and high aggregate bandwidth through advanced packaging.

ECC such as SECDED corrects selected bit errors but does not solve every device or system failure. Stronger RAS mechanisms can add chip-level protection, mirroring, retry, poisoning, and error reporting.

Persistent storage

HDD access contains seek time, rotational latency, and transfer time.

SSDs remove mechanical motion but introduce flash-management behavior such as FTL mapping and garbage collection.

Magnetic tape remains important for low-cost, high-capacity long-term archives despite poor random access.

Virtual memory

Virtual memory provides process isolation, protection, sparse address spaces, shared mappings, copy-on-write, memory-mapped files, and a mechanism for paging.

The MMU translates:

virtual address → physical address

Page tables and TLBs

A page table maps a virtual page number to a physical frame number and also stores permission and state bits.

Multilevel tables allocate lower-level structures only for populated address regions.

RISC-V provides paging modes such as Sv39, Sv48, and Sv57.

A TLB caches translations.

A TLB miss can trigger a page-table walk without causing a page fault. A page fault means the mapping is absent, invalid, or violates permissions.

Huge pages and replacement

Larger pages increase TLB reach and reduce page-table overhead, at the cost of fragmentation and larger allocation/migration units.

Classical page-replacement algorithms include FIFO, LRU, and Clock/Second Chance. Real operating systems use more sophisticated approximations and working-set signals.

FIFO can exhibit Belady's anomaly.

Memory protection

Page permissions commonly separate read, write, execute, and privilege state.

W^X policies avoid mappings that are simultaneously writable and executable where practical. Execute-disable mechanisms help keep ordinary data pages from being executed as code.

Unit 10: Multicore Systems, NUMA, Coherence, and Memory Ordering

Why multicore became dominant

Frequency scaling became increasingly constrained by power, heat, the memory wall, and limited instruction-level parallelism.

Additional transistors were therefore invested in cores, caches, accelerators, and memory controllers.

More cores only help software that can expose useful parallel work.

SMP, UMA, and NUMA

In UMA, memory-access cost is approximately uniform across CPUs.

In NUMA, memory is physically closer to particular sockets or nodes:

CPU0 ─ local memory0
  ╲
   interconnect
  ╱
CPU1 ─ local memory1

Remote access uses the interconnect and normally costs more latency and bandwidth.

A system can have plenty of aggregate RAM and still perform poorly because data is placed on the wrong NUMA node.

First touch and affinity

Many operating systems place a physical page near the CPU that first touches it.

Initializing all memory from one thread and later distributing computation across nodes can therefore create remote traffic.

Thread affinity can preserve cache locality and local-memory access, though excessive pinning can reduce the scheduler's flexibility.

Shared and distributed memory

A shared-memory multiprocessor communicates through common memory and requires synchronization, coherence, and a memory consistency model.

A distributed-memory system gives each node local memory and communicates explicitly, typically with messages.

HPC often combines both: shared memory within a node and MPI-like messaging across nodes.

Interconnect topologies

Possible structures include:

• shared bus,
• multiport memory,
• crossbar,
• multistage networks,
• ring,
• mesh,
• torus,
• hypercube,
• network-on-chip.

A shared bus is simple and easy to snoop but becomes a bandwidth bottleneck.

A crossbar permits many concurrent source-destination pairs but grows expensive with endpoint count.

Multistage networks reduce crossbar cost by composing smaller switching stages.

Mesh and NoC designs scale wiring more naturally across many-core chips, at the cost of hop-dependent latency.

Classical interconnection networks

Historically important structures include multiport memory, multistage networks, Omega/butterfly networks, tori, and hypercubes.

A 2^n-node hypercube connects each node to n neighbors; binary node identifiers that differ by one bit are adjacent.

These structures remain useful for understanding modern NoCs and fabrics: scaling endpoints requires balancing bandwidth, hop count, wiring, and arbitration cost.

Bus arbitration

A shared resource that can serve only one master at a time requires arbitration.

Policies include fixed priority, round-robin, age/dynamic priority, and traffic-class or real-time priority.

Fixed priority is simple but can starve low-priority masters. Round-robin improves long-term fairness. Practical fabrics often combine fairness with QoS and latency classes.

Synchronization

Shared state can require mutexes, semaphores, spinlocks, reader-writer locks, condition variables, atomics, and barriers.

The right mechanism depends on critical-section length, contention, fairness requirements, scheduling, and NUMA topology.

Atomic operations

Atomic read-modify-write operations include test-and-set, compare-and-swap, fetch-add, and load-reserved/store-conditional.

RISC-V's A extension provides atomic memory operations and LR/SC.

Atomicity does not by itself define all memory-ordering relationships.

Spinlocks

A simple spinlock repeatedly tries to acquire a cache line.

This can be efficient for very short waits because it avoids sleep/wakeup overhead.

Under high contention it can waste CPU and generate heavy coherence traffic. Backoff and queue locks can scale better.

False sharing

Two threads can update different variables and still interfere if those variables share one cache line.

cache line:
counter0 | counter1
core0       core1

The line can bounce among cores even though the program does not logically share the counters.

Padding, layout changes, and per-core data can reduce false sharing.

Cache coherence

Private caches can hold multiple copies of the same line.

Coherence provides properties such as propagation of writes and a consistent order of writes to one location.

Invalidate, update, snooping, directory

Write-invalidate protocols invalidate other cached copies before ownership is obtained.

Write-update protocols propagate new data more eagerly.

Small shared-fabric systems can snoop all coherence transactions.

At larger scale, directories track sharers and send messages only to relevant nodes.

MESI and MOESI

MESI stable states are:

M Modified
E Exclusive
S Shared
I Invalid

MOESI adds Owned.

Real controllers contain many transient states beyond these simple names. The acronym is a conceptual model, not the entire implementation.

Coherence versus consistency

Coherence asks how copies of one memory location remain consistent.

Consistency asks which orders of loads and stores to different locations can be observed across processors.

A system can be cache coherent and still implement a weak memory model.

Sequential consistency

Sequential consistency provides the intuitive model that all memory operations form one global interleaving consistent with each thread's program order.

It is easy to reason about but can restrict hardware and compiler reordering.

RISC-V RVWMO

RISC-V uses a weak memory-ordering model by default.

Portable concurrent software expresses required ordering through acquire/release semantics, fences, and atomics.

A program that appears to work only because one processor happened not to reorder operations is not a portable synchronization design.

Fences, acquire, and release

A fence constrains ordering among classes of memory operations.

Acquire prevents selected later operations from being observed before the acquire boundary. Release constrains earlier operations with respect to the release.

Stronger ordering everywhere can reduce performance; synchronization should express the ordering actually required.

Lock-free does not mean contention-free

A CAS loop can avoid blocking locks yet still serialize ownership of one cache line.

For a lock-free structure, important measurements include:

• retry rate,
• cache-line migration,
• backoff,
• NUMA placement,
• fairness,
• selected memory order.

Unit 11: SoCs, Chiplets, CXL, UCIe, and Heterogeneous Systems

SoC

A system-on-chip can integrate CPU cores, GPUs/NPUs, memory controllers, media engines, security blocks, and I/O controllers.

The on-chip fabric and power-management system coordinate them as one platform.

Limits of a monolithic die

A large die is constrained by yield, reticle size, cost, and the fact that not every IP block benefits equally from the newest process technology.

I/O and analog logic can be cheaper on an older process than high-density compute.

Chiplets

A chiplet design separates the system into dies such as:

compute die
I/O die
cache die
accelerator die

Benefits include better yield, process-node specialization, reusable IP, and easier product-family construction.

Costs include die-to-die latency, package power, protocol complexity, test, thermals, and package-level yield.

2.5D and 3D packaging

2.5D designs place dies side by side on an interposer.

3D stacking places dies vertically.

Shorter interconnects can increase bandwidth density and reduce energy per bit, while thermal extraction becomes harder.

UCIe

Universal Chiplet Interconnect Express standardizes package-level die-to-die connectivity.

As of August 2026, UCIe 3.0 supports 48 and 64 GT/s data rates together with expanded manageability and sideband features while preserving backward compatibility.

UCIe is a package-level chiplet interface, not an external expansion-slot standard.

PCIe versus UCIe

PCI Express is a board/system I/O fabric.

UCIe targets die-to-die communication inside a system-in-package.

UCIe can carry mappings associated with PCIe and CXL, but the physical deployment boundary is different.

CXL

Compute Express Link builds cache- and memory-oriented semantics on a PCIe-based physical ecosystem.

Its purpose extends beyond conventional DMA toward accelerator memory access, host/device coherence, memory expansion, and pooling.

As of August 2026, CXL 4.0 is the current specification generation.

CXL-attached memory

A CXL memory expander can add capacity into a host memory hierarchy.

Its latency and bandwidth do not have to match local DDR.

This creates a new memory tier:

local DRAM
↓
CXL-attached memory
↓
storage

Placement policy becomes another NUMA-like architectural decision.

Memory pooling

Fabric-attached capacity can be allocated more flexibly among hosts.

Pooling can reduce stranded memory but adds fabric hops, contention, device latency, and bandwidth constraints that software must understand.

Heterogeneous compute

A system containing CPU, GPU, NPU, and FPGA can assign different kernels to different engines.

Acceleration overhead includes more than kernel execution:

host-device transfer
synchronization
format conversion
memory placement
kernel launch

Amdahl's Law still applies. A 100× accelerator provides little total benefit if most execution time remains in unaccelerated code.

Unified and coherent memory

The phrase "unified memory" is overloaded.

It can describe a common physical pool, shared virtual addressing, page migration, or truly coherent CPU/GPU access. These are different guarantees and must be checked for the actual platform.

Unit 12: Security, RAS, Power, and Real-Time Behavior

Architectural security boundaries

Protection uses several layers:

ISA privilege
MMU/IOMMU
page permissions
interrupt isolation
virtualization
device isolation
microarchitectural state

Privilege alone is not a complete security design.

Speculation and side channels

A speculative instruction on a wrong branch path can be architecturally discarded while still affecting microarchitectural state such as caches.

The central lesson of Spectre-class attacks is that an architecturally rolled-back result can still leave timing information behind.

Mitigations can involve hardware, microcode, operating systems, and compilers: speculation barriers, predictor isolation, bounds-check hardening, and address-space isolation are examples.

Architectural correctness and side-channel resistance are different properties.

Precise exceptions

When an exception is reported, the machine should expose a well-defined instruction boundary.

Out-of-order processors use structures such as the reorder buffer to retire older instructions, discard younger effects, and present precise architectural state.

Parity, ECC, and RAS

Parity detects many bit errors but does not correct them.

ECC adds redundant information and can support correction. SECDED means single-error correction and double-error detection.

Protection can be applied to caches, memories, links, and selected register structures.

Platform RAS can report corrected errors, uncorrected errors, poisoned data, link failures, and thermal events.

The desired sequence is detection, containment, reporting, recovery when possible, and controlled shutdown when not.

Fault tolerance

Redundancy can be spatial or temporal:

• replicated hardware,
• lockstep cores,
• retry,
• ECC,
• mirrored storage/memory,
• replicated systems.

A spacecraft, a data-center server, and an automotive controller optimize for different fault models.

Thermal design and dark silicon

Power becomes heat. Hot spots constrain frequency, leakage, and long-term reliability.

Thermal throttling lowers performance to keep a device safe.

TDP is a product thermal-design class, not a guarantee that every workload always consumes exactly that power.

Because all available transistors cannot necessarily run at maximum activity simultaneously, some blocks remain inactive at a given time—one aspect of the dark-silicon problem.

Specialized accelerators are valuable partly because they can do a fixed task with much less energy than a general-purpose core.

Real-time systems

Real-time design is about meeting deadlines, not merely obtaining a low average latency.

An average of 1 ms is insufficient if occasional 100 ms outliers violate a 10 ms deadline.

Deterministic systems care about bounded interrupt latency, predictable memory behavior, priority scheduling, worst-case execution time, controlled frequency scaling, and resource isolation.

Safety and security

Safety protects against unintended faults causing harm.

Security protects against malicious behavior.

Some mechanisms help both, but the fault model and threat model are different.

Unit 13: Measurement, Simulation, and Architectural Analysis

Measure first

A useful sequence is:

define workload
↓
measure baseline
↓
find bottleneck
↓
change one variable
↓
measure again

Clock rate, cache size, or core count alone is not diagnosis.

Hardware performance counters

Processors can expose counters for cycles, retired instructions, branches, branch misses, cache behavior, TLB events, and implementation-specific stalls.

RISC-V cycle and instret illustrate the basic mechanism.

CPI = cycles / retired instructions
IPC = retired instructions / cycles

The measurement scope and multiplexing of counters must be understood before interpreting results.

CPI stacks

A conceptual CPI stack separates:

base execution
+ front-end stalls
+ branch misses
+ cache/memory stalls
+ resource contention

Real hardware events are not always perfectly separable, but the model is useful for deciding where time is lost.

Benchmark methodology

Good benchmarking uses representative data, accounts for warm-up, records frequency and power state, fixes compiler options, repeats tests, and reports variance.

Selecting only the fastest run is not scientific measurement.

SPEC and MLPerf

SPEC CPU provides standard general-purpose CPU workloads and reporting rules.

MLPerf standardizes important AI training and inference scenarios, including accuracy and workload constraints.

A raw TOPS number without model, precision, batch size, and accuracy target is not enough to compare AI systems.

Peak and sustained performance

Theoretical peak may be estimated from unit count, operations per cycle, and clock rate.

Sustained performance is lower when data supply, dependencies, occupancy, communication, or synchronization limits execution.

Roofline and profiling

The Roofline model asks whether a kernel is limited by compute throughput or by the memory-bandwidth ceiling.

A memory-bound loop is not fixed by adding more ALUs. A compute-bound kernel may not benefit much from another cache tweak.

Disassembly

Compiler output should be inspected for instruction count, load/store ratio, vector instructions, branches, calls, spills/reloads, alignment, and generated library calls.

Simple source code does not imply simple machine code.

Instruction, pipeline, cache, and predictor simulators

A useful learning progression is:

1. ISA interpreter,
2. pipeline simulator,
3. cache simulator,
4. branch-predictor simulator.

A cycle-by-cycle pipeline trace:

cycle | IF | ID | EX | MEM | WB

makes forwarding, stalls, and flushes visible.

Cache simulation varies capacity, line size, associativity, and replacement policy against the same trace.

Branch simulation can compare one-bit, two-bit, gshare, and tournament predictors while accounting for storage cost, aliasing, warm-up, and misprediction penalty.

RTL, testbenches, and FPGA prototypes

RTL separates combinational behavior from state transitions.

A testbench provides clock/reset, stimulus, expected results, and assertions.

Successful synthesis does not prove functional correctness.

An FPGA implementation of a small RISC-V core exposes timing closure, reset, clock-domain, metastability, and physical I/O concerns that are easy to ignore in an ISA simulator.

Unit 14: Fast Conceptual Review

ISA is not microarchitecture. The ISA is the contract; caches, pipelines, and OoO execution implement it.

Clock rate is not performance. Instruction count and CPI matter too.

Latency is not throughput. A pipeline primarily raises throughput.

CISC and RISC are not absolute modern categories. x86 uses internal micro-operations; RISC ISAs have rich extensions.

A pseudo-instruction is not necessarily one machine instruction.

Architectural registers are not necessarily physical registers. Renaming separates them.

RAW differs from WAR and WAW. RAW is a true data dependence; WAR/WAW can be removed through renaming.

Forwarding is not out-of-order execution.

Execution is not retirement. Speculative work can execute and later be discarded.

Branch prediction is a performance mechanism, not a correctness shortcut.

Carry is not signed overflow.

Floating point is not the real-number system. Representation is finite and rounded.

FMA is not simply two separately rounded floating-point operations.

FP16, BF16, TF32, and FP8 are different numerical tradeoffs.

SIMD and SIMT are different execution models.

GPU cores and CPU cores cannot be compared by raw core count.

Peak FLOPS is not application performance. Arithmetic intensity and memory bandwidth matter.

MMIO is not ordinary RAM even when it shares the address space.

DMA does not remove the CPU from I/O management. It removes per-word data movement.

The CPU MMU and IOMMU protect different requesters.

PCIe and CXL are not the same layer. CXL adds cache/memory semantics.

PCIe and UCIe target different physical domains. UCIe is package-level die-to-die.

A cache hit and a TLB hit are different events.

A TLB miss is not a page fault.

**Cache coherence and memory consistency are different problems.**

An atomic operation is not a fence.

A coherent system is not automatically data-race-free.

UMA and NUMA differ in access locality, not merely memory capacity.

False sharing can occur without logical variable sharing.

ECC is not backup.

A benchmark is not automatically representative of production.

More transistors do not automatically make a program faster. Power, memory, parallelism, and software determine whether they can be used.

Applied Connections

I use the SIMD, memory-hierarchy, NUMA and CPU/GPU data-movement concepts in this course as practical engineering tools rather than only architecture terminology. My IBM POWER9 AC922 work is one applied context where AltiVec/VSX, OpenMP, CUDA and heterogeneous data movement had to be considered inside the same compute problem.

Related concepts:

• SIMD
• Memory Bandwidth
• NUMA
• Perceptual Image Similarity with DCT and SIMD

These links connect architectural mechanisms to measured software behavior.

Scope Boundary with Computer Organization

The more foundational organization of processor components is kept in Computer Organization and Basic Processor Structure. This page takes the wider engineering view through ISA design, pipelines, memory hierarchy, NUMA and modern system architecture.

Mechanical-computation comparison

For a physical pre-electronic view of arithmetic state and carry mechanisms, my FACIT mechanical calculator examination provides a historical counterpoint.

From Architecture Diagrams to Observable Behaviour

Instruction sets, pipelines, and cache hierarchies are often taught separately, yet a running program encounters them as one latency chain. Fewer instructions do not automatically mean a faster loop: dependency depth, branch prediction, cache-line placement, and memory bandwidth can dominate the result. This is why architecture becomes more useful when read alongside system-level concepts such as P99 Latency, False Sharing, and CPU Affinity.

The distinction matters even more on multicore systems, where aggregate throughput can improve while an individual request waits longer. Runtime Optimization in Java Systems follows the same hardware effects through a managed-runtime workload.

How Architecture Appears in Software Performance

Instruction-set features, microarchitecture and the memory hierarchy become visible together in software performance. SIMD is more than an instruction-set feature; its benefit depends on data layout, alignment, cache behavior and compiler vectorization. A branch-prediction failure likewise costs more than one branch instruction because the pipeline must recover and dependent work is delayed.

For that reason, a performance investigation should correlate high-level code with cache misses, memory bandwidth, branch misprediction, instruction-level parallelism and vectorization. Even in managed runtimes such as those discussed in Runtime Optimization in Java Systems, processor and memory behavior does not disappear; it is observed through the JVM and JIT layers.

The memory wall, prefetching, and memory-level parallelism

As core execution capacity has grown, many workloads have become limited by data movement rather than arithmetic. Hardware prefetchers work well on regular patterns while pointer chasing defeats much of that advantage.

Latency can also be hidden by keeping multiple independent memory requests in flight. Dependent pointer chains restrict this memory-level parallelism.

Data layout is therefore architectural. Array-of-structures versus structure-of-arrays should be chosen based on cache-line use, SIMD opportunities, and which fields are consumed together.

Testing architecture-performance claims

IPC, CPI, cache misses, and memory latency are not individually equivalent to “speed.” The same program can hit different bottlenecks under another data layout, core count, frequency policy, or compiler configuration.

A microbenchmark should isolate the mechanism being measured. Verify that the compiler has not removed the work, that the dataset exercises the intended cache level, and that scheduler/OS noise is controlled. Hardware counters should be interpreted alongside architecture documentation.

Separating ISA guarantees from microarchitectural optimization makes conclusions more portable. The presence of an instruction does not imply identical latency or throughput on every processor.

The Architectural Form of AI Workloads

Modern AI computation exposes several architectural concepts at once: data-level parallelism, memory hierarchy, low-precision arithmetic, high bandwidth, and domain-specific acceleration. The relationship is bidirectional. Architecture constrains how efficiently a model can run, while large AI workloads drive new architectural structures.

Dense neural computation is often dominated by matrix multiplication:

C = A × B

The arithmetic count may be enormous, yet FLOP/s alone does not determine performance. If matrix tiles do not reach execution units on time, compute remains idle. Arithmetic intensity is therefore useful:

arithmetic intensity = operations / bytes moved

Tiling and reuse increase this ratio by keeping useful data in caches or local buffers. Kernel fusion similarly reduces unnecessary intermediate traffic.

CPUs exploit SIMD/vector execution, while GPUs provide many more parallel lanes and a throughput-oriented memory system. These are different execution models. Small batches, branch-heavy preprocessing, or latency-sensitive serial work may remain efficient on CPUs, while highly parallel tensor operations may favor GPUs. Heterogeneous scheduling should be measured rather than assumed.

Tensor or matrix units specialize common multiply-accumulate patterns. Domain-specific accelerators go further by organizing dataflow around local reuse. Systolic arrays, for example, move weights and activations through processing elements so that external-memory traffic is reduced.

Precision is another architectural intersection. FP32, BF16, FP16, INT8, and narrower formats trade numerical range and error against storage, bandwidth, energy, and compute density. A format supported efficiently by hardware is useful only if the model tolerates its numerical behavior.

Transformer workloads can have different bottlenecks from convolutional networks. Attention creates sequence-length-dependent intermediate data, while autoregressive inference retains key/value state across generated tokens. KV-cache capacity and memory bandwidth can therefore dominate even when peak compute is high. Hardware suited to training throughput may not be optimal for single-request low-latency inference.

NUMA placement and accelerator interconnect topology add another layer. Multi-device training and inference can become communication-bound; theoretical aggregate compute is unavailable if synchronization or data movement dominates.

Model and architecture metrics should therefore remain distinct:

model quality        → accuracy / loss / task metric
architectural outcome → latency / throughput / energy / memory

High utilization alone is not the objective; user-visible latency and resource cost per unit of work matter.

Computer architecture contributes more than “using a GPU.” It explains how algebraic model operations become a schedule of data movement, parallel execution, and bounded physical resources. AI workloads influence architecture for the same reason: repeated computational patterns justify new data paths, arithmetic units, and memory structures.

What each performance quantity actually measures

Clock frequency, instruction count, and cycles per instruction are different quantities. A simplified execution-time model is

CPU time = instruction count × CPI / clock rate

so a higher clock rate improves time only if the other terms remain comparable. A different ISA, compiler, or microarchitecture can change both instruction count and CPI; comparing GHz alone does not explain architectural performance.

Pipelining does not remove the dependencies of one instruction. It overlaps stages of different instructions to improve throughput. Data, control, and structural hazards limit that overlap. Forwarding can reduce some stalls but cannot remove true data dependence. Branch prediction improves the control path when predictions are correct and creates recovery cost when they are wrong.

Cache questions are not only capacity questions. Block size, associativity, set mapping, replacement policy, and the memory-access pattern jointly determine hit and miss behaviour. A working set can show strong locality under sequential access and behave very differently under strided or conflict-heavy access.

Amdahl's law places an important bound on optimisation:

S = 1 / ((1 - p) + p / s)

where p is the fraction improved and s is the speedup of that fraction. Making a small part of a program arbitrarily fast produces only limited total improvement. Architectural optimisation should therefore be evaluated by its contribution to end-to-end execution time, not only by the peak capability of the improved component.

References

• AMD. AMD64 Architecture Programmer's Manual. Current online release.

• Arm. Arm Architecture Reference Manual for A-profile Architecture. Current online release.

• Behrooz Parhami. Computer Architecture: From Microprocessors to Supercomputers. Oxford University Press, 2005.

• Bruce Jacob; Spencer W. Ng; David T. Wang. Memory Systems: Cache, DRAM, Disk. Morgan Kaufmann, 2008. DOI: 10.1016/B978-0-12-379751-3.X5001-2.

• Cengiz Uğurkaya; Osman Aliefendioğlu. Modern Bilgisayar Mimarisi. Papatya Bilim, 2026.

• Christian Bienia et al. “The PARSEC Benchmark Suite: Characterization and Architectural Implications.” PACT, 2008.

• Compute Express Link Consortium. Compute Express Link Specification 4.0. https://computeexpresslink.org/

• Daniel A. Jiménez; Calvin Lin. “Dynamic Branch Prediction with Perceptrons.” HPCA, 2001.

• David A. Patterson; David R. Ditzel. “The Case for the Reduced Instruction Set Computer.” ACM SIGARCH Computer Architecture News, 8(6), 1980.

• David A. Patterson; John L. Hennessy. Bilgisayar Mimarisi ve Tasarım. Turkish translation, 5th Edition. Nobel Akademik Publishing, 2021.

• David A. Patterson; John L. Hennessy. Computer Organization and Design: The Hardware/Software Interface. 6th Edition, RISC-V Edition. Morgan Kaufmann, 2020.

• Gene M. Amdahl. “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities.” AFIPS, 1967. DOI: 10.1145/1465482.1465560.

• Gordon E. Moore. “Cramming More Components onto Integrated Circuits.” Electronics, 38(8), 1965.

• IEEE. IEEE Std 754-2019: IEEE Standard for Floating-Point Arithmetic.

• Intel. Intel 64 and IA-32 Architectures Software Developer's Manual. Current online release.

• John L. Gustafson. “Reevaluating Amdahl's Law.” Communications of the ACM, 31(5), 1988.

• John L. Hennessy; David A. Patterson. Computer Architecture: A Quantitative Approach. 6th Edition. Morgan Kaufmann, 2019.

• Linda Null; Julia Lobur. The Essentials of Computer Organization and Architecture. 5th Edition. Jones & Bartlett Learning, 2019.

• M. Morris Mano. Computer System Architecture. 3rd Edition. Prentice Hall, 1992.

• Mark D. Hill; Michael R. Marty. “Amdahl's Law in the Multicore Era.” Computer, 41(7), 2008.

• Mehmet Bodur. Bilgisayar Organizasyonu: RISC Donanımına Giriş. TMMOB Elektrik Mühendisleri Odası, 2003.

• Norman P. Jouppi et al. “In-Datacenter Performance Analysis of a Tensor Processing Unit.” Proceedings of ISCA, 2017. https://doi.org/10.1145/3079856.3080246

• NVIDIA. CUDA C++ Programming Guide. Current online release.

• NVM Express. NVM Express Base Specification. Current release. https://nvmexpress.org/

• OpenPOWER Foundation. Power ISA. Current architecture specification.

• PCI-SIG. PCI Express Base Specification Revision 7.0, 11 June 2025. https://pcisig.com/

• Peter Mattson et al. “MLPerf Training Benchmark.” Proceedings of Machine Learning and Systems, 2020.

• Randal E. Bryant; David R. O'Hallaron. Computer Systems: A Programmer's Perspective. 3rd Edition. Pearson, 2016.

• RISC-V International. RISC-V Formal Memory Model / RVWMO. https://docs.riscv.org/

• RISC-V International. RISC-V Ratified Specifications Library, Unprivileged and Privileged ISA, 2026 editions. https://docs.riscv.org/

• RISC-V International. V Standard Extension for Vector Operations, Version 1.0. https://docs.riscv.org/

• Sarah L. Harris; David Harris. Digital Design and Computer Architecture: RISC-V Edition. Morgan Kaufmann, 2021.

• Scott McFarling. Combining Branch Predictors. DEC WRL Technical Note TN-36, 1993.

• Şirzat Kahramanlı. Bilgisayar Mimarisi. Nobel Akademik Publishing, 2006.

• SPEC. SPEC CPU2017. https://www.spec.org/cpu2017/

• UCIe Consortium. Universal Chiplet Interconnect Express Specification 3.0. 2025. https://www.uciexpress.org/

• Vivienne Sze et al. “Efficient Processing of Deep Neural Networks: A Tutorial and Survey.” Proceedings of the IEEE, 105(12), 2017. https://doi.org/10.1109/JPROC.2017.2761740

• William Stallings. Computer Organization and Architecture: Designing for Performance. 11th Edition. Pearson, 2022.