Robotics Engineering

Robotics engineering studies the interaction of mechanical structure, actuation, sensing, power electronics, embedded computing, real-time software, state estimation, planning, and control on the same physical system. Robot behavior is not merely the output of source code; measurement uncertainty, coordinate frames, latency, backlash, saturation, power limits, and communication state all contribute to the physical result.

These notes proceed from robot architecture through actuators and sensors, kinematics and dynamics, Jacobians, control, motion planning, embedded software, telemetry, human-robot interaction, simulation, reliability, artificial intelligence, localization, and SLAM. The units are organized around engineering problem classes and system contracts rather than around any particular robot brand or software product.

In robotics, a correct computation is not automatically a correct physical action. A perception result may be numerically accurate but stale; a motor command may have been transmitted without being applied; a geometrically valid trajectory may violate velocity, torque, or collision constraints. State, time, uncertainty, and physical limits therefore need explicit semantics at every layer.

The professional application context for some of these topics is documented separately on the Akınsoft and Akın Robotics experience page. The link is intentionally kept as context rather than allowing career narrative to dominate the course material.

Unit 1: Systems Perspective of Robotics Engineering

1.1. The Boundary Between Robots and Automation

Not every automated machine is a robot, and not every robot is fully autonomous. Robotic systems generally interact with the physical environment through sensing and action, produce programmable behavior, and can vary that behavior according to at least some changes in state or environment.

Autonomy is not one binary attribute. The same robot may execute:

• some actions directly under operator command,
• some functions semi-autonomously,
• some tasks automatically.

The more useful engineering question is therefore not “is this robot autonomous?” but which decision is made at which layer, under which constraints, and with which fallback behavior?

1.2. Sense-decide-act Loop

A robot can be described by the general loop:

environment -> sensors -> state estimation -> decision / planning -> control -> actuators -> environment

Each arrow is an engineering boundary. Converting a physical signal to a digital sensor value, timestamping it, filtering it, and delivering it to a higher layer are different problems. Likewise, a high-level request such as “extend the arm” cannot be translated directly into motor voltage, current, or PWM; kinematics, trajectory generation, and control lie between those levels.

1.3. Why Robotics Is Multidisciplinary

Robotics may depend simultaneously on:

• mechanical design and strength,
• electrical and electronic systems,
• sensing and measurement,
• motors, drives, hydraulics, and pneumatics,
• control theory,
• signal processing,
• embedded systems,
• real-time software,
• computer vision,
• speech processing,
• artificial intelligence,
• networking and communications,
• human-machine interaction.

This is the direct connection between robotics and mechatronics. Mechatronics teaches concurrent design of the physical and computational subsystems; robotics extends that integration into perception, behavior, and purposeful motion.

1.4. From Requirements to high-level Design

A robotics architecture should begin by separating what the system must do from how the system will implement it. A functional requirement describes observable behavior, operating conditions, and acceptance boundaries. Processor families, buses, motor drivers, and software modules are implementation choices used to realize that behavior.

If this distinction is lost, an early technology choice can begin to dictate the entire system. A more disciplined flow is:

mission / need
      ↓
observable system requirements
      ↓
allocation to subsystems
      ↓
interface contracts
      ↓
high-level design
      ↓
implementation
      ↓
verification and acceptance evidence

Robot requirements are not limited to motion targets. At minimum, the design should consider:

• operating modes and mode transitions,
• limits on speed, force, position, and repeatability,
• mission duration and energy constraints,
• environmental operating conditions,
• sensor and actuator availability,
• behavior under communication loss,
• transition to a safe state,
• restart and recovery behavior,
• serviceability and replaceability,
• acceptance criteria that prove mission completion.

The high-level design then makes visible how these requirements will be realized. For a robot, a block diagram of software modules alone is insufficient. Power distribution, computing hardware, motor drives, sensor interfaces, buses, timing, thermal limits, memory and peripherals, fault boundaries, and the operator interface belong to the same system view.

physical structure ── actuator/drive ── control
        │                                  │
        ├─ power / thermal limits          ├─ state / fault model
        │                                  │
        └─ sensors ── data bus ── compute ── task / HRI

The diagram is not a product architecture by itself; its purpose is to expose dependencies early. A more powerful actuator may require a larger drive and energy source, increasing mass. Increased mass can then change dynamics, mission endurance, and controller behavior. Robotics is full of such system-level feedback from decisions that initially look local.

Technical risks should therefore be attacked early. Whether a new sensor can meet the required accuracy, whether the compute platform can sustain worst-case load, or whether a mechanism can survive the expected duty cycle should be tested through focused prototypes and measurement before the rest of the system is built around an assumption.

Unit 2: Robot Architecture and Fundamental Subsystems

Robot architecture begins by separating required update rates for motion, sensing, and task planning. Fast mechanical feedback and high-level route decisions operate on different time scales. Intermodule data contracts should include coordinate frames, timestamps, and validity conditions. When a module stops supplying data, define explicitly whether the system may use a prior value, stop, or enter a safe state.

2.1. Mechanical Body and Kinematic Structure

The physical body carries loads, constrains joints, and defines the workspace. In manipulators, links and joints form a kinematic chain. In mobile robots, chassis geometry, wheels/tracks/legs, and mass distribution determine mobility.

Software cannot treat geometry as an implementation detail owned only by mechanical engineering. Joint limits, backlash, friction, transmission ratio, and inertia directly change control behavior.

2.2. Joints and Degrees of Freedom

Two fundamental joint types are:

• revolute joints,
• prismatic joints.

A revolute joint produces angular motion, while a prismatic joint produces linear displacement.

The degree of freedom of a robotic mechanism describes the number of independent motion variables. A free rigid body in three-dimensional space has six degrees of freedom:

• x, y, z translations,
• roll, pitch, yaw rotations.

The number of robot joints and the effective degrees of freedom in task space do not always have the same meaning. Constraints, closed kinematic chains, and singular configurations change the usable motion space.

2.3. End Effectors

The end effector is the part of a manipulator that performs the task directly on the environment. It may be:

• a two- or three-finger gripper,
• a vacuum gripper,
• an electromagnet,
• a welding tool,
• a paint tool,
• a screwdriver,
• a measurement probe.

Selection is more than a question of whether the tool can hold an object. Surface, mass, fragility, permitted gripping force, positional tolerance, and cycle time must be considered together.

2.4. Robot Classes

Common mechanical and application classes include:

• Cartesian robots,
• cylindrical robots,
• spherical/polar robots,
• SCARA mechanisms,
• articulated arms,
• parallel robots,
• wheeled or tracked mobile robots,
• legged robots,
• humanoid robots.

These classifications do not identify one universally superior architecture. Every geometry establishes a different trade-off among workspace, rigidity, speed, payload, control complexity, and cost.

Unit 3: Actuators, Drives, and Power Systems

Robot drive selection starts by calculating joint torque or linear force along the trajectory. Consider repetitive-cycle thermal load alongside peak demand. If backlash, compliance, or friction causes deviation from nominal kinematics, revise controller tolerances. Verify reachable motion, current limiting, and emergency-stop behavior experimentally.

3.1. Electric Motors

Robotic motion is frequently generated with electric motors. DC, BLDC, stepper, and servo systems serve different requirements.

Maximum speed alone is not a meaningful selection criterion. Relevant quantities include:

• continuous and peak torque,
• speed range,
• torque-speed curve,
• rotor inertia,
• efficiency,
• supply voltage,
• drive requirements,
• thermal limits,
• feedback resolution.

A motor capable of high no-load speed does not imply that the robot joint can safely operate at that speed. Gearbox behavior, load inertia, mechanical strength, and controller bandwidth impose additional constraints.

3.2. Servo Systems

A servo is more than a small model motor. In the general control sense, a servo system tracks a commanded quantity through feedback.

A position servo may form the loop:

target position -> controller -> motor drive -> motor/joint -> position sensor -> feedback

In servo control, transmitting a command and the joint actually reaching its target are different events. Higher-level behavior should not advance solely from a command queue; it should verify physical realization through feedback such as position, velocity, current/torque, and timeout state. “Command accepted,” “motion started,” “target reached,” and “faulted” are distinct states.

3.3. Transmissions

Motor rotation may be transformed through gearboxes, belts, chains, screws, harmonic drives, or other mechanisms in order to:

• reduce speed,
• increase torque,
• change direction,
• convert rotation to translation.

Higher reduction can reduce motor-side torque demand, but backlash, friction, compliance, and efficiency losses can degrade positioning and control.

3.4. Hydraulic and Pneumatic Actuation

Hydraulics can be appropriate for high-force systems, while pneumatics are common for fast and relatively simple linear tasks. Their control behavior differs from electric actuation: pressure, valve dynamics, compressibility, leakage, and flow characteristics become part of the model.

The fact that electric motors, hydraulics, pneumatics, electronics, and mechanics were studied together in my Sakarya University mechatronics education became much more concrete in robotics: the abstract software command “move” maps to very different physical dynamics depending on the actuator.

3.5. Power Budget

Energy is a bounded resource, particularly in mobile robots. The power budget includes more than motors:

• processors or GPUs,
• sensors,
• cameras,
• communication units,
• drives,
• audio systems,
• auxiliary electronics.

Motor current transients can pull down supply voltage and destabilize processors or sensors. A seemingly random software reset or communication error can therefore originate from power integrity rather than code.

3.6. From Power Budget to Energy States

A power budget is necessary for instantaneous load analysis, but mission endurance is governed by energy over time:

E = ∫ P(t) dt

For that reason, asking only for the robot's average power draw is incomplete. The same hardware can produce very different energy profiles across operating states. A system may, for example, distinguish:

ACTIVE
  full mission and motion functions available

READY
  no motion; sensing and essential communication remain active

LOW POWER
  only functions required for wake-up remain active

SAFE SHUTDOWN
  physical state is made safe before energy is removed

The names are product-specific. What matters is that each state clearly defines:

• which subsystems remain powered,
• which events trigger transitions,
• wake-up or reactivation latency,
• how much software and physical state is retained,
• whether control loops may be suspended,
• validity of sensor data during transitions,
• safe behavior under power interruption.

Energy management is not exclusively a power-electronics problem. Software scheduling, mission planning, and motion coordination affect consumption as well. A processor that never becomes idle, unnecessarily high sensing rates, joints that continuously generate holding torque, or motion plans with avoidable repetition all consume mission energy.

The opposite mistake is to slow time-critical control indiscriminately in the name of savings. Motor-control sampling, sensor freshness, and safety supervision are constrained by dynamics. A safer optimization sequence is:

define timing and safety constraints
                 ↓
identify unnecessary activity
                 ↓
coordinate hardware and software states
                 ↓
measure transition latency and physical effect
                 ↓
verify the resulting endurance gain

Motion itself is also an energy-management problem. Two trajectories reaching the same pose can differ in duration, peak torque, braking behavior, and coordinated joint motion. When mission requirements justify it, a planner may therefore include energy alongside distance, time, and risk.

A power interruption is also a fault-semantic issue. Automatically resuming the previous software task after reboot may be unsafe unless joint positions, held payload, sensor references, and communication state are re-established. Persistent software state must not be mistaken for verified physical state.

Unit 4: Sensors, Measurement, and Robot Perception

4.1. Internal and External Sensing

Sensors can be viewed functionally as two broad groups.

Proprioceptive sensing measures the robot's own state:

• joint position,
• velocity,
• motor current,
• temperature,
• battery state,
• orientation.

Exteroceptive sensing measures the external environment:

• cameras,
• lidar,
• ultrasonic sensors,
• force/torque sensors,
• proximity sensors,
• microphones,
• light sensors.

The distinction is functional rather than absolute. A camera may support both environmental perception and localization/calibration.

4.2. Measurement Is Not Reality

Every sensor should be interpreted together with questions such as:

• What is its resolution?
• What is its accuracy?
• Does it have bias?
• What does its noise look like?
• What is the sampling rate?
• What is the latency?
• Where does it saturate?
• When should the value be marked invalid?

Treating a measured number as physical truth without these semantics is one of the most dangerous abstraction errors in robot software.

4.3. Sampling, Digitization, and Aliasing

Many robot sensors begin with a continuous-time physical quantity while the control software operates on discrete samples. That transition involves more than ADC resolution:

physical quantity
      ↓
transducer / sensor
      ↓
analog conditioning
      ↓
anti-alias filter
      ↓
sampling + A/D conversion
      ↓
timestamp
      ↓
digital filter / state estimator

The sampling rate must be consistent with the bandwidth of the signal to be observed. For an ideal band-limited signal, the basic Nyquist-Shannon condition is:

f_s > 2 f_max

In a physical robot, merely exceeding the theoretical boundary is not a complete design rule. Sensor bandwidth, analog-filter transition region, mechanical vibration, electrical noise, and the phase/delay requirements of the controller must be considered together.

Aliasing is especially important: a high-frequency component can appear after sampling as a false component at a lower frequency. Once that ambiguity is created, a digital filter cannot reconstruct the lost information. Unwanted high-frequency content therefore has to be sufficiently attenuated before A/D conversion.

Other digitization errors include:

• quantization resolution,
• poor selection of the input range,
• saturation,
• sampling-time uncertainty,
• channel-switching latency,
• clock drift and jitter,
• lack of synchronization across sensors.

A label such as 100 Hz sensor is therefore not a complete data-quality specification. The time at which the physical quantity was sampled, transport latency, and the amount of history introduced by filtering can matter as much to the control layer as nominal sampling frequency.

4.4. Encoders

Rotary and linear encoders measure position and support velocity estimation. Incremental encoders report relative motion, while absolute encoders can preserve a coded position reference across power cycles.

Velocity estimated by finite difference may amplify measurement noise:

v[k] ≈ (x[k] - x[k-1]) / Δt

Filtering therefore trades noise reduction against delay. A smoother plot is not necessarily a better control signal if it describes an older physical state.

4.5. IMUs and Orientation

IMUs combine sensors such as accelerometers and gyroscopes for motion estimation. Gyroscope integration drifts over time; accelerometers observe both linear acceleration and gravity; magnetometers can be distorted by the environment.

Sensor fusion is therefore common. Depending on the problem, state estimation may use complementary filters, Kalman-family filters, or other methods.

4.6. Cameras and Computer Vision

A camera provides a dense signal, but raw pixels are not semantic state. A simplified vision pipeline is:

camera -> frame capture -> preprocessing -> feature/object extraction -> tracking -> decision layer

Single-frame algorithmic accuracy is not sufficient for robot perception. A result can be numerically correct yet operationally unusable when latency, data age, or sensor/control timing no longer matches the robot's current physical state. Perception outputs should therefore carry timing, confidence, and sensor context together with the estimated value.

4.7. Audio Perception

Microphones can provide command input and environmental information. In a robot, an ASR result is more than text; it can be a control input capable of triggering physical action. Confidence, context, false acceptance, and cancellation behavior therefore matter.

4.8. The Coordinate Chain from Image Error to Actuator Command

Let the observed image center of a target be (u,v) and the desired center (u*,v*). The image error e = [u-u*, v-v*]ᵀ is not a motor torque command. Camera intrinsics and an approximate target depth first determine the local sensitivity of pixels to motion. The camera-to-robot transform and the robot Jacobian then connect a desired image correction to admissible joint movement. Identical two-pixel errors can correspond to different physical displacements when target depths differ.

Camera acquisition time belongs in the feedback state. At 30 frames/s, consecutive frames arrive about 33.3 ms apart, before accounting for transport and processing. A correction computed from an old image can overshoot while the robot continues moving. Controller gain, aggregate delay, physical velocity limits, and possible loss of tracked features must be evaluated together. If tracking fails, holding the last measurement indefinitely is unsafe; motion must follow a specified restricted behavior.

Unit 5: Coordinate Frames and Kinematics

5.1. Position and Orientation

Robotics requires both “where?” and “in which orientation?”. The state of an end effector is commonly represented by position and orientation together.

Position can be represented as:

p = [x y z]^T

Orientation may use rotation matrices, Euler angles, or quaternions.

5.2. Reference Frames

A robot can contain multiple coordinate frames:

• world,
• base,
• body,
• links/joints,
• end effector,
• camera,
• target object.

A target detected in camera coordinates must be transformed into the coordinate system required by the manipulator controller.

A homogeneous transformation combines rotation and translation:

T = [ R  p ]
    [ 0  1 ]

where R is a 3x3 rotation matrix and p is a 3x1 position vector.

5.3. Forward Kinematics

Forward kinematics computes end-effector pose from joint variables:

q -> x

For a serial manipulator, link transforms are chained:

T_0^n(q) = A_1(q_1) A_2(q_2) ... A_n(q_n)

This calculation underpins visualization, control, collision checking, and many other functions.

5.4. Inverse Kinematics

Inverse kinematics searches for joint variables that realize a desired end-effector pose:

x -> q

The problem can have:

• one solution,
• multiple solutions,
• infinitely many solutions,
• no solution.

Software should not stop at a mathematical solution. Joint limits, self-collision, cabling constraints, and distance from the current configuration can determine which solution is physically appropriate.

5.5. Workspace

The workspace is the set of poses that the robot can reach under its kinematic and mechanical constraints. A coordinate that is valid numerically is not necessarily reachable by the robot.

5.6. Separating Coordinate Frames, State Estimation, and Control

In robotics, the same physical position or orientation has different numerical coordinates in different reference frames. Coordinates relative to the robot base and coordinates relative to a camera frame may describe the same point while using different components. A transformation chain therefore has to preserve not only matrix multiplication but the explicit meaning of “from which frame to which frame.” Changing transformation order does not in general preserve the result.

Forward kinematics computes end-effector pose from joint variables. Inverse kinematics searches for joint values that produce a desired pose. The inverse problem need not have one solution: there may be multiple solutions, no reachable solution, or a singular configuration. “Invert the matrix to obtain inverse kinematics” is therefore not a general solution for robot mechanisms.

State estimation and control are also separate layers. Encoders, IMUs, cameras, and other sensors do not deliver the exact physical state without uncertainty. Position, velocity, or orientation is estimated from measurements; the controller then generates commands from the difference between estimated state and desired state. Poor estimation can undermine a sound control law, while actuator saturation can make a good estimate lead to poor physical behaviour.

The distinction also matters in autonomy:

perception → state estimation → planning → control → actuator → plant
                  ↑                                      ↓
                  └────────────── feedback ──────────────┘

Machine learning can improve one or more blocks in this chain, but it does not remove coordinate transformations, timing constraints, physical limits, or feedback requirements. Knowing which layer of the robot problem is being solved is more fundamental than the name of the algorithm used there.

From wheel speeds to the path of a differential-drive robot. Consider a no-slip robot with wheel radius r = 0.05 m and track width b = 0.30 m. Let right and left wheel angular velocities be 12 rad/s and 8 rad/s. Forward speed is v = r(ω_R + ω_L)/2 = 0.50 m/s, while yaw rate is ω = r(ω_R − ω_L)/b ≈ 0.667 rad/s. For nonzero yaw rate, the instantaneous turn radius is R = v/ω = 0.75 m. Equal wheel speeds give ω = 0, yielding straight-line motion in the ideal model; equal speeds of opposite sign give v = 0, producing rotation in place. This model assumes planar no-slip contact. Divergence between encoder odometry and an independent position measurement may indicate radius calibration error or wheel slip.

Unit 6: Jacobians, Singularities, and Robot Dynamics

6.1. Differential Kinematics

The Jacobian relates joint velocity to end-effector velocity:

x_dot = J(q) q_dot

Because J(q) depends on configuration, it changes as the robot moves. It is central to velocity control, force mapping, numerical inverse kinematics, and singularity analysis.

6.2. Singularities

At some configurations the Jacobian loses the rank needed for full local motion. The robot may lose mobility in a task-space direction, or require very large joint velocities to produce a small desired end-effector velocity.

A singularity is therefore not merely a mathematical corner case. It can cause:

• velocity commands to grow,
• torque demand to increase,
• numerical sensitivity,
• path-planning failure.

6.3. Force and Torque Mapping

In an idealized form, an end-effector force maps to joint torques through:

τ = J(q)^T F

This relationship helps connect task-space force requirements to actuator loading.

6.4. Dynamic Model

A common compact representation of serial-robot dynamics is:

M(q) q_ddot + C(q, q_dot) q_dot + g(q) + f(q_dot) = τ

where:

• M(q) is the mass/inertia matrix,
• C represents Coriolis and centrifugal effects,
• g(q) represents gravity,
• f represents friction,
• τ is the joint-torque vector.

Detailed dynamics may be unnecessary for slow, simple systems, but high speed, precision, and variable payloads make a purely kinematic treatment insufficient.

6.5. Damped Jacobian Inversion Near Singularities

The velocity relation \(\dot x=J(q)\dot q\) cannot be inverted directly when the Jacobian is singular or poorly conditioned. In a least-squares solution, small singular values can amplify small Cartesian-velocity changes into large joint velocities. Damped least squares limits this amplification:

\[ \dot q=J^T(JJ^T+\lambda^2I)^{-1}\dot x_d,\qquad \lambda>0. \]

Larger \(\lambda\) values constrain the magnitude of joint velocities but increase the mismatch with the requested end-effector velocity. Damping does not remove the physical singularity; it regularizes the numerical solution near it. Rotation and translation components require consistent scaling when they use different physical units.

The algebraic result alone is insufficient for control. Joint velocity, acceleration, position, and torque limits may require clamping \(\dot q\), after which the achievable Cartesian motion should be recomputed. Tracking the residual task error, smallest singular value, and applied saturation prevents the controller from silently reporting an unreachable command as successful.

Unit 7: Robot Control and Feedback

7.1. Open-loop and closed-loop Control

Open-loop systems issue commands without correcting from the measured output. Closed-loop systems compare measured state against a reference:

error = target - measured

This is the robotics manifestation of the principles developed in Automatic Control.

7.2. PID Control

One common controller is PID:

u(t) = Kp e(t) + Ki ∫e(t)dt + Kd de(t)/dt

• P responds to current error,
• I responds to accumulated error,
• D responds to the rate of error change.

PID is widely used in robot joints, but it is not a universal solution. Feedforward, gravity compensation, state feedback, torque control, and model-based approaches may be needed.

7.3. Cascaded Control

Servo systems can use nested loops:

position loop -> velocity loop -> current/torque loop -> motor

The inner loop usually runs faster. The software architecture must respect these different time scales.

7.4. Saturation and anti-windup

Real actuators have voltage, current, speed, and mechanical limits. A controller that ignores those bounds can accumulate integral action and recover poorly after saturation.

Physical constraints must therefore be explicit in robot software.

7.5. Safe States

Communication loss, invalid sensing, or controller faults require predefined behavior. Keeping the last command may be dangerous in one robot, while immediate stop can be more dangerous in another. Safety behavior has to be designed for the mission and physical mechanism.

Unit 8: Motion Planning, Task Execution, and Autonomy

8.1. Path versus Trajectory

A path describes where the robot should move geometrically. A trajectory adds timing.

A collision-free path may still be infeasible if acceleration, velocity, or torque limits cannot satisfy it.

8.2. Search and Planning

Grid or graph environments may use Dijkstra or A*, while continuous high-dimensional spaces may use sampling-based approaches. The objective is not always shortest distance. A cost function can include:

• distance,
• energy,
• collision risk,
• number of turns,
• time,
• visibility.

8.3. Behavior Layer

Higher-level robot behavior is not a single “AI function.” A behavior layer manages preconditions, modes, timeouts, events, failure states, and transitions among lower-level actions. It may be represented as a state machine, behavior tree, or task graph; the important properties are observable execution and explicit cancellation/failure semantics.

• start,
• decompose into steps,
• react to events,
• report failure,
• support interruption/recovery,
• coexist with competing priorities.

Reliable autonomy therefore often becomes an execution problem involving state machines, behavior trees, task graphs, event systems, and planners.

8.4. Tree- and circuit-based Visual Programming

In tree- or circuit-oriented visual programming, the hard problem is not placing blocks on a screen but executing a user-defined graph reliably at runtime. Node types, data types, cycles, timing, cancellation, error propagation, and persistent state require explicit contracts; otherwise visual convenience becomes runtime ambiguity.

A behavior editor needs to control:

• invalid connections,
• data types,
• parameter validation,
• cycles and dependencies,
• runtime error visibility,
• separation between UI and execution engine.

This reinforced a principle I later applied in many other software domains: giving users expressive power requires aggressively reducing invalid state space.

8.5. Autonomy under Uncertainty

Perception is not exact. Higher-level behavior may need to carry confidence, age, source, and validity instead of reducing everything to a hard “detected/not detected” state.

A false positive in a physical robot can become movement, not merely an incorrect label.

8.6. Evaluating a learning-based Route Policy

For a robot moving on a discrete grid, let state s denote a cell and action a an admissible move to a neighboring cell. Q-learning uses Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]. If Q(s,a)=0.5, α=0.2, r=-1, γ=0.9, and the greatest value at the successor state is 3, the target is 1.7 and the updated estimate is 0.74. This calculation estimates future reward; it does not establish that the resulting physical motion is safe.

A fair baseline is a deterministic search method tested on the same map with the same motion constraints and computational budget. Performance should include collisions, completion time, worst-case deviation, and sensitivity to changed conditions, not mean reward alone. Prohibited physical motions must be excluded through action constraints rather than handled only by negative rewards. Exploration can take place in simulation, while hardware action filters and emergency-stop behavior remain independent of the learned policy.

Unit 9: Embedded Systems and Robot Software Architecture

9.1. Layered Architecture

A useful abstraction for robot software is:

Hardware / drivers
        ↓
Device abstraction and communications
        ↓
Real-time control
        ↓
State and capability services
        ↓
Behavior / mission / autonomy
        ↓
Operator interface and external systems

Not every project needs these exact layers, but clear responsibility boundaries prevent hardware-specific changes from propagating through the entire software stack.

9.2. Selecting the Compute Platform: Capacity, I/O, and Margin

Choosing a robot computer is not equivalent to choosing the fastest processor. The platform must match the workload together with the physical and operational constraints of the robot.

The workload should first be decomposed:

real-time control
sensor acquisition and timing
state estimation
communications
telemetry / logging
human-robot interaction
vision or audio processing
high-level planning

Worst-case behavior matters as much as average computational cost. Vision may normally finish quickly but take longer on particular scenes; network traffic may arrive in bursts; logging may stall on storage. Time-critical control should not become unpredictable because of those variations.

Platform evaluation should therefore consider, together:

• deadlines of time-critical tasks,
• compute load and growth margin,
• memory capacity and bandwidth,
• peripherals such as ADC, PWM, and counters,
• I/O needs such as UART, SPI, I2C, CAN, and Ethernet,
• interrupt and DMA behavior,
• power consumption and thermal limits,
• physical size and environmental constraints,
• debugging and observability facilities,
• operating-system and driver support,
• development toolchain,
• component availability and maintenance lifetime.

A single catalog performance number may not represent the real workload. The strongest evidence comes from running representative algorithms, data sizes, and I/O patterns on candidate hardware. If control loops share a processor with intensive vision or audio processing, latency distribution should be measured rather than only its average.

Capacity margin is not automatically waste. Interrupt bursts, retransmission, fault logging, maintenance functions, and future software growth may be absent from nominal benchmarks. On a continuously saturated platform, small additional load can become queue growth and deadline misses.

Using several compute nodes is not an automatic cure either. Partitioning introduces new costs:

• synchronization,
• data movement,
• communication delay,
• clock and timestamp alignment,
• distributed fault handling,
• version and configuration consistency.

The centralized-versus-distributed decision should therefore be based not only on processing power but on time, data ownership, and fault boundaries.

9.3. State, Command, and Hardware Boundaries in Control Software

Robot control software must distinguish desired state, command execution state, and measured physical state. Giving a joint a 90° target, transmitting the command to a driver, and observing the encoder enter the target tolerance are different events.

In a robust architecture, high-level task logic does not depend directly on device-specific protocol details. Communication with motors, servos, I/O boards, and sensors is kept behind explicit adapter contracts. The adapter is responsible not only for data conversion but also for exposing timeout, disconnection, invalid-value, and device-fault semantics.

Each state field should have an authoritative source. If GUI, task executor, and device layer independently maintain their own “correct” joint position, races and stale-state errors become likely. Instead, source, timestamp, validity policy, and derived views should be defined explicitly.

A practical command lifecycle is:

1. create the command and validate preconditions,
2. transmit it with a unique identity,
3. receive acceptance or rejection,
4. observe physical feedback,
5. terminate as completed, timed out, cancelled, or failed,
6. record enough context for diagnosis.

This separation prevents the common assumption that physical motion occurred merely because a message was sent.

9.4. Microcontrollers and Peripherals

A robot's lower layers may interact with:

• GPIO,
• PWM,
• ADC,
• encoder counters,
• UART,
• SPI,
• I2C,
• CAN or comparable buses.

At the microcontroller layer, timing, interrupt behavior, buffer management, and failure modes have physical consequences. The peripheral and real-time execution principles detailed in Microprocessors, Microcontrollers, FPGA and PLC become concrete robotics concerns in sensor sampling, motor drive, encoder acquisition, and safety I/O.

9.5. Real-time Behavior

Real-time does not simply mean fast. Correct computation must complete within the relevant timing bound.

A 200 ms vision delay can be a minor UI issue in one context and unacceptable in a balance or joint-control loop. Latency therefore has to be interpreted by where it appears in the control hierarchy.

9.6. Concurrency

A robot may process simultaneously:

• sensor input,
• control loops,
• telemetry,
• vision,
• speech recognition,
• operator commands,
• logging.

More threads do not automatically improve the system. State ownership, lock duration, priority inversion, queue growth, and stale-data processing require explicit design.

9.7. Latency and Queue Buildup in Robotic Pipelines

Real-time engineering considers end-to-end latency budgets, worst-case or high-percentile delay, jitter, resource saturation, and recovery behavior together rather than optimizing an average in isolation. In a robotics loop, perception, localization, planning, control, and actuator communication form successive service and waiting times; low average latency at one node does not guarantee that the control period remains deterministic under queueing and synchronization pressure. Limits, overload behavior, and degradation modes need explicit definitions.

• Define backpressure behavior when camera/sensor production exceeds inference or control capacity.
• Identify cases where processing stale frames is worse than dropping frames.
• Measure p95/p99 end-to-end latency across perception, planning, and control.

Unit 10: Communications, Telemetry, and Distributed State

10.1. A Command Is Not State

Sending a command does not mean that it was received or executed. A robust protocol may distinguish states such as:

created -> sent -> received -> accepted -> executing -> completed / failed

The UI must define which state is sufficient before it presents success.

10.2. Telemetry

Telemetry is more than a debugging window. It is engineering evidence that helps explain what the physical system was doing at a given time.

Useful telemetry may include:

• timestamps,
• state identifiers,
• sensor values,
• commands,
• controller outputs,
• error codes,
• connectivity state,
• resource use.

Telemetry is a primary mechanism for evaluating consistency between the physical robot and the operator/software view. Carrying only instantaneous values is insufficient; source time, sequence information, validity interval, mode, and fault state are also relevant. Otherwise an interface can display a correct value at the wrong time and still mislead the operator.

10.3. Freshness

The “last received value” is not always the “currently valid value”. A packet may be delayed, queued, or retransmitted.

Data may therefore need metadata such as:

• measurement time,
• reception time,
• sequence number,
• validity horizon.

10.4. Link Failure

Loss of connectivity is more than a networking-layer exception. It changes robot behavior.

The system must decide whether the robot should:

• continue the current task,
• stop,
• return to a safe location,
• continue with local autonomy,
• expose a particular state to the operator.

This is a direct connection between computer networks and robot control.

10.5. Source Time, Event Ordering, and Link Loss in Telemetry

Telemetry reception time is distinct from the instant when a robot sensor performed its measurement. Delayed or retransmitted packets can make an old state appear new. A telemetry record is interpreted with source timestamp, sequence number, origin identity, and quality indicators; the latest received message is not necessarily the freshest physical observation.

Holding the last value after link loss can be unsafe depending on the control loop. Expiry of stale measurements, transition to fail-safe behavior, separation of commands from feedback, and state reconciliation after reconnection must be specified in advance. Validation injects packet loss, out-of-order delivery, duplication, and clock differences among sensors.

Unit 11: Human-robot Interaction, GUI, Speech, and Vision

11.1. Operator Interfaces and State Consistency

A robot-control interface is different from a conventional CRUD screen because displayed information must be consistent with the physical system in time as well as value. A position value can be numerically correct and still be operationally wrong if it is stale.

At minimum, the UI model should distinguish:

• desired value from measured value,
• latest received telemetry from its source timestamp,
• command transmission from device acceptance,
• warning from critical fault,
• automatic, semi-automatic, and manual modes.

When connectivity is lost, continuing to display the last value without an explicit stale indication is unsafe. When an operator issues a command, the UI should not optimistically change physical state; it should show command state first and measured outcome afterwards.

In human-robot interaction, good GUI design is primarily a problem of situational awareness, prevention of unintended commands, recoverability, and diagnosis rather than decoration.

11.2. Speech Recognition

HMM-based automatic speech recognition was a common architecture before modern transformer-based methods, with relatively explicit separation among acoustic features, state transitions, pronunciation lexicons, and language models. In robotics, the method-independent issue is that ASR output should be treated as an uncertain perception result rather than an unquestioned actuator command.

For a robot, word accuracy is not the only metric. Important dimensions include:

• false command acceptance,
• behavior under noise,
• endpointing,
• recognition latency,
• confidence thresholds,
• current behavioral context.

A transcription error in desktop dictation can be edited. A falsely accepted physical command has a different safety profile.

That early work became one of the foundations for my later work on speech feature extraction with MFCC, automatic speech recognition, and production-scale ASR systems.

11.3. Speech Synthesis

TTS is not only a way to make the robot “talk”; it is an output channel for system state. Spoken feedback needs to be correctly timed with movement, interruptible, and tied to the same state transition that the robot has actually reached.

11.4. Computer Vision

Robot vision cannot be separated from camera calibration, frame timing, processing latency, and the behavior layer that consumes the result.

High algorithmic accuracy is insufficient if information becomes stale before the physical action uses it. In such pipelines, throughput and age of information both matter.

Optical flow relies on assumptions that cannot always separate motion from illumination changes. Pixel values may change over a stationary surface when lighting varies, while saturated image regions lose usable gradient information. In local methods such as Lucas–Kanade, a singular or ill-conditioned image-gradient matrix should be treated as an uncertainty warning, not just as another motion estimate. A robot should downweight a low-confidence visual observation rather than promote it to an exact position measurement.

Four point correspondences on a planar surface can estimate a homography in general position, but near-collinear points make the problem ill-conditioned. A planar projective transformation cannot be applied unchanged to arbitrary three-dimensional objects at differing depths. Robust estimators such as RANSAC can reduce the influence of mismatched features, but a geometrically degenerate sample still cannot identify the correct model. Calibration uncertainty, reprojection residuals, and the planar-scene assumption must be checked separately.

Unit 12: Industrial, Mobile, and Humanoid Robots

12.1. Industrial Robots

Robots are particularly effective for tasks that are:

• repetitive,
• precise,
• heavy,
• hazardous,
• high-cycle-rate.

Welding, painting, loading/unloading, and assembly are classical industrial examples in which repeatability and process safety have to be evaluated together.

Robot selection should consider more than reach:

• payload,
• repeatability,
• cycle time,
• workspace,
• process force,
• safety envelope,
• integration cost,
• maintainability.

12.2. Mobile Robots

Mobile robots emphasize localization, mapping, and path planning because the base itself moves through the environment.

Wheel kinematics, slip, terrain, and sensor field of view influence localization quality. Outdoor GPS may help, while indoor systems may combine lidar, camera, odometry, and other sources.

12.3. Robots for Hazardous Environments

One major use of robots is work that would expose humans to unacceptable risk. Mines, biochemical sampling, radiation, and firefighting environments illustrate a different requirement: the robot must be engineered to survive the hazardous environment as well as perform the task.

The design problem is not simply replacing a person. The robot itself can fail because of:

• temperature,
• radiation,
• water,
• dust,
• explosive atmospheres,
• chemical contamination.

Environmental survivability therefore belongs inside mission design.

12.4. Humanoid Robots

Humanoid geometry is not valuable only because it looks human. It can support operation in environments designed around human proportions, tools, doors, stairs, and social interaction.

The cost is rapid growth in:

• degrees of freedom,
• kinematic complexity,
• balance control,
• energy use,
• real-time workload,
• wiring,
• mechanical constraints,
• software state space.

Humanoid robots combine many joints, high-dimensional state, contact dynamics, balance, perception, and human-robot interaction in one platform, making integration complexity particularly visible. A subsystem can be locally correct while timing or state-contract faults still break the overall behavior.

Unit 13: Simulation, Testing, and Debugging

13.1. Verification Boundaries of Simulation

Simulation allows conditions that are expensive, risky, or slow to reproduce on a physical robot to be tested under controlled conditions. Correct behavior in simulation, however, does not guarantee identical physical behavior. Motor saturation, backlash, friction, sensor noise, packet delay, CPU load, and mechanical tolerances may be missing or simplified.

Simulation tests can therefore be separated into three classes:

• algorithmic verification: checking planning, kinematics, and control logic against mathematical expectations,
• interface verification: testing message formats, state transitions, timeouts, and fault paths,
• reality-gap analysis: measuring systematic differences between the model and the physical platform.

Simulation-in-the-loop, software-in-the-loop, and hardware-in-the-loop are not synonyms. Each includes a different amount of real hardware in the test loop and exposes different defect classes.

13.2. SIL, HIL, and Physical Testing

Robot software can be validated at several layers:

• software/model simulation,
• algorithm tests,
• Software-in-the-Loop,
• Hardware-in-the-Loop,
• bench tests,
• physical robot tests.

Each level reveals different defects. A kinematic unit test can detect a sign error in a transform but cannot detect electrical noise from a motor drive.

13.3. Traceability from Requirements to Acceptance Testing

One of the most important questions in robot testing is: “When will we consider the system complete?” If that question is deferred until implementation is finished, the test process tends to invent criteria around whatever behavior the current system already exhibits. Acceptance criteria are more useful when they are defined with the requirements.

A simple traceability chain is:

requirement
    ↓
design allocation
    ↓
implementation unit
    ↓
verification method
    ↓
measurement / log / test evidence

For example, “the robot shall behave safely when communication is lost” is not yet a sufficiently precise test statement. Which link is lost? In which operating modes? Does safe behavior mean stopping, returning, or continuing under local autonomy? Within what time must actuators transition, and what must the operator interface display?

The answers influence architecture. Acceptance testing is therefore not only a final quality gate; it is an early test of whether the requirement itself is precise enough.

Robotics verification should include boundary and fault scenarios as well as nominal operation:

• invalid or stale sensor data,
• communication loss or delay,
• actuator saturation,
• low-energy conditions,
• restart behavior,
• partial subsystem loss,
• deadline misses in time-critical tasks,
• operator-command changes during task execution.

When risk is high, the most uncertain or destructive assumption should be tested early so that mechanics, electronics, and software are not all built around the same unverified premise.

Evidence should also be richer than a pass/fail flag when diagnosis matters. Timestamped telemetry, sensor and command history, software/configuration version, and physical test conditions make the result reproducible. This connects the reproducibility principles of Unit 13 directly to requirements engineering.

13.4. End-to-end Debugging

A complaint such as “the arm moves to the wrong position” can originate anywhere in:

target -> inverse kinematics -> trajectory -> controller -> drive -> motor -> mechanics -> sensor -> feedback

Good debugging follows data across those layers.

End-to-end debugging should trace software state, electronic interfaces, and physical signals as parts of the same event chain. If a motor does not move, the cause may lie in task logic, a driver command, communication, power electronics, or the mechanical load; stopping the investigation at the first software layer is insufficient.

13.5. Logging and Reproducibility

When a field failure cannot be reproduced immediately, the following become essential:

• event timeline,
• commands,
• sensor data,
• configuration,
• software version,
• fault codes,
• operating mode.

“It worked on my machine” is not sufficient verification for a physical system.

In reinforcement learning, state s, action a, reward r and next state s' must refer to the same physical decision cycle. Q-learning updates values by Q(s,a) ← Q(s,a) + α[r + γ max Q(s',·) − Q(s,a)]. With Q(s,a)=2, α=0.5, r=1, γ=0.9 and a maximum next-state value of 4, the target is 1+0.9×4=4.6 and the updated value is 2+0.5×(4.6−2)=3.3. A valid numerical update does not make exploratory robot actions safe. Collision boundaries, actuator speed limits and safety filtering must remain enforceable independently of the learned policy.

Unit 14: Reliability, Safety, and Maintainability

14.1. Enforcing Physical Bounds

Joint, velocity, temperature, and current limits should not exist only as GUI validation. Important constraints should be enforced as close as practical to the layer that can guarantee them.

One higher-level software defect should not be enough to push a mechanism beyond safe physical bounds.

14.2. Watchdogs and Liveness

A component that is technically running may still be failing to perform its time-critical function. A watchdog or heartbeat can represent different questions:

• Is the process alive?
• Is the control loop advancing?
• Is the data fresh?
• Is the deadline being met?

These meanings should not be conflated.

14.3. Fault Isolation

Should a camera failure stop all motion? The answer depends on the mission. Separating critical functions can prevent one subsystem failure from unnecessarily disabling independent capabilities.

14.4. Reliability, Availability, and Repair Time

A robot operating correctly once and a robot operating reliably over its intended life are different engineering problems. Reliability concerns performing the required function for the required duration under stated conditions. Availability concerns whether the system is ready for service when it is needed. In a repairable system, recovery time matters alongside failure frequency.

Metrics such as MTTF/MTBF and repair-oriented measures such as MTTR therefore answer different questions. A single impressive value does not prove system reliability; the operating profile, failure definition, population, and measurement conditions must be understood.

Practical robot failures are not limited to processors or algorithms. In a moving physical system, lifecycle constraints may be dominated by:

• connectors and contact surfaces,
• repeatedly flexed cables,
• batteries and power distribution,
• thermal loading of motors and drives,
• bearings, gear trains, and other moving parts,
• fans and cooling elements,
• mechanical joints exposed to vibration and shock.

A cable harness working on the first day says little about its state after repeated joint motion. Likewise, an actuator passing a short bench test does not establish its life under the mission's continuous and peak loads and thermal profile.

Life and cycle testing should therefore be a distinct verification layer. The goal is not simply to run every component to destruction, but to reproduce the usage profile in repeatable or accelerated conditions so that weak points become visible before deployment.

Maintainability is also an architectural property. If replacing one sensor requires disassembling the whole structure, repair time increases. If diagnostics expose only a generic device fault, fault localization becomes expensive. Replaceable modules, accessible connectors, explicit telemetry, diagnostic modes, and version information can reduce repair time and improve availability.

Redundancy can provide fault tolerance for some missions, but a second component is not sufficient by itself. Fault detection, switchover logic, state reconciliation, and common-cause failures must also be designed. Two controllers sharing the same vulnerable power path do not provide meaningful protection against a failure of that path.

Reliable robot design therefore combines three goals:

reduce failures
      +
detect failures early and correctly
      +
recover safely and quickly

14.5. Coding Standards

Coding standards and static analysis in physical-system software are not merely style concerns. In long-lived robot code, explicit ownership, boundary checks, deterministic error paths, controlled object lifetimes, and machine-checkable rules support maintainability and reliability together.

• isolated device dependencies,
• shared and explicit data models,
• deterministic state transitions,
• clear failure contracts,
• separation of responsibilities.

These reduce the cost of hardware revisions and limit fault propagation.

14.6. Safety

Robot safety is not only an emergency-stop button. Risk depends on speed, force, workspace, proximity to people, and control mode. Industrial systems combine mechanical guarding, safety circuits, software limits, and operating procedures.

Unit 15: System Integration, State Contracts, and Safe Execution

Robot subsystems can all be locally correct while the complete robot behaves incorrectly. The central integration problem is making state, timing, and failure contracts explicit across subsystem boundaries.

15.1. Authoritative State

When different layers maintain independent copies of the same physical quantity, it becomes unclear which value is current and trustworthy. Each state field should define its source, production time, validity interval, and transformation chain. GUI, task execution, and logging should use views derived from that authoritative state.

15.2. Separating a Command from Desired and Measured State

Sending a command does not prove that the physical effect occurred. Reliable execution uses an observable lifecycle such as created -> accepted -> executing -> completed/failed/cancelled. Higher-level behavior should advance from appropriate physical feedback rather than merely from an empty message queue.

15.3. Hardware Adapters and Interface Contracts

Device-specific details for motor drivers, servos, cameras, lidars, and I/O boards should not leak through the entire stack. An adapter normalizes not only data representation but also timeout, reconnection, sequence, device error, and invalid-measurement semantics.

This boundary reduces changes to task logic when hardware changes, but it does not mean forcing all devices into a lowest-common-denominator abstraction. Safety- or timing-relevant device properties must remain visible.

15.4. Calibration and Configuration Management

Encoder offsets, joint limits, camera intrinsics/extrinsics, controller gains, and kinematic parameters should be versioned configuration data rather than hidden constants. A test result is meaningful only in relation to both software revision and the configuration set used during that run.

15.5. Fault Propagation and Restart Boundaries

Failure of one sensor should not automatically place the whole robot into an undefined state. The design should specify which faults stop which subsystems, which functions can continue in degraded mode, and what state a restart actually resets. Restarting software does not automatically make unknown physical state safe.

15.6. End-to-end Observability

When a behavior fails, the event timeline should allow an engineer to trace from the high-level decision through sensor input, decision node, command transmission, driver response, and physical feedback. In robotics, such observability is part of verification infrastructure rather than merely operational monitoring.

Unit 16: Robotics System-Design Principles

The different subdisciplines of robotics converge on several common engineering principles.

16.1. Software State Is Not Physical Reality

Software operates on an internal model assembled from sensors, prior state, and communications. That model can be delayed, incomplete, or based on an incorrect measurement. Control and task logic should therefore consider data source and age as well as value.

16.2. Latency Is Not Merely a Performance Metric

Latency along the perception-to-actuation path can affect stability and safety. Low average latency is insufficient; jitter, queue buildup, and worst-case behavior also matter.

16.3. Interfaces Are Part of System Behavior

Serial links, CAN, Ethernet, shared memory, or inter-process messaging are not mere transport details. Framing, ordering, timeout, retry, idempotency, and fault reporting become behavior semantics at higher layers.

16.4. Simulation and Physical Validation Are Complementary

Simulation provides broad scenario coverage; physical testing exposes unmodeled dynamics and hardware defects. They are complementary verification layers aimed at different failure classes.

16.5. Safe Systems Manage Failure Explicitly

Robot software must design for sensor loss, driver rejection, communication interruption, energy constraints, and software restart in addition to nominal flow. Conditions for safe stop, degraded operation, and transfer to human control should be system requirements.

16.6. Mathematics and Systems Engineering Are Both Necessary

Kinematics, dynamics, estimation, and control provide the mathematical foundation of robot behavior. A real robot, however, also depends on timing, protocol semantics, calibration, power, software lifecycle, and human interaction. Academic robotics education should connect these levels within one system model.

Unit 17: Uncertainty, Learning, and Artificial Intelligence

17.1. State Estimation and Carrying Uncertainty into Decisions

Robot position, velocity, and obstacle range are estimates rather than exact values. Sensor noise, timing, and model error give every estimate uncertainty.

A Kalman filter is not merely a smoothing tool. Its covariance expresses estimated uncertainty, while innovation behavior can reveal model or sensor mismatch.

Planning can use this uncertainty directly: obstacle margins can grow as localization uncertainty grows, connecting perception quality to behavior safety.

17.2. Recording State and Uncertainty in Robotics Experiments

Robot performance should not be reduced to “task completed.” Initial pose, map, sensor conditions, battery state, surface, and software version all affect reproducibility.

Localization and perception are uncertain. Report distributions, outlier rates, tracking failures, and relocalization events rather than only average error. Control performance should not be interpreted independently of perception uncertainty.

The simulation-to-reality gap is often visible in friction, sensor noise, latency, and actuator dynamics. Simulation results should be revalidated on the physical platform, with discrepancies treated as model limitations.

17.3. The Interaction Between Robotics and Artificial Intelligence

The relationship between robotics and AI is bidirectional. Robotics contributes the physical world, sensors, latency, dynamics, and safety constraints; AI contributes learned methods for perception, representation, planning, and selected control problems. Robotics AI is therefore more than attaching a neural network to a robot.

17.3.1. Perception Is Not State

Object detection from a camera, speech recognition from a microphone, or classification from lidar are perception tasks. Decisions require these observations to be transformed into state with time, coordinate frames, and uncertainty.

sensor observation
↓
perception
↓
state estimation
↓
planning / decision
↓
control

Detecting an object in an image does not imply that its reliable 3-D robot-frame position is known.

17.3.2. Sensor Fusion and Learned Measurements

Kalman filtering and other Bayesian estimators combine models with measurements. A learned component may produce landmarks or pose estimates, but its error characteristics must be understood before the output is treated as a sensor measurement.

17.3.3. Planning and Learning

Classical motion planning searches for a valid path under obstacles and kinematic constraints. Learning can estimate costs, heuristics, or policies. Physical collision constraints, however, are not equivalent to learned scores and can remain deterministic.

Reinforcement learning may learn a policy:

u = π(s)

but unsafe exploration is expensive on real hardware. Simulation, offline data, constrained action spaces, and hybrid control are therefore important.

17.3.4. Control Boundary

Low-level joint or velocity loops are often naturally handled by deterministic control. Learned components may operate at a higher level, provide feed-forward compensation, estimate residual dynamics, or choose task policies.

17.3.5. Human-robot Interaction

Language and vision models can enrich the interface, but free-form natural language should not become direct motor command without translation into explicit task state and permitted actions.

17.3.6. Sim-to-real

Policies learned in simulation can fail because of friction, backlash, sensor noise, delay, or unmodeled contact. Domain randomization and fine-tuning can reduce but not eliminate this gap.

Robotics is a demanding AI domain because an error can become physical motion. The central discipline is to keep learned estimates distinct from physical truth, plans distinct from actuator commands, and simulation success distinct from field assurance.

Unit 18: Localization, SLAM, and Probabilistic State Estimation

For a mobile robot, the (x, y, θ) state used by a controller is not a directly measured exact truth. Encoders, IMUs, GNSS, cameras, and lidar carry noise, bias, latency, and environmental uncertainty. Localization estimates robot state probabilistically from these measurements.

18.1. Bayesian State Estimation

State estimation repeatedly combines two sources of information:

previous state
   + motion model
        ↓
     prediction
        +
  new measurement
        ↓
      update

Prediction follows the motion model; correction follows the sensor model. Sensor fusion is not simply averaging all sensors. Each measurement has its own uncertainty and observability.

Wheel-odometry error is not merely an independent position offset at each time step. A biased wheel-radius scale creates distance-dependent systematic error, while a small heading error accumulates as lateral displacement during a long straight drive. When planar position and heading are estimated jointly, covariance cross-terms capture this coupling. Reporting only independent x and y variances can conceal the true uncertainty ellipse. The gain assigned to an external position observation should reflect that joint uncertainty rather than treating all state components independently.

18.2. EKF and UKF

The Kalman filter provides closed-form mean and covariance updates for a linear Gaussian model.

The Extended Kalman Filter (EKF) linearizes nonlinear motion or measurement functions around the current estimate using Jacobians. Strong nonlinearity or a poor initial estimate can make this approximation inadequate.

The Unscented Kalman Filter (UKF) propagates selected sigma points through nonlinear functions instead of explicitly linearizing with Jacobians. There is no universal rule that UKF is always superior; state dimension, model cost, and the form of nonlinearity matter.

18.3. Particle Filters

When a state distribution is not represented well by one Gaussian, it can be approximated by weighted samples:

{x_i, w_i}

Particles are propagated by the motion model, reweighted by measurement likelihood, and resampled when necessary. The ability to represent multimodal uncertainty is valuable, while computational cost grows with particle count.

18.4. SLAM

Simultaneous Localization and Mapping estimates both robot pose and a map of an initially unknown environment. The two unknowns are coupled:

localization uses the map
map update uses localization

Odometric drift accumulates over time. Recognizing a previously visited place creates a loop closure constraint that can reduce accumulated error. A false loop closure can instead corrupt the global map, making data-association validation critical.

18.5. Front-end and back-end

A modern SLAM system can be separated conceptually as:

sensors
  ↓
features / scan matching / odometry   <- front-end
  ↓
constraints
  ↓
pose graph / optimization             <- back-end
  ↓
trajectory + map

The front-end derives candidate relationships from measurements; the back-end optimizes global consistency under those constraints. This keeps visual-processing algorithms distinct from state-estimation logic.

18.6. Where ROS 2 Fits

ROS 2 is not a localization or SLAM algorithm. It is a robotics middleware layer providing nodes, topic/service/action communication, time, QoS, and an ecosystem of tools. Algorithmic correctness and middleware transport behavior must be verified separately.

Reliable localization is therefore not produced by one sensor returning a "correct coordinate." It emerges from the combined design of motion models, measurement uncertainty, time alignment, and consistency checks.

18.7. Conclusion

Robotics engineering combines mechanical structure, actuation, sensing, estimation, kinematics, dynamics, control, planning, communication, and human-robot interaction in the same physical system. Each layer can be locally correct while the complete system fails because timing or state contracts are violated.

A sound robot design therefore does more than produce a working algorithm. It treats measurement validity, command lifecycle, physical feedback, latency, fault propagation, and verification evidence as one engineering problem. High-level methods such as SLAM, artificial intelligence, and advanced planning are reliable only when these underlying system contracts are reliable.

The central learning outcome is to analyze a robot not as disconnected mechanical and software components, but as an integrated real-time system that senses, estimates, decides, acts, and closes the feedback loop.

18.8. Preserving Hypotheses in Particle Filtering

When mobile-robot localization uncertainty cannot be described by one narrow Gaussian, several poses may explain the same observation. In a particle filter, each sample represents a candidate pose with an associated weight. The motion model propagates candidates, and the sensor model evaluates the likelihood of the measurement at each pose. In two visually similar corridors, maintaining two distinct groups of candidate poses can be more faithful than prematurely collapsing the distribution into one location.

Resampling addresses concentrated weights, but excessive resampling can eliminate low-weight hypotheses that later turn out to be correct. Effective particle count and resampling frequency should be monitored together, and recovery after localization failure should be tested explicitly. An inadequate sensor model or incorrect timestamp is not corrected merely by increasing particle count. With delayed sensor packets, the time state to which prediction and measurement updates apply must be well defined.

Particle-based SLAM can represent pose and map uncertainty together, but map size and particle count raise computation cost. Using a particle-based SLAM implementation does not imply that every SLAM method relies on particle filtering. Errors in front-end matching and data association may propagate into back-end optimization, so geometric consistency and timestamp alignment warrant independent tests.

Related Course Notes

• Mechatronic Systems
• Automatic Control
• Avionics Systems and Unmanned Aerial Vehicles

Incorporating Learned Observations into Probabilistic Localization

If a visual or audio network provides a measurement z_t to a robot observer, it must also have an uncertainty model R_t, a correct timestamp, and a defined coordinate frame. An underestimated R_t can make EKF or UKF updates dangerously overconfident. An outlier gate can use covariance-normalized innovation distance, but the gate itself requires calibration.

Low average error in static scenes does not capture jumps caused by temporal misalignment during motion. Lost, reordered, and duplicated observations and novel environments need separate tests. Learned sensing, state estimation, and control decisions are distinct validation layers.

18.9. Innovation-Based Validation of SLAM Observations

A visual correspondence can produce a large residual against the current robot pose and map. Mahalanobis distance and innovation covariance can test consistency with expected uncertainty. Incorrect data association, novel geometry and real rapid motion remain distinct causes. Rejection policies must account for track continuity, map integrity and re-localisation. The mathematical basis is in Anomaly Detection.

References

• Brendan Gregg. Systems Performance: Enterprise and the Cloud. Pearson Education, 2014.
• Bruno Siciliano, Lorenzo Sciavicco, Luigi Villani, Giuseppe Oriolo. Robotics: Modelling, Planning and Control. Springer.
• Charles M. Bergren. Anatomy of a Robot. McGraw-Hill, 2003. DOI: https://doi.org/10.1036/0071429301.
• Heather Adkins; Betsy Beyer; Paul Blankinship; Piotr Lewandowski; Ana Oprea; Adam Stubblefield. Building Secure & Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems. O’Reilly Media, 2020.
• John J. Craig. Introduction to Robotics: Mechanics and Control. Pearson.
• Mark W. Spong, Seth Hutchinson, M. Vidyasagar. Robot Modeling and Control. Wiley.
• Mor Harchol-Balter. Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press, 2013.
• Peter Corke. Robotics, Vision and Control: Fundamental Algorithms in MATLAB. Springer.
• Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018.
• Sebastian Thrun, Wolfram Burgard, Dieter Fox. Probabilistic Robotics. MIT Press, 2005.

• Walter W. Piegorsch, Richard A. Levine, Hao Helen Zhang, Thomas C. M. Lee (eds.). Computational Statistics in Data Science. Wiley, 2022. ISBN 9781119561071.
• Billingsley, J. Mechatronics Experiments with Machine Vision. Springer Nature Switzerland, 2026. DOI: https://doi.org/10.1007/978-3-032-11218-7.
• Naveenkumar, P.; Sharma, V.; Devarajan, G.; Abdullah, A.; Elngar, A. A. (eds.). Handbook of AI-Based Mechatronics Systems and Smart Solutions in Industrial Automation. CRC Press, 2026. DOI: https://doi.org/10.1201/9781032615790.
• Chunwei Zhang, Tianpeng Li, Ying Dai, Li Sun, and Ardashir Mohammadzadeh. Practical Neural Networks in Python and MATLAB. Springer Nature, 2026. DOI: https://doi.org/10.1007/978-3-032-14746-2.
• Simo Särkkä; Lennart Svensson. Bayesian Filtering and Smoothing, 2nd ed.. Cambridge University Press, 2023. https://doi.org/10.1017/9781108917407