Image Generation and Reconstruction

Image generation covers a broad family of methods ranging from geometric resampling to learned generative models. Their common property is that the output is derived from existing data or a learned distribution rather than directly measured by a sensor.

Resampling

Nearest-neighbour, bilinear and bicubic interpolation compute samples on a new grid. Scaling, rotation and geometric correction transform existing information but do not automatically recover physical detail that was never captured.

Generative models

Autoencoders map inputs to compact latent representations. Variational autoencoders introduce probabilistic structure into that space. A generative adversarial network trains a generator and discriminator with opposing objectives.

Diffusion models learn to reverse a gradual noising process and can produce highly realistic samples. Realism does not imply that the generated content was present in an original scene.

Synthetic data and identity transfer

Synthetic data can expand rare classes and reduce the cost of physical data collection. Synthetic artefacts may nevertheless become shortcuts learned by downstream models.

Face replacement, identity transfer and reenactment require explicit provenance management. Visual realism, geometric consistency and content provenance are different properties.

Evaluation should therefore include provenance, labelling, model version and the points at which synthetic and measured data interact, in addition to visual quality.

Parent topic: Image and Audio Processing

Quality and limits in geometric resampling

Nearest-neighbor, bilinear, and bicubic methods are not merely different "quality settings" for resizing or geometric transformation; they encode different local reconstruction assumptions. Without anti-alias filtering before reduction, high-frequency content can fold into false lower frequencies. Enlargement, meanwhile, cannot measure physical detail that was absent from the source samples.

Inverse mapping makes it natural to map every destination pixel back to a source coordinate and reduces holes. Repeated resampling, however, can accumulate blur or ringing. When possible, several geometric transforms should be composed and the original source sampled once.

Quality in reconstruction and generation pipelines is not captured by pixel similarity alone. Geometric fidelity, aliasing, edge structure, color consistency, and task-specific perceptual measures may all matter.

The boundary between reconstruction and generation

Reconstruction attempts to infer a missing or degraded quantity from observed data and an explicit or implicit acquisition model. Generation can create content that is not uniquely determined by the observation.

The same model family can be used for both tasks; the distinction lies in the epistemic claim, not the model name. A diffusion model can perform super-resolution or free-form generation. In the first case the output is conditioned on an observation, yet some high-frequency detail may still originate from the model prior.

A trustworthy computing pipeline should therefore make traceable which information was measured, which was deterministically transformed, and which was estimated or generated by a model.

References

• Rafael C. Gonzalez; Richard E. Woods. Digital Image Processing, Fourth Edition. Pearson, 2018.

• Richard Szeliski. Computer Vision: Algorithms and Applications, Second Edition. Springer, 2022.