Tensor Parallelism
A model-parallel strategy that splits individual tensor operations and model parameters across multiple devices that cooperate on each layer.
Related Concepts
- Data Parallelism
- Pipeline Parallelism
- Memory Bandwidth
- Inference Engine
A model-parallel strategy that splits individual tensor operations and model parameters across multiple devices that cooperate on each layer.