RFC: Distributed Computing Support for Large-Scale Tensor Network Contraction

[GiggleLiu]

Summary

This RFC proposes adding distributed computing support to omeinsum-rs for large-scale tensor network contraction across compute clusters. As tensor networks grow in size (e.g., quantum circuit simulation, PEPS/MERA for condensed matter), single-node computation becomes infeasible. Distributed execution is essential for:

• Quantum circuit simulation: 50+ qubit circuits require distributed memory
• Tensor network machine learning: Large-scale TNML models
• Condensed matter physics: PEPS/MERA ground state calculations
• Combinatorial optimization: Large QUBO problems via tropical tensor networks

Current State

What We Have

• ✅ CPU backend with BLAS acceleration (faer)
• ✅ CUDA backend with cuTENSOR integration
• ✅ Contraction order optimization (greedy, TreeSA via omeco)
• ✅ Multiple algebras (Standard, MaxPlus, MinPlus, MaxMul)
• ✅ Automatic differentiation for gradients
• ✅ Stride-based tensors with zero-copy views

What's Missing

• ❌ Multi-node execution
• ❌ Distributed tensor storage
• ❌ Inter-node communication
• ❌ Distributed contraction scheduling
• ❌ Fault tolerance

Technology Analysis

Option 1: MPI via mpi crate (rsmpi)

Aspect	Assessment
GitHub Stars	577
Downloads/month	4,585
Last Release	Dec 2025 (v0.8.1)
Reliability	⭐⭐⭐⭐ (4/5)

Pros:

• Industry standard for HPC
• Works on existing cluster infrastructure (SLURM, PBS)
• High-performance interconnects (InfiniBand, RDMA)
• Collective operations (allreduce, scatter, gather)
• Mature ecosystem

Cons:

• Requires MPI runtime installation
• Build complexity (needs libclang)
• Static process model (no dynamic scaling)
• Error handling is coarse-grained

Option 2: Actor Framework via ractor

Aspect	Assessment
GitHub Stars	1,900
Downloads/month	25,796
Last Release	Dec 2025 (v0.15.10)
Reliability	⭐⭐⭐⭐ (4/5)

Pros:

• Pure Rust, no external dependencies
• Dynamic scaling and fault tolerance
• Location-transparent messaging
• Supervision trees for reliability
• Cloud-native friendly

Cons:

• Different paradigm from traditional HPC
• Higher per-message overhead than MPI
• No native RDMA support
• Smaller HPC community

Option 3: gRPC via tonic

Aspect	Assessment
GitHub Stars	11,800
Downloads/month	9.3M
Last Release	Sep 2025 (v0.14.2)
Reliability	⭐⭐⭐⭐⭐ (5/5)

Pros:

• Extremely mature and widely used
• Excellent tooling (protobuf, streaming)
• Works across languages
• Good for heterogeneous clusters

Cons:

• Higher latency than MPI
• Not optimized for HPC workloads
• Requires service mesh for complex topologies

Option 4: Hybrid (MPI + Rayon)

Aspect	Assessment
Rayon Stars	12,600
Rayon Downloads/month	11.4M
Reliability	⭐⭐⭐⭐⭐ (5/5)

Pros:

• Best of both worlds
• MPI for inter-node, Rayon for intra-node
• Standard HPC pattern
• Optimal resource utilization

Cons:

• Two parallel models to reason about
• Thread affinity management complexity

---

Deep Dive: MPI for Tensor Network Contraction

MPI Communication Patterns for Tensor Networks

Tensor network contraction has specific communication patterns that map well (or poorly) to MPI primitives:

Point-to-Point vs Collective Operations

Operation	MPI Primitive	Tensor Network Use Case	Efficiency
Tensor slice transfer	MPI_Send/Recv	Moving tensor shards between nodes	⭐⭐⭐ Medium
Broadcast input tensor	MPI_Bcast	Replicate small tensors to all nodes	⭐⭐⭐⭐⭐ Excellent
Gather partial results	MPI_Gather	Collect output shards to root	⭐⭐⭐⭐ Good
Reduce contractions	MPI_Allreduce	Sum partial GEMM results	⭐⭐⭐⭐⭐ Excellent
Scatter input shards	MPI_Scatter	Distribute tensor along one dim	⭐⭐⭐⭐ Good
All-to-all redistribution	MPI_Alltoall	Change tensor distribution	⭐⭐ Poor (high cost)

MPI Topology Awareness

MPI can optimize communication based on physical network topology:

// Create Cartesian topology matching cluster layout
let dims = [num_nodes_x, num_nodes_y];
let cart_comm = comm.create_cart(&dims, &[false, false], true);

// Neighbor communication is optimized
cart_comm.shift(0, 1); // Get neighbors in x-direction

Benefits for Tensor Networks:

• PEPS contractions naturally map to 2D process grids
• Nearest-neighbor communication dominates
• Can exploit locality in contraction order

---

MPI-Specific Distribution Strategies

Strategy A: Replicated Inputs, Sharded Output

Node 0: A (full), B (full) → C[0:n/4, :]
Node 1: A (full), B (full) → C[n/4:n/2, :]
Node 2: A (full), B (full) → C[n/2:3n/4, :]
Node 3: A (full), B (full) → C[3n/4:n, :]

MPI Pattern:

// Broadcast inputs
comm.broadcast(0, &mut tensor_a);
comm.broadcast(0, &mut tensor_b);

// Local computation (embarrassingly parallel)
let local_c = einsum::<A>(&[&a, &b], ...);

// Gather results (optional, if root needs full output)
comm.gather(0, &local_c, &mut full_c);

Pros	Cons
Simple to implement	Memory inefficient (inputs replicated)
No communication during compute	Doesn't scale for large inputs
Perfect load balance	Broadcast cost O(n × log(p))

Best for: Small-to-medium inputs, large outputs

---

Strategy B: SUMMA (Scalable Universal Matrix Multiply Algorithm)

For large matrix contractions ij,jk→ik:

Process Grid (2×2 example):
┌─────────┬─────────┐
│ P(0,0)  │ P(0,1)  │
│ A[0,:], │ A[0,:], │
│ B[:,0]  │ B[:,1]  │
├─────────┼─────────┤
│ P(1,0)  │ P(1,1)  │
│ A[1,:], │ A[1,:], │
│ B[:,0]  │ B[:,1]  │
└─────────┴─────────┘

Each step:
1. Broadcast A block along row
2. Broadcast B block along column
3. Local GEMM
4. Accumulate to C

MPI Pattern:

for k in 0..num_blocks {
    // Broadcast A[i, k] along process row
    let a_owner = (my_row, k % grid_cols);
    row_comm.broadcast(a_owner.1, &mut a_block);
    
    // Broadcast B[k, j] along process column
    let b_owner = (k % grid_rows, my_col);
    col_comm.broadcast(b_owner.0, &mut b_block);
    
    // Local GEMM: C_local += A_block × B_block
    local_gemm(&a_block, &b_block, &mut c_local);
}

Pros	Cons
Memory efficient O(n²/p) per node	Complex implementation
Communication efficient O(n²/√p)	Requires square-ish process grid
Scales to large matrices	Block size tuning needed
Overlaps compute and communication	Not optimal for small tensors

Best for: Large matrix multiplications, dense tensor contractions

---

Strategy C: 1D Distribution with Allreduce

For contractions with reduction ij,ij→ (element-wise + sum):

Node 0: A[0:n/4], B[0:n/4] → partial_sum_0
Node 1: A[n/4:n/2], B[n/4:n/2] → partial_sum_1
...
Allreduce(SUM) → total_sum on all nodes

MPI Pattern:

// Each node has a shard
let local_result = local_einsum(&a_shard, &b_shard);

// Reduce across all nodes
let mut global_result = local_result.clone();
comm.allreduce(&local_result, &mut global_result, MPI_SUM);

Pros	Cons
Simple, efficient for reductions	Only works for full contractions
Allreduce is highly optimized	Result replicated (memory overhead)
O(log p) communication rounds

Best for: Trace operations, full tensor contractions, loss computation

---

Strategy D: Tree-Based Pairwise Contraction

For multi-tensor networks A,B,C,D,E → result:

Contraction Tree (from omeco optimizer):
        result
       /      \
     AB        CDE
    /  \      /   \
   A    B    CD    E
            /  \
           C    D

Distributed Assignment:
- Nodes 0-1: Compute A⊗B → AB
- Nodes 2-3: Compute C⊗D → CD  
- Node 2: Compute CD⊗E → CDE (after receiving CD)
- Node 0: Compute AB⊗CDE → result (after receiving both)

MPI Pattern:

match my_assignment {
    Task::Contract(a, b, dest) => {
        let result = local_einsum(&tensors[a], &tensors[b]);
        if dest != my_rank {
            comm.send(dest, &result);
        }
    }
    Task::Receive(src, then_contract) => {
        let received = comm.recv(src);
        // Continue with next contraction
    }
}

Pros	Cons
Exploits contraction order optimization	Load imbalance (tree structure)
Minimizes intermediate tensor size	Complex scheduling
Natural for tensor networks	Point-to-point latency sensitive

Best for: Complex tensor networks with many tensors

---

MPI Performance Considerations

1. Latency vs Bandwidth

Interconnect	Latency	Bandwidth	Best Strategy
InfiniBand EDR	~1 μs	100 Gb/s	Fine-grained, SUMMA
InfiniBand HDR	~0.6 μs	200 Gb/s	Fine-grained, SUMMA
100G Ethernet	~5 μs	100 Gb/s	Coarse-grained, fewer messages
10G Ethernet	~50 μs	10 Gb/s	Very coarse, batch transfers

Rule of thumb: Message size > bandwidth × latency for efficiency

• InfiniBand: Messages > 100 KB are efficient
• Ethernet: Messages > 500 KB preferred

2. MPI Implementation Differences

Implementation	Strengths	Considerations
OpenMPI	Flexible, many transports	Default for most clusters
MPICH	Reference impl, ABI standard	Good compatibility
Intel MPI	Optimized for Intel hardware	Best on Intel clusters
MVAPICH2	Best InfiniBand support	HPC-focused
Cray MPI	Cray system optimized	Supercomputers only

The mpi crate (rsmpi) works with all of these.

3. CUDA-Aware MPI

For GPU clusters, CUDA-aware MPI eliminates host staging:

// Without CUDA-aware MPI:
gpu_tensor.copy_to_host(&mut host_buffer);
comm.send(dest, &host_buffer);  // CPU memory
// ... on receiver ...
comm.recv(src, &mut host_buffer);
gpu_tensor.copy_from_host(&host_buffer);

// With CUDA-aware MPI:
comm.send(dest, gpu_tensor.as_ptr());  // Direct GPU memory!
comm.recv(src, gpu_tensor.as_mut_ptr());

Supported by: OpenMPI, MVAPICH2, Cray MPI
Speedup: 2-10x for GPU-heavy workloads

4. Non-Blocking Operations

Overlap communication with computation:

// Start async receive for next iteration
let recv_req = comm.irecv(src, &mut next_buffer);

// Compute with current data
let result = local_einsum(&current_a, &current_b);

// Wait for communication to complete
recv_req.wait();

---

MPI Challenges for Tensor Networks

Challenge 1: Dynamic Tensor Shapes

MPI typically requires knowing message sizes upfront:

// Problem: receiver doesn't know tensor shape
// Solution 1: Send shape first
comm.send(dest, &tensor.shape());  // Small metadata message
comm.send(dest, tensor.data());    // Large data message

// Solution 2: Use MPI_Probe
let status = comm.probe(src);
let count = status.count();  // Learn size before receiving

Challenge 2: Irregular Communication

Tensor networks often have irregular patterns unlike regular stencils:

// PEPS boundary contraction - neighbors only
for neighbor in tensor_network.neighbors(my_region) {
    comm.sendrecv(neighbor, &boundary_data, &mut incoming);
}

Mitigation: Use MPI_Neighbor_alltoall with graph topology

Challenge 3: Load Balancing

Contraction costs vary dramatically:

// Tensor dimensions affect cost cubically
let cost_a = a.shape().iter().product::<usize>();  // e.g., 1000
let cost_b = b.shape().iter().product::<usize>();  // e.g., 1000000

// Static assignment leads to imbalance
// Solution: Work stealing or dynamic scheduling

Challenge 4: Memory Constraints

Large intermediate tensors may exceed node memory:

// Check before allocation
let required = intermediate_size * size_of::<f64>();
if required > available_memory {
    // Use tiled/streaming algorithm
    // Or spill to disk/NVMe
}

---

Comparison: MPI vs Alternatives for Tensor Networks

Aspect	MPI	Actors (ractor)	gRPC (tonic)
Latency	~1 μs (IB)	~100 μs	~1 ms
Bandwidth	200 Gb/s	Network limited	Network limited
RDMA	✅ Native	❌	❌
GPU-Direct	✅ CUDA-aware	❌	❌
Fault Tolerance	❌ Process fails = job fails	✅ Supervision	✅ Retries
Dynamic Scaling	❌ Static	✅ Runtime spawn	✅ Service mesh
Debugging	⚠️ Complex	✅ Better tooling	✅ Standard tooling
Cloud Deploy	⚠️ Needs MPI runtime	✅ Native	✅ Native
HPC Cluster	✅ Standard	⚠️ Unusual	⚠️ Unusual

---

Recommended Approach: Hybrid MPI + Rayon

Based on analysis, we recommend a hybrid approach with abstraction layers:

┌─────────────────────────────────────────────────────────────┐
│                     User API Layer                          │
│  einsum_distributed<A, T>(&tensors, &indices, &output)     │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              Distributed Scheduler                          │
│  - Partition contraction tree across nodes                  │
│  - Schedule data movement                                   │
│  - Handle load balancing                                    │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   Node 0        │ │   Node 1        │ │   Node N        │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │Local Einsum │ │ │ │Local Einsum │ │ │ │Local Einsum │ │
│ │ (CPU/GPU)   │ │ │ │ (CPU/GPU)   │ │ │ │ (CPU/GPU)   │ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │   Rayon     │ │ │ │   Rayon     │ │ │ │   Rayon     │ │
│ │ (intra-node)│ │ │ │ (intra-node)│ │ │ │ (intra-node)│ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
         │                   │                   │
         └───────────────────┼───────────────────┘
                             │
                    ┌────────▼────────┐
                    │  Communication  │
                    │     Layer       │
                    │  (MPI / Actor)  │
                    └─────────────────┘

Tiered Backend Selection

pub enum CommunicationBackend {
    /// MPI for HPC clusters (best performance)
    Mpi(MpiCommunicator),
    
    /// Actors for cloud/fault-tolerant (good flexibility)
    Actor(RactorCommunicator),
    
    /// gRPC for heterogeneous systems (best compatibility)
    Grpc(TonicCommunicator),
    
    /// Single-node with Rayon (baseline)
    Local(RayonExecutor),
}

impl CommunicationBackend {
    /// Auto-detect best backend for environment
    pub fn detect() -> Self {
        if mpi_available() && on_hpc_cluster() {
            Self::Mpi(MpiCommunicator::new())
        } else if kubernetes_detected() {
            Self::Actor(RactorCommunicator::new())
        } else {
            Self::Local(RayonExecutor::new())
        }
    }
}

Priority Order for Implementation:

1. MPI - Primary target for HPC users (largest performance gains)
2. Rayon - Intra-node parallelism (complements MPI)
3. Actors - Cloud deployment and fault tolerance
4. gRPC - Heterogeneous and polyglot environments

---

Proposed Architecture

1. Communication Trait Abstraction

/// Abstract communication layer - can be MPI, actors, or gRPC
pub trait Communicator: Clone + Send + Sync {
    type Error: std::error::Error;
    
    fn rank(&self) -> usize;
    fn size(&self) -> usize;
    
    // Point-to-point
    async fn send<T: Scalar>(&self, dest: usize, data: &[T]) -> Result<(), Self::Error>;
    async fn recv<T: Scalar>(&self, src: usize, len: usize) -> Result<Vec<T>, Self::Error>;
    
    // Collective operations
    async fn broadcast<T: Scalar>(&self, root: usize, data: &mut [T]) -> Result<(), Self::Error>;
    async fn allreduce<T: Scalar, F: Fn(T, T) -> T>(&self, data: &mut [T], op: F) -> Result<(), Self::Error>;
    async fn scatter<T: Scalar>(&self, root: usize, send: &[T], recv: &mut [T]) -> Result<(), Self::Error>;
    async fn gather<T: Scalar>(&self, root: usize, send: &[T], recv: &mut [T]) -> Result<(), Self::Error>;
    
    // Synchronization
    async fn barrier(&self) -> Result<(), Self::Error>;
}

2. Distributed Tensor

/// Tensor distributed across multiple nodes
pub struct DistributedTensor<T: Scalar, B: Backend, C: Communicator> {
    /// Local shard of the tensor
    local: Tensor<T, B>,
    /// Distribution strategy
    distribution: Distribution,
    /// Global shape
    global_shape: Vec<usize>,
    /// Communicator handle
    comm: C,
}

pub enum Distribution {
    /// Tensor replicated on all nodes
    Replicated,
    /// Tensor sharded along specific dimension
    Sharded { dim: usize, chunks: Vec<(usize, usize)> },
    /// Block-cyclic distribution (ScaLAPACK style)
    BlockCyclic { block_size: Vec<usize> },
}

3. Distributed Contraction Strategies

pub enum ContractionStrategy {
    /// Each node computes part of output (embarrassingly parallel)
    OutputParallel {
        output_partition: Vec<Range<usize>>,
    },
    
    /// Tensor A sharded, B replicated, reduce partial results
    ReduceScatter {
        shard_dim: usize,
    },
    
    /// SUMMA-style 2D distribution for large matrix multiply
    Summa {
        proc_grid: (usize, usize),
        block_size: (usize, usize),
    },
    
    /// Tree-based reduction for multi-tensor contractions
    TreeReduction {
        tree: NestedEinsum<usize>,
        node_assignment: Vec<usize>,
    },
}

4. Distributed Einsum API

impl<L: Label> Einsum<L> {
    /// Optimize for distributed execution
    pub fn optimize_distributed(&mut self, comm: &impl Communicator) -> &mut Self {
        // Consider communication costs in optimization
        // Assign subtrees to nodes
        // Minimize data movement
    }
    
    /// Execute across cluster
    pub async fn execute_distributed<A, T, B, C>(
        &self,
        tensors: &[DistributedTensor<T, B, C>],
        comm: &C,
    ) -> Result<DistributedTensor<T, B, C>, DistributedError>
    where
        A: Algebra<Scalar = T>,
        T: Scalar,
        B: Backend,
        C: Communicator,
    {
        // 1. Analyze data locality
        // 2. Schedule contractions to minimize communication
        // 3. Execute local contractions in parallel
        // 4. Communicate intermediate results
        // 5. Combine final result
    }
}

---

Implementation Phases

Phase 1: Foundation (Core Infrastructure)

• Define Communicator trait
• Implement MPI backend (MpiCommunicator)
• Implement mock communicator for testing
• Add DistributedTensor type
• Basic point-to-point tensor transfer
• Serialization for tensor data

Phase 2: Basic Distribution

• Implement Distribution strategies
• Tensor sharding and gathering
• Replicated tensor broadcast
• Simple output-parallel contraction
• Integration tests on multi-process

Phase 3: Optimized Contractions

• Communication-aware contraction optimizer
• SUMMA-style distributed GEMM
• Tree-based reduction for multi-tensor
• Overlap computation and communication
• Load balancing heuristics

Phase 4: Advanced Features

• Rayon integration for intra-node parallelism
• CUDA-aware MPI for GPU clusters
• Checkpointing for long-running contractions
• Fault tolerance (actor-based fallback)
• Dynamic load balancing

Phase 5: Ecosystem Integration

• Actor-based communicator (RactorCommunicator)
• Cloud deployment support (Kubernetes)
• Benchmark suite for distributed performance
• Documentation and examples

---

Challenges and Considerations

1. Communication Cost Modeling

The contraction optimizer (omeco) currently optimizes for FLOPs. For distributed execution, we need:

/// Cost model for distributed contraction
pub struct DistributedCost {
    /// Compute cost (FLOPs)
    compute: f64,
    /// Communication cost (bytes transferred)
    communication: f64,
    /// Memory high-water mark per node
    memory_per_node: f64,
    /// Load imbalance factor
    imbalance: f64,
}

2. Tensor Redistribution

When contraction order requires different distributions:

// A is sharded on dim 0, B is sharded on dim 1
// For A @ B, we need to redistribute
async fn redistribute<T, B, C>(
    tensor: &DistributedTensor<T, B, C>,
    new_dist: Distribution,
) -> DistributedTensor<T, B, C>

3. Memory Management

Large tensors may not fit in single-node memory:

• Streaming/tiled execution
• Out-of-core algorithms
• Memory-aware scheduling

4. Heterogeneous Clusters

Mixed CPU/GPU nodes:

pub struct NodeCapabilities {
    cpus: usize,
    gpus: Vec<GpuInfo>,
    memory: usize,
    interconnect: Interconnect,
}

5. Debugging and Profiling

Distributed debugging is hard:

• Per-node logging with timestamps
• Communication tracing
• Deadlock detection
• Performance profiling (communication vs compute)

---

References

1. Cyclops Tensor Framework - Distributed tensor contractions in C++
2. ITensor - Tensor network library with MPI support
3. ExaTENSOR - Exascale tensor algebra
4. cuQuantum - NVIDIA's distributed quantum simulation
5. opt_einsum - Python einsum optimizer (for algorithm reference)

---

Questions for Discussion

1. MPI vs Actors: Should we prioritize MPI (HPC clusters) or actors (cloud-native) first?
2. API Design: Should distributed execution be opt-in or should the API automatically detect cluster availability?
3. Scope: Should we support arbitrary tensor distributions or focus on specific patterns (sharded, replicated)?
4. Integration: How should this interact with the existing Backend trait?

---

/cc @maintainers

🤖 Generated with Claude Code

[GiggleLiu]

Addendum: Parallel Computing Strategy Analysis (MPI Focus)

MPI Communication Patterns for Tensor Networks

Tensor network contraction has specific communication patterns that map well (or poorly) to MPI primitives:

1. Point-to-Point vs Collective Operations

Operation	MPI Primitive	Tensor Network Use Case	Efficiency
Tensor slice transfer	MPI_Send/Recv	Moving tensor shards between nodes	⭐⭐⭐ Medium
Broadcast input tensor	MPI_Bcast	Replicate small tensors to all nodes	⭐⭐⭐⭐⭐ Excellent
Gather partial results	MPI_Gather	Collect output shards to root	⭐⭐⭐⭐ Good
Reduce contractions	MPI_Allreduce	Sum partial GEMM results	⭐⭐⭐⭐⭐ Excellent
Scatter input shards	MPI_Scatter	Distribute tensor along one dim	⭐⭐⭐⭐ Good
All-to-all redistribution	MPI_Alltoall	Change tensor distribution	⭐⭐ Poor (high cost)

2. MPI Topology Awareness

MPI can optimize communication based on physical network topology:

// Create Cartesian topology matching cluster layout
let dims = [num_nodes_x, num_nodes_y];
let cart_comm = comm.create_cart(&dims, &[false, false], true);

// Neighbor communication is optimized
cart_comm.shift(0, 1); // Get neighbors in x-direction

Benefits for Tensor Networks:

• PEPS contractions naturally map to 2D process grids
• Nearest-neighbor communication dominates
• Can exploit locality in contraction order

---

MPI-Specific Strategies for Distributed Einsum

Strategy A: Replicated Inputs, Sharded Output

Node 0: A (full), B (full) → C[0:n/4, :]
Node 1: A (full), B (full) → C[n/4:n/2, :]
Node 2: A (full), B (full) → C[n/2:3n/4, :]
Node 3: A (full), B (full) → C[3n/4:n, :]

MPI Pattern:

// Broadcast inputs
comm.broadcast(0, &mut tensor_a);
comm.broadcast(0, &mut tensor_b);

// Local computation (embarrassingly parallel)
let local_c = einsum::<A>(&[&a, &b], ...);

// Gather results (optional, if root needs full output)
comm.gather(0, &local_c, &mut full_c);

Pros	Cons
Simple to implement	Memory inefficient (inputs replicated)
No communication during compute	Doesn't scale for large inputs
Perfect load balance	Broadcast cost O(n × log(p))

Best for: Small-to-medium inputs, large outputs

---

Strategy B: SUMMA (Scalable Universal Matrix Multiply Algorithm)

For large matrix contractions ij,jk→ik:

Process Grid (2×2 example):
┌─────────┬─────────┐
│ P(0,0)  │ P(0,1)  │
│ A[0,:], │ A[0,:], │
│ B[:,0]  │ B[:,1]  │
├─────────┼─────────┤
│ P(1,0)  │ P(1,1)  │
│ A[1,:], │ A[1,:], │
│ B[:,0]  │ B[:,1]  │
└─────────┴─────────┘

Each step:
1. Broadcast A block along row
2. Broadcast B block along column
3. Local GEMM
4. Accumulate to C

MPI Pattern:

for k in 0..num_blocks {
    // Broadcast A[i, k] along process row
    let a_owner = (my_row, k % grid_cols);
    row_comm.broadcast(a_owner.1, &mut a_block);
    
    // Broadcast B[k, j] along process column
    let b_owner = (k % grid_rows, my_col);
    col_comm.broadcast(b_owner.0, &mut b_block);
    
    // Local GEMM: C_local += A_block × B_block
    local_gemm(&a_block, &b_block, &mut c_local);
}

Pros	Cons
Memory efficient O(n²/p) per node	Complex implementation
Communication efficient O(n²/√p)	Requires square-ish process grid
Scales to large matrices	Block size tuning needed
Overlaps compute and communication	Not optimal for small tensors

Best for: Large matrix multiplications, dense tensor contractions

---

Strategy C: 1D Distribution with Allreduce

For contractions with reduction ij,ij→ (element-wise + sum):

Node 0: A[0:n/4], B[0:n/4] → partial_sum_0
Node 1: A[n/4:n/2], B[n/4:n/2] → partial_sum_1
...
Allreduce(SUM) → total_sum on all nodes

MPI Pattern:

// Each node has a shard
let local_result = local_einsum(&a_shard, &b_shard);

// Reduce across all nodes
let mut global_result = local_result.clone();
comm.allreduce(&local_result, &mut global_result, MPI_SUM);

Pros	Cons
Simple, efficient for reductions	Only works for full contractions
Allreduce is highly optimized	Result replicated (memory overhead)
O(log p) communication rounds

Best for: Trace operations, full tensor contractions, loss computation

---

Strategy D: Tree-Based Pairwise Contraction

For multi-tensor networks A,B,C,D,E → result:

Contraction Tree (from omeco optimizer):
        result
       /      \
     AB        CDE
    /  \      /   \
   A    B    CD    E
            /  \
           C    D

Distributed Assignment:
- Nodes 0-1: Compute A⊗B → AB
- Nodes 2-3: Compute C⊗D → CD  
- Node 2: Compute CD⊗E → CDE (after receiving CD)
- Node 0: Compute AB⊗CDE → result (after receiving both)

MPI Pattern:

match my_assignment {
    Task::Contract(a, b, dest) => {
        let result = local_einsum(&tensors[a], &tensors[b]);
        if dest != my_rank {
            comm.send(dest, &result);
        }
    }
    Task::Receive(src, then_contract) => {
        let received = comm.recv(src);
        // Continue with next contraction
    }
}

Pros	Cons
Exploits contraction order optimization	Load imbalance (tree structure)
Minimizes intermediate tensor size	Complex scheduling
Natural for tensor networks	Point-to-point latency sensitive

Best for: Complex tensor networks with many tensors

---

MPI Performance Considerations

1. Latency vs Bandwidth

Interconnect	Latency	Bandwidth	Best Strategy
InfiniBand EDR	~1 μs	100 Gb/s	Fine-grained, SUMMA
InfiniBand HDR	~0.6 μs	200 Gb/s	Fine-grained, SUMMA
100G Ethernet	~5 μs	100 Gb/s	Coarse-grained, fewer messages
10G Ethernet	~50 μs	10 Gb/s	Very coarse, batch transfers

Rule of thumb: Message size > bandwidth × latency for efficiency

• InfiniBand: Messages > 100 KB are efficient
• Ethernet: Messages > 500 KB preferred

2. MPI Implementation Differences

Implementation	Strengths	Considerations
OpenMPI	Flexible, many transports	Default for most clusters
MPICH	Reference impl, ABI standard	Good compatibility
Intel MPI	Optimized for Intel hardware	Best on Intel clusters
MVAPICH2	Best InfiniBand support	HPC-focused
Cray MPI	Cray system optimized	Supercomputers only

The mpi crate (rsmpi) works with all of these.

3. CUDA-Aware MPI

For GPU clusters, CUDA-aware MPI eliminates host staging:

// Without CUDA-aware MPI:
gpu_tensor.copy_to_host(&mut host_buffer);
comm.send(dest, &host_buffer);  // CPU memory
// ... on receiver ...
comm.recv(src, &mut host_buffer);
gpu_tensor.copy_from_host(&host_buffer);

// With CUDA-aware MPI:
comm.send(dest, gpu_tensor.as_ptr());  // Direct GPU memory!
comm.recv(src, gpu_tensor.as_mut_ptr());

Supported by: OpenMPI, MVAPICH2, Cray MPI
Speedup: 2-10x for GPU-heavy workloads

4. Non-Blocking Operations

Overlap communication with computation:

// Start async receive for next iteration
let recv_req = comm.irecv(src, &mut next_buffer);

// Compute with current data
let result = local_einsum(&current_a, &current_b);

// Wait for communication to complete
recv_req.wait();

---

MPI Challenges for Tensor Networks

Challenge 1: Dynamic Tensor Shapes

MPI typically requires knowing message sizes upfront:

// Problem: receiver doesn't know tensor shape
// Solution 1: Send shape first
comm.send(dest, &tensor.shape());  // Small metadata message
comm.send(dest, tensor.data());    // Large data message

// Solution 2: Use MPI_Probe
let status = comm.probe(src);
let count = status.count();  // Learn size before receiving

Challenge 2: Irregular Communication

Tensor networks often have irregular patterns unlike regular stencils:

// PEPS boundary contraction - neighbors only
for neighbor in tensor_network.neighbors(my_region) {
    comm.sendrecv(neighbor, &boundary_data, &mut incoming);
}

Mitigation: Use MPI_Neighbor_alltoall with graph topology

Challenge 3: Load Balancing

Contraction costs vary dramatically:

// Tensor dimensions affect cost cubically
let cost_a = a.shape().iter().product::<usize>();  // e.g., 1000
let cost_b = b.shape().iter().product::<usize>();  // e.g., 1000000

// Static assignment leads to imbalance
// Solution: Work stealing or dynamic scheduling

Challenge 4: Memory Constraints

Large intermediate tensors may exceed node memory:

// Check before allocation
let required = intermediate_size * size_of::<f64>();
if required > available_memory {
    // Use tiled/streaming algorithm
    // Or spill to disk/NVMe
}

---

Comparison: MPI vs Alternatives for Tensor Networks

Aspect	MPI	Actors (ractor)	gRPC (tonic)
Latency	~1 μs (IB)	~100 μs	~1 ms
Bandwidth	200 Gb/s	Network limited	Network limited
RDMA	✅ Native	❌	❌
GPU-Direct	✅ CUDA-aware	❌	❌
Fault Tolerance	❌ Process fails = job fails	✅ Supervision	✅ Retries
Dynamic Scaling	❌ Static	✅ Runtime spawn	✅ Service mesh
Debugging	⚠️ Complex	✅ Better tooling	✅ Standard tooling
Cloud Deploy	⚠️ Needs MPI runtime	✅ Native	✅ Native
HPC Cluster	✅ Standard	⚠️ Unusual	⚠️ Unusual

---

Recommendation: Tiered Approach

pub enum CommunicationBackend {
    /// MPI for HPC clusters (best performance)
    Mpi(MpiCommunicator),
    
    /// Actors for cloud/fault-tolerant (good flexibility)
    Actor(RactorCommunicator),
    
    /// gRPC for heterogeneous systems (best compatibility)
    Grpc(TonicCommunicator),
    
    /// Single-node with Rayon (baseline)
    Local(RayonExecutor),
}

impl Backend {
    /// Auto-detect best backend for environment
    pub fn detect() -> Self {
        if mpi_available() && on_hpc_cluster() {
            Backend::Mpi(MpiCommunicator::new())
        } else if kubernetes_detected() {
            Backend::Actor(RactorCommunicator::new())
        } else {
            Backend::Local(RayonExecutor::new())
        }
    }
}

Priority Order for Implementation:

1. MPI - Primary target for HPC users (largest performance gains)
2. Rayon - Intra-node parallelism (complements MPI)
3. Actors - Cloud deployment and fault tolerance
4. gRPC - Heterogeneous and polyglot environments

---

🤖 Generated with Claude Code

[GiggleLiu]

Rust MPI Ecosystem Analysis: Comparison with Other Languages

Ecosystem Maturity Comparison

Language	Primary Library	Maturity	Community	HPC Adoption
C	Native MPI	⭐⭐⭐⭐⭐ (30+ years)	Massive	Standard
C++	Native MPI + Boost.MPI	⭐⭐⭐⭐⭐	Large	Standard
Fortran	Native MPI	⭐⭐⭐⭐⭐	Large (legacy)	Standard
Python	mpi4py	⭐⭐⭐⭐ (15+ years)	Large	Growing
Julia	MPI.jl	⭐⭐⭐⭐ (8+ years)	Medium	Growing
Rust	rsmpi	⭐⭐⭐ (8 years)	Small	Niche

---

Rust MPI (mpi crate) - Detailed Analysis

Strengths Over Other Ecosystems

Advantage	Explanation
Memory Safety	No buffer overflows, dangling pointers, or data races at compile time
Type Safety	MPI datatypes checked at compile time via traits
Zero-Cost Abstractions	Rust iterators and generics compile to optimal code
Modern Tooling	Cargo, rustfmt, clippy vs. Makefiles
Fearless Concurrency	Safe MPI + threading (Rayon) without data races
No GC Pauses	Predictable latency like C/Fortran

Example - Type-safe communication:

// Rust: Compile-time type checking
let data: Vec<f64> = vec![1.0, 2.0, 3.0];
world.process_at_rank(1).send(&data[..]); // Type inferred

// C: Runtime errors possible
double data[3] = {1.0, 2.0, 3.0};
MPI_Send(data, 3, MPI_INT, 1, 0, MPI_COMM_WORLD); // WRONG TYPE - silent corruption!

Weaknesses Compared to Other Ecosystems

Limitation	Impact	Workaround
Smaller Community	Fewer examples, Stack Overflow answers	Use C/Fortran docs, translate patterns
Less Battle-Tested	Edge cases may have bugs	Contribute fixes upstream
Build Complexity	Requires libclang, MPI dev headers	Docker/Nix for reproducibility
Pre-1.0 API	Breaking changes possible	Pin versions carefully
Missing Advanced Features	One-sided (RMA), MPI-IO incomplete	Use mpi-sys for raw access
Interop Friction	Calling Fortran libraries awkward	C wrappers as intermediate

---

Feature Comparison: Rust vs C vs Python

Feature	C (Native)	Python (mpi4py)	Rust (rsmpi)
Point-to-point	✅ Full	✅ Full	✅ Full
Collectives	✅ Full	✅ Full	✅ Full
Derived Datatypes	✅ Full	✅ Full	⚠️ Partial (derive macro)
One-sided (RMA)	✅ Full	✅ Full	⚠️ Partial
MPI-IO	✅ Full	✅ Full	❌ Missing
Process Topologies	✅ Full	✅ Full	✅ Full
Dynamic Processes	✅ Full	✅ Full	⚠️ Partial
Error Handling	⚠️ Manual	✅ Exceptions	✅ Result types
Thread Safety	⚠️ MPI_THREAD_*	⚠️ GIL issues	✅ Compile-time
Non-blocking	✅ Full	✅ Full	✅ Full
Custom Operators	✅ Full	✅ Full	✅ Full (with feature)

---

Specific Limitations of Rust MPI

1. MPI-IO Not Supported

// NOT AVAILABLE in rsmpi:
// MPI_File_open, MPI_File_read, MPI_File_write

// Workaround: Use mpi-sys for raw bindings
unsafe {
    let mut fh: mpi_sys::MPI_File = std::ptr::null_mut();
    mpi_sys::MPI_File_open(
        comm.as_raw(),
        filename.as_ptr(),
        mpi_sys::MPI_MODE_RDONLY,
        mpi_sys::MPI_INFO_NULL,
        &mut fh
    );
}

Impact: Cannot use parallel I/O for checkpointing large tensors
Severity: High for large-scale tensor networks

2. One-Sided Communication (RMA) Incomplete

// Limited support for:
// MPI_Win_create, MPI_Put, MPI_Get, MPI_Accumulate

// These are critical for:
// - Asynchronous data movement
// - Global Arrays-style programming
// - Reducing synchronization overhead

Impact: Cannot implement efficient PGAS-style tensor distribution
Severity: Medium - can use two-sided as fallback

3. Derived Datatypes Require Unsafe or Derive Macro

// With derive macro (requires feature):
#[derive(Equivalence)]
struct Particle {
    x: f64,
    y: f64,
    mass: f64,
}

// Without: Must use unsafe mpi-sys
// Or: Serialize to bytes (overhead)

Impact: Custom struct communication less ergonomic than C
Severity: Low - derive macro works for most cases

4. No GPU-Aware MPI in Safe Rust

// Cannot do this safely:
let gpu_ptr: *mut f64 = cuda_tensor.as_mut_ptr();
comm.send(gpu_ptr, count); // Would need unsafe + raw pointers

// Must stage through CPU:
let host_data = cuda_tensor.to_vec();
comm.send(&host_data[..]);

Impact: 2-10x slowdown for GPU clusters
Severity: High for GPU-heavy workloads
Workaround: Unsafe code with careful verification

5. Error Messages Can Be Cryptic

// Rust MPI error:
// "MPI_ERR_TRUNCATE: message truncated"

// Doesn't tell you:
// - Which rank sent/received
// - Expected vs actual size
// - Which line in your code

// C at least gives you the status object

Impact: Debugging is harder
Severity: Medium

6. Async/Await Not Native

// MPI is fundamentally synchronous (blocking or polling)
// Rust's async ecosystem doesn't integrate well

// Cannot do:
async fn distributed_contract() {
    comm.send(...).await; // NOT SUPPORTED
}

// Must use:
let req = comm.immediate_send(...);
// ... do other work ...
req.wait(); // Blocking

Impact: Harder to overlap with async Rust code
Severity: Medium - non-blocking MPI still works

---

Comparison: Library Ecosystem for HPC

Library Type	C/C++/Fortran	Python	Rust
BLAS/LAPACK	Intel MKL, OpenBLAS	NumPy/SciPy	faer, nalgebra, ndarray
FFT	FFTW	scipy.fft	rustfft
Sparse Solvers	PETSc, Trilinos, MUMPS	petsc4py	❌ Limited
Tensor Contraction	CTF, TBLIS, ExaTENSOR	opt_einsum, tensorly	omeinsum-rs (us!)
Distributed Arrays	Global Arrays, ScaLAPACK	Dask, mpi4py	❌ None mature
GPU	CUDA, HIP (native)	CuPy, PyTorch	cudarc, rust-cuda (experimental)

Key Gap: Rust lacks mature distributed array libraries like ScaLAPACK or Dask.

---

When to Use Rust MPI vs Alternatives

Use Case	Recommendation	Reason
New HPC project, safety critical	✅ Rust MPI	Memory safety prevents silent corruption
Existing C/Fortran codebase	❌ Stay with C/Fortran	Rewrite cost too high
Rapid prototyping	❌ Python mpi4py	Faster iteration
Production tensor networks	⚠️ Rust MPI + FFI	Use Rust for control, C libs for numerics
GPU clusters	⚠️ Caution	GPU-aware MPI support is weak
Need MPI-IO	❌ Not yet	Feature missing
Cloud deployment	⚠️ Consider alternatives	Actors/gRPC may be easier

---

Roadmap Implications for omeinsum-rs

Given these limitations, our distributed computing implementation should:

1. Start with well-supported features:

   • Point-to-point (Send/Recv)
   • Collectives (Bcast, Reduce, Allreduce, Gather, Scatter)
   • Cartesian topologies

2. Avoid or defer:

   • MPI-IO (use standard file I/O with rank-based filenames)
   • One-sided communication (use two-sided patterns)
   • GPU-aware MPI (stage through CPU initially)

3. Plan for workarounds:

   • Checkpointing: Each rank writes own file, coordinate via barrier
   • GPU clusters: Accept CPU staging overhead or provide unsafe opt-in
   • Custom datatypes: Use #[derive(Equivalence)] or byte serialization

4. Monitor ecosystem:

   • rsmpi is actively maintained
   • Features may be added (watch GitHub issues)
   • Contribute patches if we need missing features

---

Summary

Aspect	Rust MPI Score	Notes
Core Functionality	8/10	Point-to-point, collectives work well
Advanced Features	5/10	RMA, MPI-IO gaps
Ecosystem Integration	6/10	Works but smaller community
Safety Guarantees	10/10	Best-in-class
Performance	9/10	Matches C when used correctly
Developer Experience	7/10	Better than C, worse than Python
Production Readiness	7/10	Usable with caveats

Bottom Line: Rust MPI is production-viable for tensor network contractions if we design around its limitations. The safety benefits are significant for long-running HPC jobs where silent memory corruption would be catastrophic.

---

🤖 Generated with Claude Code

[GiggleLiu]

Deep Dive: GPU Performance Bottlenecks in Rust HPC

Is "No GPU-aware MPI" the Most Serious Issue?

Short answer: It depends on the computation-to-communication ratio.

For communication-bound workloads → Yes, it's critical
For compute-bound workloads → No, other issues dominate

---

Bottleneck Analysis for GPU Tensor Networks

Bottleneck	Severity	When It Matters
1. CUDA ecosystem immaturity	🔴 Critical	Always
2. GPU-aware MPI missing	🟠 High	Multi-GPU, small tensors
3. Kernel launch overhead	🟡 Medium	Many small operations
4. Memory management	🟡 Medium	Large intermediate tensors
5. Host-device sync	🟡 Medium	Frequent CPU involvement

---

1. CUDA Ecosystem Immaturity (Most Serious Overall)

This is actually more limiting than GPU-aware MPI for most users:

Aspect	C++/CUDA	Rust
cuBLAS	Native, mature	cudarc bindings (good)
cuTENSOR	Native, mature	Manual FFI (we built this)
cuDNN	Native, mature	cudnn crate (limited)
Custom Kernels	CUDA C++	rust-gpu (experimental) or PTX
Thrust/CUB	Native	❌ No equivalent
NCCL	Native	❌ No bindings
Debugging	cuda-gdb, Nsight	Limited tool support
Profiling	Nsight Systems/Compute	Works but less integrated

Why this matters more:

// In Rust, to write a custom CUDA kernel:
// Option 1: Write PTX assembly (painful)
// Option 2: Use rust-gpu (experimental, limited)
// Option 3: Write CUDA C++ and link via FFI (defeats purpose)

// In C++:
__global__ void my_kernel(...) { /* just works */ }

Impact on tensor networks:

• cuTENSOR handles contractions well ✅
• But custom operations (normalization, activation) need workarounds
• Fused kernels for performance are hard to write

---

2. GPU-Aware MPI Analysis

When GPU-Aware MPI Matters

Communication Time = Latency + (Data Size / Bandwidth)

Without GPU-aware:  GPU → CPU → Network → CPU → GPU
With GPU-aware:     GPU → Network → GPU (via GPUDirect RDMA)

Scenario	GPU-Aware Speedup	Notes
Large tensors (>100MB)	1.5-2x	Bandwidth-bound, staging overhead amortized
Medium tensors (1-100MB)	2-5x	Sweet spot for GPU-aware benefits
Small tensors (<1MB)	5-10x	Latency-dominated, staging kills performance
Single GPU	N/A	No MPI needed

Tensor Network Contraction Profile

For typical tensor network workloads:

Contraction of 2 tensors:
  - Compute: O(n³) for matrix multiply
  - Communication: O(n²) to transfer result
  
Ratio = n (compute/communication)

For n = 1000: Compute dominates (1000:1)
For n = 10: Communication dominates (10:1)

Most tensor contractions are compute-bound because:

• Intermediate tensors are large (compute-heavy)
• We optimize contraction order to minimize intermediate size
• Communication happens between contraction steps

---

3. Kernel Launch Overhead

Each CUDA kernel launch has ~5-10μs overhead:

// Bad pattern (many small launches):
for i in 0..1000 {
    cuda_add(&a[i], &b[i], &c[i]);  // 1000 launches = 5-10ms overhead
}

// Good pattern (batched):
cuda_add_batched(&a, &b, &c);  // 1 launch

In Rust, this is harder to optimize because:

• No easy kernel fusion (no custom kernels)
• cuTENSOR handles contractions but not surrounding ops
• Each operation is a separate launch

Mitigation: omeinsum-rs uses cuTENSOR which batches well

---

4. Memory Management

GPU memory allocation is slow (~100μs per allocation):

// Bad: Allocate per operation
fn contract(a: &Tensor, b: &Tensor) -> Tensor {
    let result = Tensor::zeros(output_shape);  // Allocation here!
    cutensor_contract(a, b, &mut result);
    result
}

// Good: Pre-allocate workspace
fn contract_into(a: &Tensor, b: &Tensor, result: &mut Tensor) {
    cutensor_contract(a, b, result);  // No allocation
}

Current omeinsum-rs status:

• We allocate per contraction ⚠️
• Could add workspace pre-allocation
• cuTENSOR plan cache helps reuse plans

---

5. NCCL vs MPI for Multi-GPU

For multi-GPU on single node, NCCL is often better than MPI:

Aspect	MPI	NCCL
GPU-GPU same node	Via CPU (slow)	NVLink direct (fast)
Multi-node	Good	Good
Collective ops	General	GPU-optimized
Rust bindings	rsmpi ✅	❌ None

This is actually a bigger gap than GPU-aware MPI:

• NCCL provides 10-50x speedup for multi-GPU collectives
• No Rust bindings exist
• Would need to build FFI bindings

---

Revised Priority for GPU Performance

Priority	Issue	Solution
1	NCCL bindings missing	Build Rust NCCL bindings
2	Custom kernel difficulty	Use cuTENSOR where possible, FFI for custom
3	GPU-aware MPI	Unsafe wrapper with careful API
4	Memory allocation overhead	Workspace pre-allocation
5	Kernel launch overhead	Batch operations, use libraries

---

Practical Impact Assessment

For a 100-tensor network contraction on 4 GPUs:

Without optimizations:
  - Contraction time: 10s (cuTENSOR)
  - Communication time: 2s (CPU staging)
  - Memory overhead: 0.5s
  - Total: 12.5s

With GPU-aware MPI only:
  - Communication: 0.4s (5x faster)
  - Total: 10.9s (12% improvement)

With NCCL:
  - Communication: 0.1s (20x faster)
  - Total: 10.6s (15% improvement)

With custom fused kernels:
  - Contraction: 7s (30% faster)
  - Total: 9.5s (24% improvement)

Conclusion: For compute-heavy tensor networks, kernel optimization matters more than communication optimization.

---

Recommendations for omeinsum-rs

Short Term (High Impact, Feasible)

1. Workspace pre-allocation - Reduce allocation overhead
2. Unsafe GPU-aware MPI wrapper - For users who need it
3. Better cuTENSOR integration - Use all features (autotuning, etc.)

Medium Term (High Impact, More Work)

4. NCCL bindings - Critical for multi-GPU
5. Kernel fusion for common patterns - Via PTX or rust-gpu

Long Term (Ecosystem Improvement)

6. Contribute to rust-gpu - Make custom kernels easier
7. Contribute to cudarc - Improve CUDA ecosystem

---

Answer to Original Question

"Is No GPU-aware MPI the most serious issue?"

For single-node multi-GPU: No, missing NCCL bindings is worse (10-50x vs 2-5x impact)

For multi-node GPU clusters: It's top 3, but:

• If tensors are large (>10MB): Medium impact (compute dominates)
• If tensors are small (<1MB): High impact (communication dominates)
• Custom kernel difficulty often matters more for overall performance

For omeinsum-rs specifically: Since we use cuTENSOR for contractions (compute-heavy), GPU-aware MPI is important but not the #1 bottleneck. NCCL bindings and workspace management would likely give bigger wins.

---

🤖 Generated with Claude Code

[GiggleLiu]

Analysis: Main Complaints and Issues with rsmpi

Based on analysis of the rsmpi GitHub issues, here are the key complaints and limitations:

---

Issue Categories Overview

Category	Count	Severity
Type/Memory Safety	6	🔴 Critical
Missing Features	5	🟠 High
API Ergonomics	3	🟡 Medium
Build/Compatibility	3	🟡 Medium
Error Handling	2	🔴 Critical

---

🔴 Critical Issues

1. Type Mismatch Not Caught (#211, #133)

Problem: Send and receive buffers can have mismatched types, compiling without error:

// This compiles but is WRONG - sends u64, receives i32
let send_buf: u64 = 42;
let mut recv_buf: i32 = 0;
comm.all_reduce_into(&send_buf, &mut recv_buf, op);  // Garbage result!

Why it matters:

• Silent data corruption
• Violates Rust's safety guarantees
• MPI standard says this is undefined behavior

Status: Open since 2022, proposed fix needs trait bound changes

---

2. Error Handling Causes Deadlocks (#130)

Problem: MPI errors are currently ignored. If one rank errors and returns early, other ranks deadlock:

fn collective_operation(comm: &Comm) -> Result<(), Error> {
    let data = risky_operation()?;  // If this fails on rank 0...
    comm.all_reduce(&data);          // ...ranks 1-N deadlock here forever
    Ok(())
}

Proposed solutions:

1. result.consensus(comm)? - Collective error agreement
2. Communicator-aware error types with lifetime constraints
3. Just abort all ranks on error (simple but crude)

Status: Open since 2022, no resolution

---

3. Receive Operations Should Be Unsafe (#54)

Problem: Receiving into types with invalid bit patterns (like bool, enums) is unsound:

let mut flags: [bool; 10] = [false; 10];
comm.recv_into(&mut flags);  // Could receive 0x42, which is invalid for bool!

Why it's hard to fix:

• Would require unsafe on all receives, hurting ergonomics
• Or restrict to #[repr(C)] types only

Status: Open since 2018 (7 years!)

---

🟠 High-Priority Missing Features

4. No MPI-IO Support (#178)

Requested: MPI_File_open, MPI_File_read, MPI_File_write, etc.

Impact for tensor networks:

• Cannot checkpoint large distributed tensors efficiently
• Must use rank-based individual files (slower, more complex)

Status: Open, no implementation started

---

5. Hanging with Large Messages (#144)

Problem: Non-blocking sends hang when messages exceed MPI's "eager threshold":

// This deadlocks for large messages!
for dest in destinations {
    WaitGuard::from(comm.immediate_send(&large_data, dest));  // Waits here
}
for src in sources {
    comm.receive_into(&mut buffer, src);  // Never reached
}

Root cause: MPI can't complete large sends until matching receive is posted

Workaround: Use multiple_scope() to post all operations before waiting:

mpi::request::multiple_scope(n, |scope, coll| {
    for dest in destinations { coll.add(comm.immediate_send(scope, &data, dest)); }
    for src in sources { coll.add(comm.immediate_receive_into(scope, &mut buf, src)); }
    coll.wait_all(&mut results);
});

Status: Closed (workaround documented), but API is non-obvious

---

🟡 Medium-Priority Issues

6. API Ergonomics (#174, #199)

Complaints:

• Sending arrays/vectors is verbose
• Lack of simplified examples
• Trait bounds are complex

// Current (verbose):
world.process_at_rank(0).send_with_tag(&data[..], 0);

// Desired (simpler):
world.send(0, &data);

---

7. Custom Structs Difficult (#136)

Problem: Defining MPI-compatible structs requires understanding derive macros:

// Must use derive macro
#[derive(Equivalence)]
#[repr(C)]
struct Particle {
    x: f64,
    y: f64,
    mass: f64,
}

// Padding/alignment issues possible
// No compile-time verification of layout

---

8. Build Complexity (#1, #28)

Historical issues:

• Required bindgen (now optional)
• stddef.h not found errors
• Needs libclang installed
• MPI development headers required

Current state: Improved, but still harder than pure Rust crates

---

📊 Long-Standing Open Issues

Issue	Age	Description
#12	9 years	Thread safety (Send/Sync) semantics
#54	7 years	Unsafe receives for invalid bit patterns
#32	7 years	Multi-language restructuring
#104	5 years	0.7 release tracking

These long-standing issues suggest:

• Core architectural decisions are hard to change
• Maintainer bandwidth is limited
• Some problems may be fundamentally hard in Rust's type system

---

Community Concerns

"Dead Project?" (#109)

In 2021, users asked if the project was abandoned. The response confirmed active maintenance, but:

• Small maintainer team (1-2 active)
• Releases are infrequent
• Large features take years

---

Summary: Top Complaints

Rank	Complaint	Impact on omeinsum-rs
1	Type mismatches compile silently	Must validate types manually
2	Error handling causes deadlocks	Use abort strategy or careful design
3	No MPI-IO	Use per-rank file I/O
4	Large message deadlocks	Use multiple_scope() pattern
5	Receives can be unsound	Stick to primitive types
6	API verbosity	Abstract behind our own layer
7	Build complexity	Document setup clearly

---

Implications for Our Distributed Implementation

1. Wrap rsmpi with type-safe layer:

   // Our wrapper ensures type consistency
   pub fn all_reduce<T: Scalar>(&self, local: &T, global: &mut T, op: Op) {
       // Both parameters have same type T - can't mismatch
   }

2. Use abort-on-error strategy:

   • Configure MPI_ERRORS_ARE_FATAL
   • Simpler than consensus-based error handling
   • Acceptable for batch HPC jobs

3. Document the multiple_scope() pattern:

   • Required for our tree-based contractions
   • Non-obvious to users

4. Avoid problematic features:

   • Skip MPI-IO (use standard file I/O)
   • Stick to #[repr(C)] types with #[derive(Equivalence)]
   • Don't rely on error returns

---

🤖 Generated with Claude Code