everything im learning about GPUs. how they can be used. how they can be optimized for ml workloads.

setup

memory usage

profiling

activation memory

activation recomputation

gradient accumulation

data parallelism

zero

configuration

model parallelism