Welcome to TorchAcc’s documentation!

TorchAcc

• Introduction
  • Main Features
  • Model Performance
• Installation
  • Docker images
  • Building from Source
• Quick Start
  • Torch Native Task
  • Single GPU Acceleration with TorchAcc
  • Multiple GPUs Acceleration with TorchAcc

Functionality

• Data Parallel
  • Torch Native Task
  • Single GPU Acceleration with TorchAcc
  • Data Parallel
  • Auto Mixed Precision (AMP)
• FSDP (Fully Sharded Data Parallel)
  • Torch Native Task
  • FSDP
  • Checkpoint Save/Load
  • Configurable parameters
• Data Bucketing
  • Introduction
  • How to use

Tutorials

• Flash Models
• HuggingFace Transformers
  • Environment Preparation
  • PyTorch Native Training
  • DeepSpeed Training
  • TorchAcc Training
  • Performance

Best Practices

• Best Practices
  • Minimize the Calls to sync
  • Prefer AsyncLoader
  • Avoid Evaluating Tensors
  • Coordinate Gradient Accumulation with sync and AsyncLoader
  • Model Saving

API Reference

• torchacc
  • torchacc package
    • Submodules
    • torchacc.accelerate module
    • torchacc.config module
    • Module contents
    • Subpackages

CONTRIBUTING

• Contribute To TorchAcc
  • Building from source

Indices and tables

• Index

• Module Index

• Search Page