Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions models/integrations/accelerate.mdx
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
---
description: Training and inference at scale made simple, efficient and adaptable
description: Training and inference at scale made simple, efficient, and adaptable
title: Hugging Face Accelerate
keywords: ["multi-GPU", "mixed precision", "DeepSpeed", "FSDP"]
---

Hugging Face Accelerate is a library that enables the same PyTorch code to run across any distributed configuration, to simplify model training and inference at scale.

Accelerate includes a W&B Tracker which we show how to use below. You can also read more about [Accelerate Trackers in Hugging Face](https://huggingface.co/docs/accelerate/main/en/usage_guides/tracking).
Accelerate includes a W&B Tracker, which this page shows how to use to log metrics, configuration, and artifacts from distributed training runs to W&B. For more information, see [Accelerate Trackers in Hugging Face](https://huggingface.co/docs/accelerate/main/en/usage_guides/tracking).

## Start logging with Accelerate

To get started with Accelerate and W&B you can follow the pseudocode below:
This section shows how to configure Accelerate to log experiment data to W&B during training. To get started with Accelerate and W&B, follow this pseudocode:

```python
from accelerate import Accelerator
Expand All @@ -34,33 +35,33 @@ accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=global_step)
accelerator.end_training()
```

Explaining more, you need to:
1. Pass `log_with="wandb"` when initialising the Accelerator class
In more detail:
1. Pass `log_with="wandb"` when you initialize the `Accelerator` class.
2. Call the [`init_trackers`](https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.init_trackers) method and pass it:
- a project name via `project_name`
- any parameters you want to pass to [`wandb.init()`](/models/ref/python/functions/init) via a nested dict to `init_kwargs`
- any other experiment config information you want to log to your wandb run, via `config`
3. Use the `wandb.Run.log()` method to log to Weigths & Biases; the `step` argument is optional
4. Call `.end_training()` when finished training
- A project name via `project_name`.
- Any parameters you want to pass to [`wandb.init()`](/models/ref/python/functions/init) through a nested dict to `init_kwargs`.
- Any other experiment config information you want to log to your wandb run, through `config`.
3. Use the `wandb.Run.log()` method to log to W&B. The `step` argument is optional.
4. Call `.end_training()` when training finishes.

## Access the W&B tracker

To access the W&B tracker, use the `Accelerator.get_tracker()` method. Pass in the string corresponding to a trackers `.name` attribute, which returns the tracker on the `main` process.
Once Accelerate logs to W&B, you may want direct access to the underlying W&B run object to log artifacts, custom charts, or other data that the tracker doesn't expose. To access the W&B tracker, use the `Accelerator.get_tracker()` method. Pass in the string corresponding to a tracker's `.name` attribute, which returns the tracker on the `main` process.

```python
wandb_tracker = accelerator.get_tracker("wandb")

```
From there you can interact with wandb’s run object like normal:
From there, you can interact with the `wandb` run object as usual:

```python
wandb_tracker.log_artifact(some_artifact_to_log)
```

<Warning>
Trackers built in Accelerate will automatically execute on the correct process, so if a tracker is only meant to be ran on the main process it will do so automatically.
Trackers built in Accelerate automatically execute on the correct process, so if a tracker only needs to run on the main process it does so automatically.

If you want to truly remove Accelerates wrapping entirely, you can achieve the same outcome with:
To remove Accelerate's wrapping entirely, you can achieve the same outcome with:

```python
wandb_tracker = accelerator.get_tracker("wandb", unwrap=True)
Expand All @@ -70,13 +71,14 @@ with accelerator.on_main_process:
</Warning>

## Accelerate articles
Below is an Accelerate article you may enjoy

For a deeper walkthrough of using Accelerate with W&B, see the following article.

<details>

<summary>HuggingFace Accelerate Super Charged With W&B</summary>

* In this article, we'll look at what HuggingFace Accelerate has to offer and how simple it is to perform distributed training and evaluation, while logging results to W&B.
This article looks at what HuggingFace Accelerate offers and how to perform distributed training and evaluation while logging results to W&B.

Read the [Hugging Face Accelerate Super Charged with W&B report](https://wandb.ai/gladiator/HF%20Accelerate%20+%20W&B/reports/Hugging-Face-Accelerate-Super-Charged-with-Weights-Biases--VmlldzoyNzk3MDUx?utm_source=docs&utm_medium=docs&utm_campaign=accelerate-docs).
</details>
Expand Down
Loading
Loading