Sharing datasets cache across different users in linux

[halean]

Is it possible to share the datasets cache (and transformer models cache) to different users in ubuntu. I try to use the same cache location, but got the permission errors for the lock file.
Many thanks.

[mariosasko]

Hi! You can easily share the dataset by pushing it to the Hub (ds.push_to_hub), or if you want to keep the dataset private, you can create an organization on the Hub, invite other members and then push the dataset as follows:

# push to the Hub
ds.push_to_hub(f"{org_name}/{ds_name}", private=True)
# load from the Hub
ds = load_dataset(f"{org_name}/{ds_name}", use_auth_token=True)

[kleag]

Hi, I have the same question and your answer is not a solution for me because the reason we have to share a dataset is that we are on a machine that don't have access to the Internet when running our models. So, we want to download and cache the dataset one time, in a read only mode, and use it for different users. This is for public datasets on huggingface.

[FarisHijazi]

have you tried creating symbolic links using ln -s otheruser/.cache ~/.cache?

[AlanBlanchet]

You can maybe try changing the ENV variables for the cache to some shared space on a disk or private network.

Personnaly I have this piece of code in /etc/profile.d/custom_env.sh :

# Users can share hugging face datasets
HF="/disk/huggingface"
export HF_DATASETS_CACHE="$HF/datasets"
export HF_HUB_CACHE="$HF"