pyproject.toml file explicitly requires CUDA Toolkit 12.9:
|
[tool.uv.sources] |
|
torch = [ |
|
{ index = "pytorch-cu129", extra = "gpu", marker = "sys_platform == 'linux'" }, |
|
] |
|
[[tool.uv.index]] |
|
name = "pytorch-cu129" |
|
url = "https://download.pytorch.org/whl/cu129" |
|
explicit = true |
Which may be the environment configured out-of-the-box for enrolled students, but for anyone following at home and leasing a GPU, it is likely not the correct version -- many machines come with 13.x by default. In some cases, one can downgrade to an older version but not in others. Alternatively, they could try specifying a higher version in pyproject.toml, but that may require different package versions which could cause other conflicts or unexpected issues.
One should make sure there is enough disk space. The dependencies are very heavy. Imagine my surprise when I discovered that the 40GB hard drive space I allocated is too small for this project.
vLLM only works on Linux, so Windows and Mac are out of question. (If possible, I hope the staff could look into alternatives to vLLM. vLLM is quite heavy and slow to start.)
Finally, I remember I had issues with using the latest GPUs and had to use a one-generation-old GPU to get things done, although I don't quite remember the context (might be the outdated vLLM version, already updated recently).
I probably wasted $10 just to figure out the correct environment for this assignment. Many of these issues are not obvious until after I install all the dependencies and run my code.
It might be a good idea to ask your sponsor to provide a template that would create the exact environments Stanford students get to use. I would not have minded using that instead of runpod or vast.ai.
pyproject.tomlfile explicitly requires CUDA Toolkit 12.9:assignment5-alignment/pyproject.toml
Lines 43 to 50 in a8ad44f
Which may be the environment configured out-of-the-box for enrolled students, but for anyone following at home and leasing a GPU, it is likely not the correct version -- many machines come with 13.x by default. In some cases, one can downgrade to an older version but not in others. Alternatively, they could try specifying a higher version in pyproject.toml, but that may require different package versions which could cause other conflicts or unexpected issues.
One should make sure there is enough disk space. The dependencies are very heavy. Imagine my surprise when I discovered that the 40GB hard drive space I allocated is too small for this project.
vLLM only works on Linux, so Windows and Mac are out of question. (If possible, I hope the staff could look into alternatives to vLLM. vLLM is quite heavy and slow to start.)
Finally, I remember I had issues with using the latest GPUs and had to use a one-generation-old GPU to get things done, although I don't quite remember the context (might be the outdated vLLM version, already updated recently).
I probably wasted $10 just to figure out the correct environment for this assignment. Many of these issues are not obvious until after I install all the dependencies and run my code.
It might be a good idea to ask your sponsor to provide a template that would create the exact environments Stanford students get to use. I would not have minded using that instead of runpod or vast.ai.