Regarding Running LLM Benchmark

[ojuschugh1]

Hi,
I am trying to run the llm benchmark using ianvs -f command but getting error related to the format, Could anyone please help me out with this.

Thanks and Regards

Traceback (most recent call last): File "/mnt/c/users/ojus/desktop/lfx/myenv/bin/ianvs", line 33, in <module> sys.exit(load_entry_point('ianvs==0.1.0', 'console_scripts', 'ianvs')()) File "/mnt/c/users/ojus/desktop/lfx/ianvs/core/cmd/benchmarking.py", line 41, in main raise RuntimeError(f"benchmarkingjob runs failed, error: {err}.") from err RuntimeError: benchmarkingjob runs failed, error: testenv config file(/mnt/c/users/ojus/desktop/lfx/ianvs/examples/llm/singletask_learning_bench/simple_qa/testenv/testenv.yaml) is not supported, error: dataset file(/mnt/c/users/ojus/desktop/lfx/datasets/llm_simple_qa/train_dataset/data.jsonl)'s format(jsonl) is not supported..

[ojuschugh1]

Hi @hsj576 , @MooreZheng , I tried to fix it on my own now. The error gives related to categories-: I am trying to run single qna benchmark

`
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/c/users/ojus/desktop/lfx/myenv/bin/ianvs", line 33, in <module>
    sys.exit(load_entry_point('ianvs==0.1.0', 'console_scripts', 'ianvs')())
  File "/mnt/c/users/ojus/desktop/lfx/myenv/lib/python3.10/site-packages/ianvs-0.1.0-py3.10.egg/core/cmd/benchmarking.py", line 41, in main
RuntimeError: benchmarkingjob runs failed, error: testcase(id=143a5c54-5f0b-11ef-a826-071539134351) runs failed, error: (paradigm=singletasklearning) pipeline runs failed, error: 'categories'.`

[ojuschugh1]

Traceback (most recent call last):
  File "/mnt/c/users/ojus/desktop/e/myenv/bin/ianvs", line 33, in <module>
    sys.exit(load_entry_point('ianvs==0.1.0', 'console_scripts', 'ianvs')())
  File "/mnt/c/users/ojus/desktop/e/myenv/lib/python3.10/site-packages/ianvs-0.1.0-py3.10.egg/core/cmd/benchmarking.py", line 41, in main
RuntimeError: benchmarkingjob runs failed, error: testcase(id=d060f824-61dc-11ef-8b3d-8d77708493e4) runs failed, error: (paradigm=lifelonglearning) pipeline runs failed, error: module(type=basemodel loads class(name=BaseModel) failed, error: load module(url=/mnt/c/users/ojus/desktop/e/ianvs/examples/scene-based-unknown-task-recognition//lifelong_learning_bench/testalgorithms/rfnet/basemodel.py) failed, error: No module named 'mypath'..

[hsj576]

Traceback (most recent call last):
  File "/mnt/c/users/ojus/desktop/e/myenv/bin/ianvs", line 33, in <module>
    sys.exit(load_entry_point('ianvs==0.1.0', 'console_scripts', 'ianvs')())
  File "/mnt/c/users/ojus/desktop/e/myenv/lib/python3.10/site-packages/ianvs-0.1.0-py3.10.egg/core/cmd/benchmarking.py", line 41, in main
RuntimeError: benchmarkingjob runs failed, error: testcase(id=d060f824-61dc-11ef-8b3d-8d77708493e4) runs failed, error: (paradigm=lifelonglearning) pipeline runs failed, error: module(type=basemodel loads class(name=BaseModel) failed, error: load module(url=/mnt/c/users/ojus/desktop/e/ianvs/examples/scene-based-unknown-task-recognition//lifelong_learning_bench/testalgorithms/rfnet/basemodel.py) failed, error: No module named 'mypath'..

To fix this error you need to export the right pythonpath. For example: "export PYTHONPATH=$PYTHONPATH:/mnt/c/users/ojus/desktop/e/ianvs/examples/scene-based-unknown-task-recognition/lifelong_learning_bench/testalgorithms/rfnet"

[ojuschugh1]

Hi @hsj576 thanks for the reply. I have successfully run the ianvs and created the example even though I can;t make till the deadline for LFX. I face issue while downloading the model from baidu cloud as I from India and I am not able to access that website even after VPN I need Chinese number to verify the account so face a lot of issues.

But thanks for the help. Good luck !!