Not passing in tests

[gordicaleksa]

Hey Greg! Cool project!

It looks like you never pass the actual test case that the LLM generated for testing here:
https://github.com/Oxen-AI/GRPO-With-Cargo-Feedback/blob/main/train.py#L218

Or am i misunderstanding something? :)

[gschoeni]

Hey @gordicaleksa the task was to generate a function and the unit tests for the function in the same output, so setup_and_test_rust_project simply makes a single file rust project and puts the generated code+tests in it, then runs the reward functions run the different build tools (cargo build, clippy, test).

I outline the whole process here:
https://www.oxen.ai/blog/training-a-rust-1-5b-coder-lm-with-reinforcement-learning-grpo

You can look at some sample outputs from the trained 1.5B model here:
https://www.oxen.ai/ox/Rust/file/main/results/GRPO_82_2025-03-02_22-49-17_Qwen2.5-Coder-1.5B-Instruct/results_code_and_tests.parquet

The one issue with the benchmark is if the LLM learns to write really trivial tests to pass the rewards, I want to do some further studies on the output for that behavior.

[gordicaleksa]

Oh ok got it! I thought that extract_rust_code would only extract code and extract_test_code would extract the tests but my assumption was wrong, the former extracts both.

tnx!