Skip to content

Add ReasoningParser#287

Open
junya-takayama wants to merge 6 commits into
mainfrom
add_reasoning_parser
Open

Add ReasoningParser#287
junya-takayama wants to merge 6 commits into
mainfrom
add_reasoning_parser

Conversation

@junya-takayama
Copy link
Copy Markdown
Collaborator

@junya-takayama junya-takayama commented May 21, 2026

Add a mechanism to the VLLM and HuggingFaceLM classes to extract the reasoning content and store it in LMOutput.reasoning_text.

Example

flexeval_lm --eval_setup rakuda-v2-ja --language_model VLLM --language_model.model Qwen/Qwen3-4B --language_model.reasoning_parser UnifiedRegexReasoningParser --language_model.reasoning_parser.pattern="<think>(?P<reasoning_content>.*?)</think>(?P<content>.*)"
...
2026-05-21 21:04:48.236 | INFO     | flexeval.core.evaluate_chat_response:evaluate_chat_response:181 - Example of the output
2026-05-21 21:04:48.236 | INFO     | flexeval.core.evaluate_chat_response:evaluate_chat_response:182 - {
    "lm_output": {
        "text": "\n\n四国地方には以下の4つの都道府県があります。それぞれの県庁所在地を以下に列挙します。\n\n1. **香川県(かわかん)**  \n   - **県庁所在地**:香川市(かわかし)\n\n2. **岡山県(おうさんけん)**  \n   - **県庁所在地**:岡山市(おうさんし)\n\n3. **徳島県(とくしまけん)**  \n   - **県庁所在地**:徳島市(とくしまし)\n\n4. **広島県(ひろしまけん)**  \n   - **県庁所在地**:広島市(ひろしまし)\n\n※四国地方は日本の四つの地方の一つで、これらの4県が含まれます。",
        "raw_text": "<think>\nOkay, the user is asking for the four prefectures in the Shikoku region and their respective prefectural capitals. Let me start by recalling the four prefectures in Shikoku. I think they are Kagawa, Okayama, Tokushima, and Hiroshima. Wait, but I need to make sure. Let me think again. Shikoku is the fourth largest island in Japan, right? The four prefectures there are Kagawa, Okayama, Tokushima, and Hiroshima. Yes, that's correct.\n\nNow, the prefectural capitals. For Kagawa, I believe the capital is Kagawa City. Okayama's capital is Okayama City. Tokushima's capital is Tokushima City. And Hiroshima's capital is Hiroshima City. Let me double-check each one to be sure. Kagawa City is indeed the capital of Kagawa Prefecture. Okayama City is the capital of Okayama Prefecture. Tokushima City is the capital of Tokushima Prefecture. Hiroshima City is the capital of Hiroshima Prefecture. That seems right. \n\nWait, but sometimes people might confuse the capitals. For example, Hiroshima is a major city, but is it the capital? Yes, Hiroshima City is the capital of Hiroshima Prefecture. Okay, so the answer should be the four prefectures and their capitals as listed. I think that's all. Let me just confirm once more. Yes, those are the four prefectures in Shikoku, and their capitals are as mentioned. No mistakes there.\n</think>\n\n四国地方には以下の4つの都道府県があります。それぞれの県庁所在地を以下に列挙します。\n\n1. **香川県(かわかん)**  \n   - **県庁所在地**:香川市(かわかし)\n\n2. **岡山県(おうさんけん)**  \n   - **県庁所在地**:岡山市(おうさんし)\n\n3. **徳島県(とくしまけん)**  \n   - **県庁所在地**:徳島市(とくしまし)\n\n4. **広島県(ひろしまけん)**  \n   - **県庁所在地**:広島市(ひろしまし)\n\n※四国地方は日本の四つの地方の一つで、これらの4県が含まれます。",
        "reasoning_text": "\nOkay, the user is asking for the four prefectures in the Shikoku region and their respective prefectural capitals. Let me start by recalling the four prefectures in Shikoku. I think they are Kagawa, Okayama, Tokushima, and Hiroshima. Wait, but I need to make sure. Let me think again. Shikoku is the fourth largest island in Japan, right? The four prefectures there are Kagawa, Okayama, Tokushima, and Hiroshima. Yes, that's correct.\n\nNow, the prefectural capitals. For Kagawa, I believe the capital is Kagawa City. Okayama's capital is Okayama City. Tokushima's capital is Tokushima City. And Hiroshima's capital is Hiroshima City. Let me double-check each one to be sure. Kagawa City is indeed the capital of Kagawa Prefecture. Okayama City is the capital of Okayama Prefecture. Tokushima City is the capital of Tokushima Prefecture. Hiroshima City is the capital of Hiroshima Prefecture. That seems right. \n\nWait, but sometimes people might confuse the capitals. For example, Hiroshima is a major city, but is it the capital? Yes, Hiroshima City is the capital of Hiroshima Prefecture. Okay, so the answer should be the four prefectures and their capitals as listed. I think that's all. Let me just confirm once more. Yes, those are the four prefectures in Shikoku, and their capitals are as mentioned. No mistakes there.\n",
...

@junya-takayama junya-takayama force-pushed the add_reasoning_parser branch from 3827b57 to 9661b17 Compare May 21, 2026 09:57
@junya-takayama junya-takayama marked this pull request as ready for review May 21, 2026 12:06
@junya-takayama junya-takayama changed the title [WIP] Add ReasoningParser Add ReasoningParser May 21, 2026
]
lm_outputs = self._batch_complete_text(chat_messages_as_string, **kwargs)
if self.reasoning_parser:
for lm_output in lm_outputs:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ここではtoolのところにある

            if lm_output.text is None:
                continue

的な処理はいらないんですかね?

lm_output.text = reasoning.text
lm_output.reasoning_text = reasoning.reasoning_text

if self.tool_parser:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resoning parserとtool_parserの同時使用って想定されていますか?
想定しているなら下でraw_textとかが上書きされてしまうのでまずい気がします

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants