Question on Result Discrepancies Between Framework Reproduction and Original Paper

[XU-zhaojing]

Hello, thank you very much for your excellent work on this project!
While reproducing some models using your framework, I noticed that the results differ significantly from those reported in the original papers and their official implementations.
Could you please share your thoughts on what might cause such discrepancies?
Looking forward to your reply, and thank you again for your great contribution!

[enoche]

Hi, @XU-zhaojing Thank you for your feedback and interest in this project! Based on my own experiments, I've noticed significant performance variations in some models (both with the MMRec framework and their official implementations) compared to reported results.
These differences may stem from:

0. Incorrect implementation or overstated performance claims.
1. Data splitting variations between this framework and official implementations.
2. Metric implementation issues. Some official implementations use the test set as validation, which can skew results.
3. Hardware/software differences. These typically have a minor impact on performance.

Please report any models you suspect have non-reproducible results, and I'll check them and note them in the repo. if necessary.

[XU-zhaojing]

Subject: Suspected Reproducibility Issue with Model PGL Hi , Thank you for the prompt reply and for outlining the possible sources of deviation. After thoroughly checking the data splits, metric implementations,  I suspect the performance gap I observe with model PGL (in the MMRec framework) may be due to factors (1) or (4) in your list. Could you please verify the PGL results logged in the repository? I’m happy to provide my detailed logs, hyper-parameters, and environment details if that would help with the diagnosis. Additionally, I have noticed that users have raised similar concerns about the reproducibility of PGL’s reported results in the issue tracker of its original repository. Looking forward to your confirmation and any notes you can add to the repo. Best regards, XU-zhaojing

…

----------回复的邮件信息---------- Zhou Xin ***@***.***> 在 Mon,Aug 11,2025 4:04 PM写道： enoche left a comment (enoche/MMRec#58) Hi, @XU-zhaojing Thank you for your feedback and interest in this project! Based on my own experiments, I've noticed significant performance variations in some models (both with the MMRec framework and their official implementations) compared to reported results. These differences may stem from: Incorrect implementation or overstated performance claims. Data splitting variations between this framework and official implementations. Metric implementation issues. Some official implementations use the test set as validation, which can skew results. Hardware/software differences. These typically have a minor impact on performance. Please report any models you suspect have non-reproducible results, and I'll check them and note them in the repo. if necessary. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[enoche]

Noted with thanks. Will check PGL soon~