Fix sign-extension bug in fbgemm MX4 Python reference dequantize (#5706)#5706
Open
purvisa-at-meta wants to merge 2 commits intopytorch:mainfrom
Open
Fix sign-extension bug in fbgemm MX4 Python reference dequantize (#5706)#5706purvisa-at-meta wants to merge 2 commits intopytorch:mainfrom
purvisa-at-meta wants to merge 2 commits intopytorch:mainfrom
Conversation
Contributor
|
@purvisa-at-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D102195911. |
…h#5705) Summary: X-link: facebookresearch/FBGEMM#2642 Scale exponents are stored as uint8 bit patterns in int8 storage. When cast to int16, biased values >= 128 were incorrectly sign-extended, producing wildly wrong negative exponents. Apply & 0xFF mask to recover the original unsigned value before subtracting FP32_EXP_BIAS. Same fix as D101680517 but for the upstream fbgemm_gpu Triton kernel. GH PR: pytorch#5705 Reviewed By: q10 Differential Revision: D102195912
…orch#5706) Summary: X-link: facebookresearch/FBGEMM#2643 py_dequantize_mx4 viewed packed data as int8 and subtracted FP32_EXP_BIAS directly. For biased exponents >= 128, the int8 value is negative, producing incorrect results. Fix by viewing as uint8 then casting to int32 before subtracting the bias. Same class of bug as D101680517 and the Triton kernel fix. GH PR: pytorch#5706 Reviewed By: q10 Differential Revision: D102195911
meta-codesync
Bot
changed the title
purvisa-at-meta
force-pushed
the
export-D102195911
branch
from
bb5405f to
0de13f3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
X-link: https://github.com/facebookresearch/FBGEMM/pull/2643
py_dequantize_mx4 viewed packed data as int8 and subtracted
FP32_EXP_BIAS directly. For biased exponents >= 128, the int8 value
is negative, producing incorrect results. Fix by viewing as uint8
then casting to int32 before subtracting the bias.
Same class of bug as D101680517 and the Triton kernel fix.
GH PR: #5706
Reviewed By: q10
Differential Revision: D102195911