Skip to content

Commit

Permalink
Fix the large range issue on QuantizeAvx2 (pytorch#1784)
Browse files Browse the repository at this point in the history
Summary:
Pull Request resolved: pytorch#1784

We have seg fault when the range of tensor quantization is big. The common int32_t vs. int64_t issue.

Reviewed By: sryap

Differential Revision: D46123138

fbshipit-source-id: d5f0c9c2b9a2d10159e7d227d1207dad3917aa87
  • Loading branch information
jianyuh authored and facebook-github-bot committed May 24, 2023
1 parent 6c216be commit f97fe45
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/QuantUtilsAvx2.cc
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ void QuantizeAvx2(
// that is exactly representable in float
constexpr int32_t int32_float_max_val =
std::numeric_limits<int32_t>::max() - 127;
int i = 0;
int64_t i = 0;
float inverse_scale = 1.f / qparams.scale;
__m256 inverse_scale_v = _mm256_set1_ps(inverse_scale);
// clang-format off
Expand Down

0 comments on commit f97fe45

Please sign in to comment.