Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix interleaved_numeric_conversion.h #582

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

yz-chen18
Copy link

uint4 fast cast should minus 64.5 for elt_45 and elt_67, which is 1032 / 16

uint4 fast cast should minus 64.5 for elt_45 and elt_67, which is 1032 / 16
@yz-chen18 yz-chen18 changed the title Update interleaved_numeric_conversion.h Fix interleaved_numeric_conversion.h Apr 26, 2023
@byshiue
Copy link
Collaborator

byshiue commented Apr 27, 2023

Can you explain what problem do you encounter? How do reproduce the issue you encounter?

@rhenry-nv
Copy link

rhenry-nv commented Apr 27, 2023

@yz-chen18 In addition to @byshiue's question, I am not sure your suggestion is correct. Due to the bit positioning, we have:

elt_45/elt_67 = 1024 + 16X

Then, we divide by 16 with fma to give us 64 + X. However the original value is biased to be unsigned so, X = O + 8 where O is the original, signed value. So to extract the final value we must do X - 64 - 8 which gives -72. Do you have an example where that produces incorrect results?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants