Refactor LLMEdgeManager's to_dtype #9542
Conversation
iseeyuan
requested review from
jackzhxng,
larryliu0820,
swolchok and
lucylq
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9542
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit 9192d58 with merge base 012f120 ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@pytorchbot label "topic: not user facing" |
| @@ -588,25 +588,9 @@ def _prepare_for_llama_export(args) -> LLMEdgeManager: | |||
| ) | |||
|
|
|||
| # At this point, the model is loaded in the default fp32. | |||
|
|
|||
| # Checkpoint dtype should be lower or equal precision to the dtype override. | |||
| # override dtype | |||
| checkpoint_dtype = edge_manager.model.checkpoint_dtype | |||
There was a problem hiding this comment.
Not sure why we need to keep this original checkpoint_dtype, since it would be overriden in the line below. Just to keep the behavior of the existing code. @jackzhxng and @kimishpatel , what's the reason behind this, for weight quantization?
There was a problem hiding this comment.
Yep it's for quantization
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| @@ -588,25 +588,9 @@ def _prepare_for_llama_export(args) -> LLMEdgeManager: | |||
| ) | |||
|
|
|||
| # At this point, the model is loaded in the default fp32. | |||
|
|
|||
| # Checkpoint dtype should be lower or equal precision to the dtype override. | |||
| # override dtype | |||
| checkpoint_dtype = edge_manager.model.checkpoint_dtype | |||
There was a problem hiding this comment.
Yep it's for quantization
|
@iseeyuan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: Small refactor to aggregate dtype override in to_dtype method. Test Plan: CI Reviewed By: billmguo Differential Revision: D71736608 Pulled By: iseeyuan
facebook-github-bot
force-pushed
the
dtype
branch
from
3c6c1ab to
9192d58
Compare
|
This pull request was exported from Phabricator. Differential Revision: D71736608 |
Summary
Small refactor to aggregate dtype override in to_dtype method.
Test plan
CI