Skip to content

Commit

Permalink
Update VLLM serving example for compatibility with new vLLM release (r…
Browse files Browse the repository at this point in the history
…ay-project#46697)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: akshay-anyscale <[email protected]>
  • Loading branch information
akshay-anyscale authored Jul 24, 2024
1 parent 2f4302a commit 519aeab
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion doc/source/serve/doc_code/vllm_openai_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
)
from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
from vllm.entrypoints.openai.serving_engine import LoRAModulePath
from vllm.utils import FlexibleArgumentParser

logger = logging.getLogger("ray.serve")

Expand Down Expand Up @@ -94,7 +95,11 @@ def parse_vllm_args(cli_args: Dict[str, str]):
Currently uses argparse because vLLM doesn't expose Python models for all of the
config options we want to support.
"""
parser = make_arg_parser()
arg_parser = FlexibleArgumentParser(
description="vLLM OpenAI-Compatible RESTful API server."
)

parser = make_arg_parser(arg_parser)
arg_strings = []
for key, value in cli_args.items():
arg_strings.extend([f"--{key}", str(value)])
Expand Down

0 comments on commit 519aeab

Please sign in to comment.