Skip to content

Commit

Permalink
[Test][Client] Only start ray once in client tests (ray-project#28835)
Browse files Browse the repository at this point in the history
It looks like we're frequently starting and shutting down Ray in this test because `ray_start_client_server` isn't connecting to the Ray created by `ray_start_regular_shared`, and is instead starting a new Ray head process every time it launches.

Ray client tests are failing frequently with:

```
[2022-10-06 07:31:46,253 E 13235 13751] core_worker_process.cc:277: The core worker has already been shutdown. This happens when the language frontend accesses the Ray's worker after it is shutdown. The process will exit
```

Which is probably caused by having multiple ray clusters running simultaneous, with some shutting down asynchronously. This refactor forces all of the tests in the module to use the same Ray cluster.

Also fixes two other sources of potential flakiness:
* Joins the thread in test_client_thread_safe (seems like this has a bad interaction when the client server is cleaned up)
* Calls ray.get in `test_stdout_log_stream`, to make sure that the remote function is done running before we try searching for its output

Should also have the happy side effect of speeding up test_client.

Ran the `Small & Client` tests (regular and external redis) twice each, no flakes, and windows version of test_client.
  • Loading branch information
ckw017 authored Oct 11, 2022
1 parent d89a664 commit 297341e
Show file tree
Hide file tree
Showing 3 changed files with 142 additions and 116 deletions.
6 changes: 6 additions & 0 deletions python/ray/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,12 @@ def ray_start_object_store_memory(request, maybe_external_redis):

@pytest.fixture
def call_ray_start(request):
with call_ray_start_context(request) as address:
yield address


@contextmanager
def call_ray_start_context(request):
default_cmd = (
"ray start --head --num-cpus=1 --min-worker-port=0 "
"--max-worker-port=0 --port 0"
Expand Down
Loading

0 comments on commit 297341e

Please sign in to comment.