Skip to content

Commit

Permalink
[RPC] Fix a few flaky RPC tsan tests (pytorch#71460)
Browse files Browse the repository at this point in the history
Summary:
Pull Request resolved: pytorch#71460

When running with TSAN, we use a larger RPC timeout: https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/dist_utils.py#L68. As a result, the assertions here are invalid.

Tried to fix this by just setting `self.rpc_backend_options.rpc_timeout` to the new timeout, but `rpc_backend_options` is reconstructed every time it is accessed, so this doesn't work:: https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/distributed/rpc/tensorpipe_rpc_agent_test_fixture.py#L15

Just removing the asserts should be fine as they don't really add value to what's being tested.
ghstack-source-id: 147208455

Test Plan: CI

Reviewed By: fduwjj

Differential Revision: D33648421

fbshipit-source-id: 9a5052b1c851fe7f838792d8bdf17d0563b4aa00
(cherry picked from commit 96ddab3)
  • Loading branch information
rohan-varma authored and pytorchmergebot committed Jan 19, 2022
1 parent 9515213 commit fbc3b8c
Showing 1 changed file with 0 additions and 2 deletions.
2 changes: 0 additions & 2 deletions torch/testing/_internal/distributed/rpc/rpc_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -4747,7 +4747,6 @@ def test_rpc_builtin_timeout(self):

# Ensure that the currently set default timeout is large enough such
# that RPCs with delays still complete.
self.assertEqual(rpc.constants.DEFAULT_RPC_TIMEOUT_SEC, rpc.get_rpc_timeout())
fut = rpc.rpc_async(
dst_worker, torch.add, args=(torch.tensor(1), torch.tensor(1))
)
Expand Down Expand Up @@ -4783,7 +4782,6 @@ def test_rpc_script_timeout(self):

# Ensure that the currently set default timeout is large enough such
# that RPCs with delays still complete.
self.assertEqual(rpc.constants.DEFAULT_RPC_TIMEOUT_SEC, rpc.get_rpc_timeout())
fut = rpc.rpc_async(
dst_worker, my_script_func, args=(torch.tensor(1),)
)
Expand Down

0 comments on commit fbc3b8c

Please sign in to comment.