Skip to content

Commit

Permalink
Remove duplicate call to sync_metadata inside DagFileProcessorManager (
Browse files Browse the repository at this point in the history
…apache#15121)

`_process_message` already calls sync_metadata if it's a DagParsingStat,
so there's no need to call it again.  This isn't expensive, but no point
doing it

I've also added a runtime check to ensure this function is only used in
sync mode, which makes the purpose/logic used in this function clearer.
  • Loading branch information
ashb authored Mar 31, 2021
1 parent b0e68eb commit 116a8a0
Showing 1 changed file with 5 additions and 4 deletions.
9 changes: 5 additions & 4 deletions airflow/utils/dag_processing.py
Original file line number Diff line number Diff line change
Expand Up @@ -313,16 +313,17 @@ def wait_until_finished(self) -> None:
"""Waits until DAG parsing is finished."""
if not self._parent_signal_conn:
raise ValueError("Process not started.")
if self._async_mode:
raise RuntimeError("wait_until_finished should only be called in sync_mode")
while self._parent_signal_conn.poll(timeout=None):
try:
result = self._parent_signal_conn.recv()
except EOFError:
break
return
self._process_message(result)
if isinstance(result, DagParsingStat):
# In sync mode we don't send this message from the Manager
# until all the running processors have finished
self._sync_metadata(result)
# In sync mode (which is the only time we call this function) we don't send this message from
# the Manager until all the running processors have finished
return

@staticmethod
Expand Down

0 comments on commit 116a8a0

Please sign in to comment.