From 597e32d9a0fb08cabc09d1a6449ec5707b05c984 Mon Sep 17 00:00:00 2001 From: zepingguo Date: Wed, 13 Nov 2024 17:17:19 +0800 Subject: [PATCH 1/8] update doc --- .../running-jobs/environment-variables.rst | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index f7138af95fa..db9c46ac7ad 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -24,9 +24,19 @@ You can specify environment variables to be made available to a task in two ways envs: MYVAR: val - + - ``--env`` flag in ``sky launch/exec`` :ref:`CLI ` (takes precedence over the above) +- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI ` (lowest precedence over the above) + + .. code-block:: text + + # cat my_app.env + # --env-file my_app.env + MYVAR=val + WANDB_API_KEY=MY_WANDB_API_KEY + HF_TOKEN=MY_HF_TOKEN + .. tip:: To mark an environment variable as required and make SkyPilot forcefully check @@ -145,9 +155,9 @@ Environment variables for ``setup`` - 0 * - ``SKYPILOT_SETUP_NODE_IPS`` - A string of IP addresses of the nodes in the cluster with the same order as the node ranks, where each line contains one IP address. - + Note that this is not necessarily the same as the nodes in ``run`` stage: the ``setup`` stage runs on all nodes of the cluster, while the ``run`` stage can run on a subset of nodes. - - + - .. code-block:: text 1.2.3.4 @@ -158,19 +168,19 @@ Environment variables for ``setup`` - 2 * - ``SKYPILOT_TASK_ID`` - A unique ID assigned to each task. - - This environment variable is available only when the task is submitted + + This environment variable is available only when the task is submitted with :code:`sky launch --detach-setup`, or run as a managed spot job. - + Refer to the description in the :ref:`environment variables for run `. - sky-2023-07-06-21-18-31-563597_myclus_1 - + For managed spot jobs: sky-managed-2023-07-06-21-18-31-563597_my-job-name_1-0 * - ``SKYPILOT_CLUSTER_INFO`` - A JSON string containing information about the cluster. To access the information, you could parse the JSON string in bash ``echo $SKYPILOT_CLUSTER_INFO | jq .cloud`` or in Python : .. code-block:: python - + import json json.loads( os.environ['SKYPILOT_CLUSTER_INFO'] @@ -200,7 +210,7 @@ Environment variables for ``run`` - 0 * - ``SKYPILOT_NODE_IPS`` - A string of IP addresses of the nodes reserved to execute the task, where each line contains one IP address. Read more :ref:`here `. - - + - .. code-block:: text 1.2.3.4 @@ -221,13 +231,13 @@ Environment variables for ``run`` If a task is run as a :ref:`managed spot job `, then all recoveries of that job will have the same ID value. The ID is in the format "sky-managed-_(_)_-", where ```` will appear when a pipeline is used, i.e., more than one task in a managed spot job. Read more :ref:`here `. - sky-2023-07-06-21-18-31-563597_myclus_1 - + For managed spot jobs: sky-managed-2023-07-06-21-18-31-563597_my-job-name_1-0 * - ``SKYPILOT_CLUSTER_INFO`` - A JSON string containing information about the cluster. To access the information, you could parse the JSON string in bash ``echo $SKYPILOT_CLUSTER_INFO | jq .cloud`` or in Python : .. code-block:: python - + import json json.loads( os.environ['SKYPILOT_CLUSTER_INFO'] From 5beb6767cadd28886f572a2322a2c1b68b3285a2 Mon Sep 17 00:00:00 2001 From: zepingguo Date: Wed, 13 Nov 2024 17:24:23 +0800 Subject: [PATCH 2/8] example command --- docs/source/running-jobs/environment-variables.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index db9c46ac7ad..12c208f1c50 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -31,8 +31,8 @@ You can specify environment variables to be made available to a task in two ways .. code-block:: text + # sky launch example.yaml --env-file my_app.env # cat my_app.env - # --env-file my_app.env MYVAR=val WANDB_API_KEY=MY_WANDB_API_KEY HF_TOKEN=MY_HF_TOKEN From f237f294c6a64e1e256b47713f450313589b0606 Mon Sep 17 00:00:00 2001 From: zpoint Date: Wed, 13 Nov 2024 23:26:22 +0800 Subject: [PATCH 3/8] PR comment commit Co-authored-by: Zongheng Yang --- docs/source/running-jobs/environment-variables.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index 12c208f1c50..2c3c3def58c 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -27,7 +27,7 @@ You can specify environment variables to be made available to a task in two ways - ``--env`` flag in ``sky launch/exec`` :ref:`CLI ` (takes precedence over the above) -- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI ` (lowest precedence over the above) +- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, path to a dotenv file (lowest precedence over the above) .. code-block:: text From 1f9f7dc38148c9a3a9fba0069a14f5f6ffe8a5c7 Mon Sep 17 00:00:00 2001 From: zpoint Date: Wed, 13 Nov 2024 23:33:45 +0800 Subject: [PATCH 4/8] resolve PR comment --- .../running-jobs/environment-variables.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index 2c3c3def58c..6b60c0dfc50 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -16,18 +16,9 @@ User-specified environment variables User-specified environment variables are useful for passing secrets and any arguments or configurations needed for your tasks. They are made available in ``file_mounts``, ``setup``, and ``run``. -You can specify environment variables to be made available to a task in two ways: +You can specify environment variables to be made available to a task in several ways: -- ``envs`` field (dict) in a :ref:`task YAML `: - - .. code-block:: yaml - - envs: - MYVAR: val - -- ``--env`` flag in ``sky launch/exec`` :ref:`CLI ` (takes precedence over the above) - -- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, path to a dotenv file (lowest precedence over the above) +- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, path to a dotenv file: .. code-block:: text @@ -37,6 +28,15 @@ You can specify environment variables to be made available to a task in two ways WANDB_API_KEY=MY_WANDB_API_KEY HF_TOKEN=MY_HF_TOKEN +- ``envs`` field (dict) in a :ref:`task YAML `: (takes precedence over the above) + + .. code-block:: yaml + + envs: + MYVAR: val + +- ``--env`` flag in ``sky launch/exec`` :ref:`CLI ` (takes precedence over the above) + .. tip:: To mark an environment variable as required and make SkyPilot forcefully check From 85d7a138888eeac5926a809defbeb1dc4056126f Mon Sep 17 00:00:00 2001 From: zpoint Date: Wed, 13 Nov 2024 23:37:43 +0800 Subject: [PATCH 5/8] dotenv to code format --- docs/source/running-jobs/environment-variables.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index 6b60c0dfc50..db4c56e79cc 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -18,7 +18,7 @@ User-specified environment variables are useful for passing secrets and any argu You can specify environment variables to be made available to a task in several ways: -- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, path to a dotenv file: +- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, path to a `dotenv` file: .. code-block:: text From ef24f3fdec0f668805dfe211c65069270d3fe2e8 Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 14 Nov 2024 10:16:24 +0800 Subject: [PATCH 6/8] Update docs/source/running-jobs/environment-variables.rst Co-authored-by: Zongheng Yang --- docs/source/running-jobs/environment-variables.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index db4c56e79cc..9741faa93f4 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -18,7 +18,7 @@ User-specified environment variables are useful for passing secrets and any argu You can specify environment variables to be made available to a task in several ways: -- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, path to a `dotenv` file: +- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, which is a path to a `dotenv` file: .. code-block:: text From 88476f80dab407b2fa1d0168302a1c633c9d519d Mon Sep 17 00:00:00 2001 From: zpoint Date: Thu, 14 Nov 2024 10:16:29 +0800 Subject: [PATCH 7/8] Update docs/source/running-jobs/environment-variables.rst Co-authored-by: Zongheng Yang --- docs/source/running-jobs/environment-variables.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index 9741faa93f4..d4e3c172485 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -28,7 +28,7 @@ You can specify environment variables to be made available to a task in several WANDB_API_KEY=MY_WANDB_API_KEY HF_TOKEN=MY_HF_TOKEN -- ``envs`` field (dict) in a :ref:`task YAML `: (takes precedence over the above) +- ``envs`` field (dict) in a :ref:`task YAML ` (takes precedence over the above): .. code-block:: yaml From 5288de037d6b4859935898d79163751775a49b35 Mon Sep 17 00:00:00 2001 From: zepingguo Date: Thu, 14 Nov 2024 12:08:16 +0800 Subject: [PATCH 8/8] change the order --- .../running-jobs/environment-variables.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst index d4e3c172485..d88424359d1 100644 --- a/docs/source/running-jobs/environment-variables.rst +++ b/docs/source/running-jobs/environment-variables.rst @@ -18,7 +18,15 @@ User-specified environment variables are useful for passing secrets and any argu You can specify environment variables to be made available to a task in several ways: -- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, which is a path to a `dotenv` file: +- ``envs`` field (dict) in a :ref:`task YAML `: + + .. code-block:: yaml + + envs: + MYVAR: val + + +- ``--env-file`` flag in ``sky launch/exec`` :ref:`CLI `, which is a path to a `dotenv` file (takes precedence over the above): .. code-block:: text @@ -28,13 +36,6 @@ You can specify environment variables to be made available to a task in several WANDB_API_KEY=MY_WANDB_API_KEY HF_TOKEN=MY_HF_TOKEN -- ``envs`` field (dict) in a :ref:`task YAML ` (takes precedence over the above): - - .. code-block:: yaml - - envs: - MYVAR: val - - ``--env`` flag in ``sky launch/exec`` :ref:`CLI ` (takes precedence over the above) .. tip::