[FLINK-29421][python] Support python 3.10

This closes apache#21699.

NOTICE

@@ -17,7 +17,7 @@ See bundled license files for details.
 This project bundles the following dependencies under the BSD license.
 See bundled license files for details.
 
-- cloudpickle:2.1.0
+- cloudpickle:2.2.0
 - net.sf.py4j:py4j:0.10.9.7
 
 This project bundles the following dependencies under SIL OFL 1.1 license (https://opensource.org/licenses/OFL-1.1).

docs/content.zh/docs/deployment/cli.md

@@ -367,11 +367,11 @@ Currently, users are able to submit a PyFlink job via the CLI. It does not requi
 JAR file path or the entry main class, which is different from the Java job submission.
 
 {{< hint info >}}
-When submitting Python job via `flink run`, Flink will run the command "python". Please run the following command to confirm that the python executable in current environment points to a supported Python version of 3.6+.
+When submitting Python job via `flink run`, Flink will run the command "python". Please run the following command to confirm that the python executable in current environment points to a supported Python version of 3.7+.
 {{< /hint >}}
 ```bash
 $ python --version
-# the version printed here must be 3.6+
+# the version printed here must be 3.7+
 ```
 
 The following commands show different PyFlink job submission use-cases:
@@ -520,7 +520,7 @@ related options. Here's an overview of all the Python related options for the ac
             <td>
                 Specify the path of the python interpreter used to execute the python UDF worker
                 (e.g.: --pyExecutable /usr/local/bin/python3).
-                The python UDF worker depends on Python 3.6+, Apache Beam (version == 2.38.0),
+                The python UDF worker depends on Python 3.7+, Apache Beam (version == 2.43.0),
                 Pip (version >= 20.3) and SetupTools (version >= 37.0.0).
                 Please ensure that the specified environment meets the above requirements.
             </td>

docs/content.zh/docs/deployment/resource-providers/standalone/docker.md

@@ -428,23 +428,6 @@ FROM flink:{{< version >}}
 FROM flink:latest
 {{< /unstable >}}
 
-# install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7 from source
-# it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink officially.
-
-RUN apt-get update -y && \
-apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libffi-dev liblzma-dev && \
-wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz && \
-tar -xvf Python-3.7.9.tgz && \
-cd Python-3.7.9 && \
-./configure --without-tests --enable-shared && \
-make -j6 && \
-make install && \
-ldconfig /usr/local/lib && \
-cd .. && rm -f Python-3.7.9.tgz && rm -rf Python-3.7.9 && \
-ln -s /usr/local/bin/python3 /usr/local/bin/python && \
-apt-get clean && \
-rm -rf /var/lib/apt/lists/*
-
 # install PyFlink
 {{< stable >}}
 RUN pip3 install apache-flink=={{< version >}}

docs/content.zh/docs/dev/python/datastream_tutorial.md

@@ -48,7 +48,7 @@ Apache Flink 提供了 DataStream API，用于构建健壮的、有状态的流
 首先，你需要在你的电脑上准备以下环境：
 
 * Java 11
-* Python 3.6, 3.7, 3.8 or 3.9
+* Python 3.7, 3.8, 3.9 or 3.10
 
 使用 Python DataStream API 需要安装 PyFlink，PyFlink 发布在 [PyPI](https://pypi.org/project/apache-flink/)上，可以通过 `pip` 快速安装。 
 

docs/content.zh/docs/dev/python/installation.md

@@ -29,11 +29,11 @@ under the License.
 
 
 ## 环境要求
-<span class="label label-info">注意</span> PyFlink 需要 Python 3.6 以上版本（3.6, 3.7 或 3.8）。请运行以下命令，以确保 Python 版本满足要求。
+<span class="label label-info">注意</span> PyFlink 需要 Python 3.7 以上版本（3.7, 3.8, 3.9 或 3.10）。请运行以下命令，以确保 Python 版本满足要求。
 
 ```bash
 $ python --version
-# the version printed here must be 3.6, 3.7, 3.8 or 3.9
+# the version printed here must be 3.7, 3.8, 3.9 or 3.10
 ```
 
 ## 环境设置

docs/content.zh/docs/dev/python/table/udfs/python_udfs.md

@@ -28,7 +28,7 @@ under the License.
 
 用户自定义函数是重要的功能，因为它们极大地扩展了 Python Table API 程序的表达能力。
 
-**注意:** 要执行 Python 用户自定义函数，客户端和集群端都需要安装 Python 3.6 以上版本(3.6、3.7 或 3.8)，并安装 PyFlink。
+**注意:** 要执行 Python 用户自定义函数，客户端和集群端都需要安装 Python 3.7 以上版本(3.7、3.8、3.9 或 3.10)，并安装 PyFlink。
 
 <a name="scalar-functions"></a>
 

docs/content.zh/docs/dev/python/table/udfs/vectorized_python_udfs.md

@@ -34,7 +34,7 @@ under the License.
 向量化用户自定义函数的定义，与[非向量化用户自定义函数]({{< ref "docs/dev/python/table/udfs/python_udfs" >}})具有相似的方式，
 用户只需要在调用 `udf` 或者 `udaf` 装饰器时添加一个额外的参数 `func_type="pandas"`，将其标记为一个向量化用户自定义函数即可。
 
-**注意:** 要执行 Python 向量化自定义函数，客户端和集群端都需要安装 Python 3.6 以上版本(3.6、3.7 或 3.8)，并安装 PyFlink。
+**注意:** 要执行 Python 向量化自定义函数，客户端和集群端都需要安装 Python 3.7 以上版本(3.7、3.8、3.9 或 3.10)，并安装 PyFlink。
 
 ## 向量化标量函数
 

docs/content.zh/docs/dev/python/table_api_tutorial.md

@@ -50,7 +50,7 @@ Apache Flink 提供 Table API 关系型 API 来统一处理流和批，即查询
 如果要继续我们的旅程，你需要一台具有以下功能的计算机：
 
 * Java 11
-* Python 3.6, 3.7, 3.8 or 3.9
+* Python 3.7, 3.8, 3.9 or 3.10
 
 使用 Python Table API 需要安装 PyFlink，它已经被发布到 [PyPi](https://pypi.org/project/apache-flink/)，你可以通过如下方式安装 PyFlink：
 

docs/content.zh/docs/dev/table/sqlClient.md

@@ -265,8 +265,8 @@ Mode "embedded" (default) submits Flink jobs from the local machine.
                                                python UDF worker (e.g.:
                                                --pyExecutable
                                                /usr/local/bin/python3). The python
-                                               UDF worker depends on Python 3.6+,
-                                               Apache Beam (version == 2.38.0), Pip
+                                               UDF worker depends on Python 3.7+,
+                                               Apache Beam (version == 2.43.0), Pip
                                                (version >= 20.3) and SetupTools
                                                (version >= 37.0.0). Please ensure
                                                that the specified environment meets

docs/content.zh/docs/flinkDev/building.md

@@ -75,11 +75,11 @@ mvn clean install -DskipTests -Dfast -Pskip-webui-build -T 1C
 
     如果想构建一个可用于 pip 安装的 PyFlink 包，需要先构建 Flink 工程，如 [构建 Flink](#build-flink) 中所述。
 
-2. Python 的版本为 3.6, 3.7 或者 3.8.
+2. Python 的版本为 3.7, 3.8, 3.9 或者 3.10.
 
     ```shell
     $ python --version
-    # the version printed here must be 3.6, 3.7, 3.8 or 3.9
+    # the version printed here must be 3.7, 3.8, 3.9 or 3.10
     ```
 
 3. 构建 PyFlink 的 Cython 扩展模块（可选的）

docs/content/docs/deployment/cli.md

@@ -365,11 +365,11 @@ Currently, users are able to submit a PyFlink job via the CLI. It does not requi
 JAR file path or the entry main class, which is different from the Java job submission.
 
 {{< hint info >}}
-When submitting Python job via `flink run`, Flink will run the command "python". Please run the following command to confirm that the python executable in current environment points to a supported Python version of 3.6+.
+When submitting Python job via `flink run`, Flink will run the command "python". Please run the following command to confirm that the python executable in current environment points to a supported Python version of 3.7+.
 {{< /hint >}}
 ```bash
 $ python --version
-# the version printed here must be 3.6+
+# the version printed here must be 3.7+
 ```
 
 The following commands show different PyFlink job submission use-cases:
@@ -518,7 +518,7 @@ related options. Here's an overview of all the Python related options for the ac
             <td>
                 Specify the path of the python interpreter used to execute the python UDF worker
                 (e.g.: --pyExecutable /usr/local/bin/python3).
-                The python UDF worker depends on Python 3.6+, Apache Beam (version == 2.38.0),
+                The python UDF worker depends on Python 3.7+, Apache Beam (version == 2.43.0),
                 Pip (version >= 20.3) and SetupTools (version >= 37.0.0).
                 Please ensure that the specified environment meets the above requirements.
             </td>

docs/content/docs/deployment/resource-providers/standalone/docker.md

@@ -428,23 +428,6 @@ FROM flink:{{< version >}}
 FROM flink:latest
 {{< /unstable >}}
 
-# install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7 from source
-# it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink officially.
-
-RUN apt-get update -y && \
-apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libffi-dev liblzma-dev && \
-wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz && \
-tar -xvf Python-3.7.9.tgz && \
-cd Python-3.7.9 && \
-./configure --without-tests --enable-shared && \
-make -j6 && \
-make install && \
-ldconfig /usr/local/lib && \
-cd .. && rm -f Python-3.7.9.tgz && rm -rf Python-3.7.9 && \
-ln -s /usr/local/bin/python3 /usr/local/bin/python && \
-apt-get clean && \
-rm -rf /var/lib/apt/lists/*
-
 # install PyFlink
 {{< stable >}}
 RUN pip3 install apache-flink=={{< version >}}

docs/content/docs/dev/python/datastream_tutorial.md

@@ -47,7 +47,7 @@ In particular, Apache Flink's [user mailing list](https://flink.apache.org/commu
 If you want to follow along, you will require a computer with: 
 
 * Java 11
-* Python 3.6, 3.7, 3.8 or 3.9
+* Python 3.7, 3.8, 3.9 or 3.10
 
 Using Python DataStream API requires installing PyFlink, which is available on [PyPI](https://pypi.org/project/apache-flink/) and can be easily installed using `pip`. 
 

docs/content/docs/dev/python/installation.md

@@ -29,12 +29,12 @@ under the License.
 ## Environment Requirements
 
 {{< hint info >}}
-Python version (3.6, 3.7, 3.8 or 3.9) is required for PyFlink. Please run the following command to make sure that it meets the requirements:
+Python version (3.7, 3.8, 3.9 or 3.10) is required for PyFlink. Please run the following command to make sure that it meets the requirements:
 {{< /hint >}}
 
 ```bash
 $ python --version
-# the version printed here must be 3.6, 3.7, 3.8 or 3.9
+# the version printed here must be 3.7, 3.8, 3.9 or 3.10
 ```
 
 ## Environment Setup

docs/content/docs/dev/python/table/udfs/python_udfs.md

@@ -28,7 +28,7 @@ under the License.
 
 User-defined functions are important features, because they significantly extend the expressiveness of Python Table API programs.
 
-**NOTE:** Python UDF execution requires Python version (3.6, 3.7, 3.8 or 3.9) with PyFlink installed. It's required on both the client side and the cluster side. 
+**NOTE:** Python UDF execution requires Python version (3.7, 3.8, 3.9 or 3.10) with PyFlink installed. It's required on both the client side and the cluster side. 
 
 ## Scalar Functions
 

docs/content/docs/dev/python/table/udfs/vectorized_python_udfs.md

@@ -33,7 +33,7 @@ These Python libraries are highly optimized and provide high-performance data st
 [non-vectorized user-defined functions]({{< ref "docs/dev/python/table/udfs/python_udfs" >}}) on how to define vectorized user-defined functions.
 Users only need to add an extra parameter `func_type="pandas"` in the decorator `udf` or `udaf` to mark it as a vectorized user-defined function.
 
-**NOTE:** Python UDF execution requires Python version (3.6, 3.7, 3.8 or 3.9) with PyFlink installed. It's required on both the client side and the cluster side. 
+**NOTE:** Python UDF execution requires Python version (3.7, 3.8, 3.9 or 3.10) with PyFlink installed. It's required on both the client side and the cluster side. 
 
 ## Vectorized Scalar Functions
 

docs/content/docs/dev/python/table_api_tutorial.md

@@ -51,7 +51,7 @@ In particular, Apache Flink's [user mailing list](https://flink.apache.org/commu
 If you want to follow along, you will require a computer with: 
 
 * Java 11
-* Python 3.6, 3.7, 3.8 or 3.9
+* Python 3.7, 3.8, 3.9 or 3.10
 
 Using Python Table API requires installing PyFlink, which is available on [PyPI](https://pypi.org/project/apache-flink/) and can be easily installed using `pip`. 
 

docs/content/docs/dev/table/sqlClient.md

@@ -207,8 +207,8 @@ Mode "embedded" (default) submits Flink jobs from the local machine.
                                                python UDF worker (e.g.:
                                                --pyExecutable
                                                /usr/local/bin/python3). The python
-                                               UDF worker depends on Python 3.6+,
-                                               Apache Beam (version == 2.38.0), Pip
+                                               UDF worker depends on Python 3.7+,
+                                               Apache Beam (version == 2.43.0), Pip
                                                (version >= 20.3) and SetupTools
                                                (version >= 37.0.0). Please ensure
                                                that the specified environment meets

docs/content/docs/flinkDev/building.md

@@ -74,11 +74,11 @@ The `fast` and `skip-webui-build` profiles have a significant impact on the buil
 
     If you want to build a PyFlink package that can be used for pip installation, you need to build the Flink project first, as described in [Build Flink](#build-flink).
 
-2. Python version(3.6, 3.7, 3.8 or 3.9) is required
+2. Python version(3.7, 3.8, 3.9 or 3.10) is required
 
     ```shell
     $ python --version
-    # the version printed here must be 3.6, 3.7, 3.8 or 3.9
+    # the version printed here must be 3.7, 3.8, 3.9 or 3.10
     ```
 
 3. Build PyFlink with Cython extension support (optional)

docs/layouts/shortcodes/generated/python_configuration.html

@@ -24,7 +24,7 @@
             <td><h5>python.executable</h5></td>
             <td style="word-wrap: break-word;">"python"</td>
             <td>String</td>
-            <td>Specify the path of the python interpreter used to execute the python UDF worker. The python UDF worker depends on Python 3.6+, Apache Beam (version == 2.38.0), Pip (version &gt;= 20.3) and SetupTools (version &gt;= 37.0.0). Please ensure that the specified environment meets the above requirements. The option is equivalent to the command line option "-pyexec".</td>
+            <td>Specify the path of the python interpreter used to execute the python UDF worker. The python UDF worker depends on Python 3.7+, Apache Beam (version == 2.43.0), Pip (version &gt;= 20.3) and SetupTools (version &gt;= 37.0.0). Please ensure that the specified environment meets the above requirements. The option is equivalent to the command line option "-pyexec".</td>
         </tr>
         <tr>
             <td><h5>python.execution-mode</h5></td>

flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java

@@ -265,7 +265,7 @@ public class CliFrontendParser {
                     true,
                     "Specify the path of the python interpreter used to execute the python UDF worker "
                             + "(e.g.: --pyExecutable /usr/local/bin/python3). "
-                            + "The python UDF worker depends on Python 3.6+, Apache Beam (version == 2.38.0), "
+                            + "The python UDF worker depends on Python 3.7+, Apache Beam (version == 2.43.0), "
                             + "Pip (version >= 20.3) and SetupTools (version >= 37.0.0). "
                             + "Please ensure that the specified environment meets the above requirements.");
 

flink-python/README.md

@@ -26,7 +26,7 @@ The auto-generated Python docs can be found at [https://nightlies.apache.org/fli
 
 ## Python Requirements
 
-Apache Flink Python API depends on Py4J (currently version 0.10.9.7), CloudPickle (currently version 2.1.0), python-dateutil(currently version 2.8.0), Apache Beam (currently version 2.38.0).
+Apache Flink Python API depends on Py4J (currently version 0.10.9.7), CloudPickle (currently version 2.2.0), python-dateutil(currently version 2.8.0), Apache Beam (currently version 2.43.0).
 
 ## Development Notices
 

flink-python/dev/build-wheels.sh

@@ -19,7 +19,7 @@ set -e -x
 dev/lint-python.sh -s py_env
 
 PY_ENV_DIR=`pwd`/dev/.conda/envs
-py_env=("3.6" "3.7" "3.8" "3.9")
+py_env=("3.7" "3.8" "3.9" "3.10")
 ## 2. install dependency
 for ((i=0;i<${#py_env[@]};i++)) do
     ${PY_ENV_DIR}/${py_env[i]}/bin/pip install -r dev/dev-requirements.txt

flink-python/dev/dev-requirements.txt

@@ -15,22 +15,19 @@
 pip>=20.3
 setuptools>=18.0
 wheel
-apache-beam==2.38.0
+apache-beam==2.43.0
 cython==0.29.24
 py4j==0.10.9.7
 python-dateutil==2.8.0
-cloudpickle==2.1.0
+cloudpickle==2.2.0
 avro-python3>=1.8.1,!=1.9.2,<1.10.0
-pandas>=1.3.0,<1.4.0; python_version >= '3.7'
-pandas>=1.0,<1.2.0; python_version < '3.7'
-pyarrow>=5.0.0,<9.0.0; python_version >= '3.7'
-pyarrow>=0.15.1,<7.0.0; python_version < '3.7'
+pandas>=1.3.0,<1.4.0
+pyarrow>=5.0.0,<9.0.0
 pytz>=2018.3
-numpy>=1.21.4,<1.22.0; python_version >= '3.7'
-numpy>=1.14.3,<1.20; python_version < '3.7'
+numpy>=1.21.4,<1.22.0
 fastavro>=1.1.0,<1.4.8
 grpcio>=1.29.0,<=1.46.3
 grpcio-tools>=1.29.0,<=1.46.3
-pemja==0.2.6; python_version >= '3.7' and platform_system != 'Windows'
+pemja==0.3.0; platform_system != 'Windows'
 httplib2>=0.19.0,<=0.20.4
 protobuf>=3.19.0,<=3.21

flink-python/dev/lint-python.sh

@@ -226,9 +226,9 @@ function install_miniconda() {
 # Install some kinds of py env.
 function install_py_env() {
     if [[ ${BUILD_REASON} = 'IndividualCI' ]]; then
-        py_env=("3.9")
+        py_env=("3.10")
     else
-        py_env=("3.6" "3.7" "3.8" "3.9")
+        py_env=("3.7" "3.8" "3.9" "3.10")
     fi
     for ((i=0;i<${#py_env[@]};i++)) do
         if [ -d "$CURRENT_DIR/.conda/envs/${py_env[i]}" ]; then
@@ -403,7 +403,7 @@ function install_environment() {
     fi
 
     # step-3 install python environment which includes
-    # 3.6 3.7 3.8 3.9
+    # 3.7 3.8 3.9 3.10
     if [ $STEP -lt 3 ] && [ `need_install_component "py_env"` = true ]; then
         print_function "STEP" "installing python environment..."
         install_py_env
@@ -584,7 +584,7 @@ function check_stage() {
 #########################
 # Tox check
 function tox_check() {
-    LATEST_PYTHON="py39"
+    LATEST_PYTHON="py310"
     print_function "STAGE" "tox checks"
     # Set created py-env in $PATH for tox's creating virtual env
     activate
@@ -789,7 +789,7 @@ usage: $0 [options]
 -l          list all checks supported.
 Examples:
   ./lint-python -s basic        =>  install environment with basic components.
-  ./lint-python -s py_env       =>  install environment with python env(3.6,3.7,3.8,3.9).
+  ./lint-python -s py_env       =>  install environment with python env(3.7,3.8,3.9,3.10).
   ./lint-python -s all          =>  install environment with all components such as python env,tox,flake8,sphinx,mypy etc.
   ./lint-python -s tox,flake8   =>  install environment with tox,flake8.
   ./lint-python -s tox -f       =>  reinstall environment with tox.

flink-python/pom.xml

@@ -116,7 +116,7 @@ under the License.
 		<dependency>
 			<groupId>com.alibaba</groupId>
 			<artifactId>pemja</artifactId>
-			<version>0.2.6</version>
+			<version>0.3.0</version>
 		</dependency>
 
 		<!-- Protobuf dependencies -->

flink-python/pyflink/datastream/formats/tests/test_csv.py

@@ -17,12 +17,9 @@
 ################################################################################
 import glob
 import os
-import sys
 import tempfile
 from typing import Tuple, List
 
-import pytest
-
 from pyflink.common import WatermarkStrategy, Types
 from pyflink.datastream import MapFunction
 from pyflink.datastream.connectors.file_system import FileSource, FileSink
@@ -114,7 +111,6 @@ class ProcessFileSourceCsvReaderFormatTests(FileSourceCsvReaderFormatTests,
     pass
 
 
-@pytest.mark.skipif(sys.version_info < (3, 7), reason="requires python3.7")
 class EmbeddedFileSourceCsvReaderFormatTests(FileSourceCsvReaderFormatTests,
                                              PyFlinkStreamingTestCase):
     def setUp(self):