[SPARK-33565][PYTHON][BUILD] Configure Python3.9 in python run-tests.py

### What changes were proposed in this pull request?
- Remove temporary workaround in run-tests
- Move the workaround to python/run-tests.py

### Why are the changes needed?

As the note mentioned:
> Python 3.9 was temporarily removed as its default Python executables to test because of Jenkins environment issue. Once Jenkins has Python 3.9 to test, we should remove this change back and add python3.9 into python/run-tests.py script.

This cleanup will also help to user only run python/run-tests.py with `python3.9` and also `pypy3` as default.

### Does this PR introduce _any_ user-facing change?
It marks `python3.9` and `pypy3` as default test runner python executors.

### How was this patch tested?
CI passed, and PySpark test are still running under python3.9 and pypy3.

Closes apache#35087 from Yikun/SPARK-33565.

Authored-by: Yikun Jiang <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>

dev/run-tests.py

@@ -490,12 +490,6 @@ def run_python_tests(test_modules, parallelism, with_coverage=False):
     if test_modules != [modules.root]:
         command.append("--modules=%s" % ','.join(m.name for m in test_modules))
     command.append("--parallelism=%i" % parallelism)
-    if "GITHUB_ACTIONS" in os.environ:
-        # See SPARK-33565. Python 3.9 was temporarily removed as its default Python executables
-        # to test because of Jenkins environment issue. Once Jenkins has Python 3.9 to test,
-        # we should remove this change back and add python3.9 into python/run-tests.py script.
-        command.append("--python-executable=%s" % ','.join(
-            x for x in ["python3.9", "pypy3"] if which(x)))
     run_cmd(command)
 
 

python/run-tests.py

@@ -173,9 +173,9 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
 
 
 def get_default_python_executables():
-    python_execs = [x for x in ["python3.6", "pypy3"] if which(x)]
+    python_execs = [x for x in ["python3.9", "pypy3"] if which(x)]
 
-    if "python3.6" not in python_execs:
+    if "python3.9" not in python_execs:
         p = which("python3")
         if not p:
             LOGGER.error("No python3 executable found.  Exiting!")