Python REPL 无法在 Docker 中启动Python REPL fails to start in Docker


当使用包含预生成 Python 库的 Docker 容器时,Python 命令会失败,并且不会创建虚拟环境。When you use a Docker container that includes prebuilt Python libraries, Python commands fail and the virtual environment is not created. 以下错误消息在驱动程序日志中可见。The following error message is visible in the driver logs.

20/02/29 16:38:35 WARN PythonDriverWrapper: Failed to start repl ReplId-5b591-0ce42-78ef3-7 Cannot run program "/local_disk0/pythonVirtualEnvDirs/virtualEnv-56a5be60-3e71-486f-ac04-08e8f2491032/bin/python" (in directory "."): error=2, No such file or directory
        at java.lang.ProcessBuilder.start(
        at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1367)
        at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1393)
        at org.apache.spark.util.Utils$.executePythonAndGetOutput(Utils.scala:
Caused by: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(
        at java.lang.ProcessImpl.start(
        at java.lang.ProcessBuilder.start(
        ... 17 more

可以通过在笔记本中运行以下命令来确认问题:You can confirm the issue by running the following command in a notebook:

%sh virtualenv --no-site-packages

结果是一条错误消息,类似于以下内容:The result is an error message similar to the following:

usage: virtualenv [--version] [--with-traceback] [-v | -q] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_separated_list] [--clear]
                  [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel]
                  [--clear-app-data] [--symlink-app-data] [--prompt prompt] [-h]
virtualenv: error: the following arguments are required: dest

virtualenv 命令无法识别 --no-site-packages 选项。The virtualenv command does not recognize the --no-site-packages option.


此问题会影响所有当前的 Databricks Runtime 版本,但包含 Conda 的 Databricks Runtime 版本除外。The problem affects all current Databricks Runtime versions, except for Databricks Runtime versions that include Conda. 它会影响 virtualenv 库 20.0.0 及更高版本。It affects virtualenv library version 20.0.0 and above.


此问题是由于在 Docker 容器中使用不支持 --no-site-packages 选项的 Python virtualenv 库版本导致的。This issue is caused by using a Python virtualenv library version in the Docker container that does not support the --no-site-packages option.

Databricks Runtime 需要支持 --no-site-packages 选项的 virtualenv 库。Databricks Runtime requires a virtualenv library that supports the --no-site-packages option. virtualenv 库 20.0.0 及更高版本中已删除此选项。This option was removed in virtualenv library version 20.0.0 and above.

可以通过在笔记本中运行以下命令来验证 virtualenv 库版本:You can verify your virtualenv library version by running the following command in a notebook:

%sh virtualenv --version


可以通过在安装 virtualenv 库时指定兼容的版本来解决此问题。You can resolve the issue by specifying a compatible version when you install the virtualenv library.

例如,在 Dockerfile 中设置 virtualenv==16.0.0 会安装 virtualenv 库版本 16.0.0。For example, setting virtualenv==16.0.0 in the Dockerfile installs virtualenv library version 16.0.0. 此版本的库支持所需的选项。This version of the library supports the required option.