Error connecting to databricks in python with databricks-connect

Question

I'm using databricks-connect on mac using pycharm but after I finished the configuration and tried to run databricks-connect test, I got the following error and have no idea what the problem is. I followed this documentation: https://docs.databricks.com/user-guide/dev-tools/db-connect.html

The error message is as below:

scala> spa
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/databricks-connect", line 11, in 
    load_entry_point('databricks-connect==5.3.1', 'console_scripts', 'databricks-connect')()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark/databricks_connect.py", line 244, in main
    test()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark/databricks_connect.py", line 213, in test
    raise ValueError("Scala command failed to produce correct result")
ValueError: Scala command failed to produce correct result

stang · Accepted Answer · 2019-10-11 11:56:29Z

2

Maybe your Java/Python version does not comply. Check your cluster, which Python version does it use (in my case it was 3.5).

And what's most important: check which JDK version do you have on your computer. In my case, I've had the latest one, which was not supported by databricks-connect. It needs to run on JDK 8.

answered Oct 11, 2019 at 11:56

stang

563 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Molly · Accepted Answer · 2019-11-25 21:51:22Z

1

I would make sure you are using the right version of the Databricks Runtime (DB Connect only currently supports 5.1-5.5). Since these is a limit on the DBR that works with DB connect, you'll have to make sure you match the python version as well (for the base Databricks runtime, I believe it is 3.5.x).

answered Nov 25, 2019 at 21:51

Molly

494 bronze badges

Comments

chá de boldo · Accepted Answer · 2022-03-25 15:51:21Z

0

To ignore the RUNTIME version, export an environment variable that resolves:

export DEBUG_IGNORE_VERSION_MISMATCH=1

answered Mar 25, 2022 at 15:51

chá de boldo

563 bronze badges

Collectives™ on Stack Overflow

Error connecting to databricks in python with databricks-connect

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related