Configuring databricks-connect using python OS module

Question

I want to configure databricks-connect configure through python OS module after installing databricks-connect through os.system("pip install databricks-connect==6.5")

Once databricks-connect is successfully install we need to configure it by passing the following values:

host= "https://<location>.azuredatabricks.net",
port= "8787",
token = "<Token>",
cluster_id = "<ClusterId>",
org_id = "<OrgId>"

In the terminal if we type databricks-connect configure , it will start asking you above parameter one by one as shown in the figure:

Now I want same thing to be run using python os.system

os.system("pip install databricks-connect")
os.system("databricks-connect configure")

After this how to pass host, port, token etc.?
After every value we have to press enter as well.

when I run this on terminal this work fine ,

echo -e 'https://adb-661130381327abc.11.azuredatabricks.net\nxxxxx\n0529-yyyy-twins608\n6611303813275431\n15001' | databricks-connect configure

but giving me error when i try to run this python os.module

os.sytem("echo -e 'https://adb-661130381327abc.11.azuredatabricks.net\nxxxxx\n0529-yyyy-twins608\n6611303813275431\n15001' | databricks-connect configure")

Error "New host value must start with https://, e.g., https://demo.cloud.databricks.com")

unix.stackexchange.com/questions/227337/… This may be of some use here. — ARUN NADAR
– ARUN NADAR, Commented Jul 25, 2020 at 16:33

Anmol Singh Jaggi · Accepted Answer · 2020-07-25 13:19:20Z

2

You can just pipe the data as stdin to the program.

import os


host= "https://<location>.azuredatabricks.net"
port= "8787"
token = "<Token>"
cluster_id = "<ClusterId>"
org_id = "<OrgId>"

stdin_list = [host, port, token, cluster_id, org_id]
stdin_string = '\n'.join(stdin_list)
command = "echo '{}' | {}".format(stdin_string, "databricks-connect configure")
os.system(command)

edited Jul 25, 2020 at 13:19

answered Jul 25, 2020 at 13:03

Anmol Singh Jaggi

8,6964 gold badges40 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

qaiser Over a year ago

getting following error The system cannot find the file specified.

Anmol Singh Jaggi Over a year ago

Just double-check if the command string is correct and if the order of arguments is ok. Try again. I had mistakenly malformed the command string.

qaiser Over a year ago

command = "echo adb-8538834050545100.10.azuredatabricks.net | databricks-connect configure" , when i am running this host is getting updated , but how to pass multiple value.., i did tried above code but not working

qaiser Over a year ago

for only first value i.e. host it is getting updated

Anmol Singh Jaggi Over a year ago

; wont work. Try echo 'adb-8538834050545100.10.azuredatabricks.net\n8787\n<Token>\n<Cluster>\n<Org>' | databricks-connect configure

|

chilin · Accepted Answer · 2022-09-05 15:02:24Z

1

A small modification to @Anmol

import subprocess

host= "https://<location>.azuredatabricks.net"
port= "8787"
token = "<Token>"
cluster_id = "<ClusterId>"
org_id = "<OrgId>"

stdin_list = [host, port, token, cluster_id, org_id]
stdin_string = '\n'.join(stdin_list)
echo = subprocess.Popen((['echo', '-e', stdin_string]), stdout=subprocess.PIPE)
# fix typo from std_out to stdout
output = subprocess.check_output(('databricks-connect', 'configure'), stdin=echo.stdout)
echo.wait()
print(output.decode())

echo -e

takes care of the enter

edited Sep 5, 2022 at 15:02

chilin

4247 silver badges9 bronze badges

answered Jul 25, 2020 at 18:08

ARUN NADAR

1057 bronze badges

1 Comment

chilin Over a year ago

Work for me, and there is typo in echo = subprocess.Popen((['echo', '-e', stdin_string]), std_out=subprocess.PIPE), should be stdout. I edited for you ^^

Collectives™ on Stack Overflow

Configuring databricks-connect using python OS module

2 Answers 2

7 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related