https://kedro.org/ logo
#questions
Title
# questions
a

Ankit Kansal

09/07/2023, 12:39 PM
Hi Team - need help!! Only has data-bricks and we are not allowed to install IDEs and local python Now we have created a repo name Main Under this repo i would like to create a kedro project through data-bricks itself I have installed kedro==0.18.13 but unable to execute
kedro new -s="pyspark"
Is there way i can create a project under my repository ?
y

Yetunde

09/07/2023, 12:52 PM
Hey Ankit, let's see if we can try help you here. So I will assume that you have Kedro installed on the cluster and that the primary issue is that you don't have access to a terminal to run
kedro new
because you can't use an IDE and have to work in the Databricks workspace and therefore a notebook? Is this correct?
a

Ankit Kansal

09/07/2023, 12:56 PM
yes we only have access to notebook! And kedro new is not working, tried specifying starter kit as well but not working....
y

Yetunde

09/07/2023, 12:58 PM
What have you tried? I have a few more questions: ā€¢ Do you access to Web Terminal in Databricks? ā€¢ Or can you use shell magic in the notebook i.e. run
%sh kedro new -s="pyspark"
?
a

Ankit Kansal

09/07/2023, 1:02 PM
i tried running the second command but databricks can not directly connect to github to pull the starter kit. no web terminal in databricks...
y

Yetunde

09/07/2023, 1:06 PM
What happens when you do
%sh kedro new
, without the starter?
a

Ankit Kansal

09/07/2023, 1:13 PM
it issues error that name has not been provided and as i am using cell to execute sh commands, it does not works.. Is there a config which can be passed along with kedro new ?
y

Yetunde

09/07/2023, 1:19 PM
Yes, you can do that and the instructions are here. This will give you a blank template without the additional files required for PySpark. You won't be able to grab the PySpark starter because you can't connect to GitHub but you can copy or create the files specific to the PySpark starter - see here. I think you need to add a
spark.yml
in
conf/base
, edit
settings.py
and edit
requirements.txt
.
Let me loop in help for this šŸ˜„
n

Nok Lam Chan

09/07/2023, 1:42 PM
Hey, let see if I can helpšŸ‘€