I recently came across the need to run Scala programs in a notebook, for this Azure notebook is readily available however it is a costly solution for individuals who want to play around. The Jupyter notebook is one of the most used tools in data science projects. It’s a great tool for developing software in python and has great support for that. It can also be used for scala development with the spylon-kernel.
Writing this blog for all the individuals who need to run the Scala programs on the Jupyter notebook.
There is a utility called spylon kernel which helps Scala to run on Jupyter.
Automation testing company on Afourtech
Prerequisite:
Software –
- Spark (http://spark.apache.org/downloads.html)
- Hadoop (http://media.sundog-soft.com/Udemy/winutils.exe)
- JDK
Once you have downloaded all the Software listed above you would need to make certain modifications, and they are listed below:
Spark:
- Create a Folder in Spark on a C drive and copy all the content from .tar to a newly created folder.
- Rename log4j.properties. template to log4j.properties
- Edit the same file and replace rootCategory=INFO, console tolog4j.rootCategory=ERROR, console and then save and close the file.
Software development company in the USA by Afourtech.
Hadoop:
We are doing the below steps to execute spark programs on our local machine.
- After you’ve successfully downloaded winutils create a folder on c drive winutils\bin and tmp\hive.
- Paste wintils.exe in the bin folder.
- open a command prompt and run the following commands.
– cd c:\winutils\bin
– winutils.exe chmod 777 \tmp\hive
- It should run successfully.
Environment variables:
- SPARK_HOME: eg: “C:\Spark”
- HADOOP_HOME: “C:\winutils”
- JAVA_HOME
(all paths should contain a path to the folder not to the bin)
After you have set the env variables, run and check if the spark is running.
– cd c:\spark
– pyspark
If everything is OK, you should see an output like the image below.
For Jupyter scala, open Anaconda prompt and run the following commands.
pip install spylon-kernel python -m spylon_kernel install jupyter notebook
Once the installation is complete you can see the spylon-kernel in a New file dropdown.
If everything goes well the scala snippets should run like Usain Bolt (Pun Intended). If in case, it does not run, you would need to perform some additional steps and they are as follows.
You need to copy the content from the following zip file
“C:\Spark\python\lib\py4j-0.10.8.1-src.zip”
“C:\Spark\python\lib\pyspark.zip”
TO
\anaconda\Lib\site-packages
That’s It Enjoy SCALA with Jupyter.
cyber security testing services & infrastructure management services in the USA on Afourtech.