When you start Spark on YARN using Spark shell as
spark/bin/spark-shell --master yarn-client
You can get the following errors on Hortonworks:
... at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala):10: error: not found: value sqlContext import sqlContext.implicits._ :10: error: not found: value sqlContext import sqlContext.sql
Additionally when you open the Application Master log you can see:
Log Type: stderr Log Upload Time: Tue Nov 17 06:59:35 -0800 2015 Log Length: 87 Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
To solve this issue, edit spark-defaults.conf
and specify:
spark.driver.extraJavaOptions -Dhdp.version=current spark.yarn.am.extraJavaOptions -Dhdp.version=current
In my case this helped launch the Spark shell successfully, and I could see the command prompt:
15/11/17 07:37:05 INFO repl.SparkILoop: Created sql context (with Hive support).. SQL context available as sqlContext. scala>
I used Spark 1.5.2 and HDP 2.2.4.8