Spark on YARN Submit Errors on Hortonworks

When you start Spark on YARN using Spark shell as

spark/bin/spark-shell --master yarn-client

You can get the following errors on Hortonworks:

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

:10: error: not found: value sqlContext
      import sqlContext.implicits._
:10: error: not found: value sqlContext
       import sqlContext.sql

Additionally when you open the Application Master log you can see:

Log Type: stderr
Log Upload Time: Tue Nov 17 06:59:35 -0800 2015
Log Length: 87
Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher

To solve this issue, edit spark-defaults.conf and specify:

spark.driver.extraJavaOptions -Dhdp.version=current -Dhdp.version=current

In my case this helped launch the Spark shell successfully, and I could see the command prompt:

15/11/17 07:37:05 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.


I used Spark 1.5.2 and HDP