[Spark] SparkConf, spark-shell, spark-submit, spark-defaults.conf 적용 예

Spark Property

• Spark는 SparkConf, spark-shell, spark-submit, spark-defaults.conf 을 통해 Spark Application 실행과 관련된 자원을 설정할 수 있습니다.

  • property 적용 순서는 SparkConf, spark-shell, spark-submit, spark-defaults.conf 입니다.

• 적용 예

  • SparkConf

      val conf = new SparkConf().setAppName("Histogram").setMaster("local")
      val sc = new SparkContext(conf)

  • spark-shell

      spark-shell --master yarn --num-executors 8 --executor-cores 2 --executor-memory 2g --driver-memory 10g

  • spark-submit

      spark-submit \
      --class <main-class> \
      --master <master-url> \
      --deploy-mode <deploy-mode> \
      --conf <key>=<value> \
      ...
      <application-jar>
      [application-arguments]

  • spark-defaults.conf(설정 정보가 담긴 파일)

    • /$SPARK_HOME/conf/spark-defaults.conf

        ...
        spark.master spark:
        spark.eventLog.enabled true
        spark.driver.memory 5g
        spark.eventLog.enabled true
        ...