최근 포토로그


dirver / executor memory configuration (spark 1.5 and before vs spark 1.6.+ and 2.x) Spark

1. check to runtime overhead

because of the runtime overhead imposed by Scala, which is usually around 3-7%, more or less.

hduser@localhost:~$ scala -J-Xmx4g
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_73).
Type in expressions for evaluation. Or try :help.

scala> Runtime.getRuntime.maxMemory
res0: Long = 3817865216

scala> hduser@localhost:~$ scala -J-Xmx8g
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_73).
Type in expressions for evaluation. Or try :help.

scala> Runtime.getRuntime.maxMemory
res0: Long = 7635730432

* real runtime memory
4G - 3817865216 / 8G - 7635730432



2. spark memory(storage / execution) equal to scala runtime overhead memory

hduser@localhost:~$ spark-shell --conf "spark.ui.port=4041" --driver-memory 4G
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
17/01/04 12:14:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/04 12:14:47 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.
Spark context Web UI available at http://x.x.x.x:4041
Spark context available as 'sc' (master = local[*], app id = local-1483499685530).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.0.2
      /_/
        
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_73)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import java.lang.management.ManagementFactory
import java.lang.management.ManagementFactory

scala> ManagementFactory.getRuntimeMXBean.getInputArguments
res0: java.util.List[String] = [-Dscala.usejavacp=true, -Xmx4G]

scala>  Runtime.getRuntime.maxMemory
res1: Long = 3817865216


3. Spark Web UI
















* Spark 1.6.+ and 2.x (
Why 2004.6MB?)

step 1. driver-memory -> 4G
step 2. java heap size (runtime max memory, except runtime overhead) -> 3817865216 (3641mb)
step 3. java heap - reserved memory(300mb) -> 3641mb - 300mb = 3341mb
* reserved memory (ref . spark 1.6.+ unified memory management, bottom picture) 
step 4. spark memory (spark.memory.fraction) -> 3341mb * 0.6 = 2004.6(mb)
spark 1.6.x : spark.memory.fraction (defalut : 0.75)
spark 2.+ : spark.memory.fraction (defalut : 0.6)

Spark Memory Management 1.6.0+
[spark 1.6.+ and spark 2.x]


* Spark 1.5 and before (Why 1966.14MB?)

step 1. driver-memory -> 4G
step 2. java heap size (runtime max memory, except runtime overhead) -> 3817865216 (3641mb)
step 3. java heap * (spark.storage.memoryFraction * spark.storage.safetyFraction) -> 3641mb * 0.6 * 0.9 = 1966.14mb
spark 1.5.0 : spark.storage.memoryFraction (defalut : 0.6)
spark 1.5.0 : spark.storage.safetyFraction (defalut : 0.9)


Spark Heap Usage

[spark 1.5 and before]



덧글

댓글 입력 영역