spark 心跳超时分析 Cannot receive any reply in 120 seconds

本文由 简悦 SimpRead 转码， 原文地址 blog.csdn.net

org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply in 120 seconds. This timeout is controlled by spark.rpc.askTimeout
       at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
       at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
       at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
       at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
       at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
       at scala.util.Try$.apply(Try.scala:161)
       at scala.util.Failure.recover(Try.scala:185)
       at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
       at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
       at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
       at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
       at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
       at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
       at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
       at scala.concurrent.Promise$class.complete(Promise.scala:55)
       at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
       at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
       at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
       at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
       at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:643)
       at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:658)
       at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
       at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
       at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
       at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:634)
       at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
       at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:685)
       at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
       at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:2

spark 任务执行的时候发声心跳超时有两种情况，

• 1.executor 所在的节点宕掉了

• 2.运行在 executor 中的任务占用较大内存，导致 executor 长时间 GC，心跳线程无法运行，从而引起心跳超时

解决方案：去掉第一种情况，引发这个问题是由于数据发生了倾斜，导致 STW，可以调整 GC 的时间使其变短，这样就不会出现长时间的 STW（-XX:-UseConcMarkSweepGC  默认不启用  启用 CMS 低停顿垃圾收集器。）