Spark Error: YarnAllocator: exit status: -100. Diagnostics: Container released on * lost * node

I am working on AWS-EMR 4.1, Spark 1.5 with the following conf:

spark-submit --deploy-mode cluster --master yarn-cluster --driver-memory 200g --driver-cores 30 --executor-memory 70g --executor-cores 8 --num-executors 90 --conf spark.storage.memoryFraction=0.45 --conf spark.shuffle.memoryFraction=0.75 --conf spark.task.maxFailures=1 --conf spark.network.timeout=1800s 

Then I got the error below. Where can I find out what "Exit Status: -100" is ? And how can I solve this problem? Thanks!


 15/12/05 05:54:24 INFO TaskSetManager: Finished task 176.0 in stage 957.0 (TID 128408) in 130885 ms on ip-10-155-195-239.ec2.internal (106/800) 15/12/05 05:54:24 INFO YarnAllocator: Completed container container_1449241952863_0004_01_000026 (state: COMPLETE, exit status: -100) 15/12/05 05:54:24 INFO YarnAllocator: Container marked as failed: container_1449241952863_0004_01_000026. Exit status: -100. Diagnostics: Container released on a *lost* node 15/12/05 05:54:24 INFO YarnAllocator: Completed container container_1449241952863_0004_01_000055 (state: COMPLETE, exit status: -100) 15/12/05 05:54:24 INFO YarnAllocator: Container marked as failed: container_1449241952863_0004_01_000055. Exit status: -100. Diagnostics: Container released on a *lost* node 15/12/05 05:54:24 ERROR YarnClusterScheduler: Lost executor 24 on ip-10-147-11-212.ec2.internal: Yarn deallocated the executor 24 (container container_1449241952863_0004_01_000026) 15/12/05 05:54:24 INFO TaskSetManager: Re-queueing tasks for 24 from TaskSet 957.0 15/12/05 05:54:24 WARN TaskSetManager: Lost task 382.0 in stage 957.0 (TID 128614, ip-10-147-11-212.ec2.internal): ExecutorLostFailure (executor 24 lost) 15/12/05 05:54:24 ERROR TaskSetManager: Task 382 in stage 957.0 failed 1 times; aborting job 15/12/05 05:54:24 WARN TaskSetManager: Lost task 208.0 in stage 957.0 (TID 128440, ip-10-147-11-212.ec2.internal): ExecutorLostFailure (executor 24 lost) 
+5
amazon-web-services emr yarn apache-spark
source share

No one has answered this question yet.

See similar questions:

14
The spark mode on the yarn ends with "Output state: -100. Diagnostics: Container released on * lost * node"

or similar:

14
The spark mode on the yarn ends with "Output state: -100. Diagnostics: Container released on * lost * node"
4
Spark 2.0 Status: COMPLETE Yarn -100 Status Code for Yarn
one
Spark with a yarn client on multi-knot HDP nodes only launches performers on a single node
one
Pyspark udf memory usage
0
PySpark on AWS Glue: container exit code 52 when writing output to csv
0
The final status of Iskra failed in thread-cluster mode if there is no spark session in the class
0
spark python: Logistic regression on PipelineModel
0
Is it possible to get Spark to run the entire Taskset on one artist?
0
Spark application exits with โ€œERROR root: EAP # 5: application configuration file missingโ€ before initializing the spark context
0
spark tasks do not work with error showing exit status: -100

All Articles