0

IMO the dashboard does not match the log error, or?

Spark master dashboard says (similar for memory as well):

Alive Workers: 2 Cores in use: 2 Total, 0 Used

Log:

17/07/21 20:48:45 WARN TaskSchedulerImpl: Initial job has not accepted any 
resources; 
check your cluster UI to ensure that workers are registered 
and have sufficient resources

Assuming there was e.g. not enough memory, how to determine that? Maybe a symptom, the workers get here and now "dissociated" but then reconnect.

0

TLDR; set workers' memory to 4G.

I reduced the test to the build-in Pi calculation:

./bin/run-example SparkPi 10

And got the same result; apparently 1G memory is insufficient even for simplest experiments. Increasing both master and workers memory to 4G and the problematic behaviour was not reproducible. However before the systems start computation the message still could appear but just once, seems it is printed in a waiting loop.

  • Does this answer the question or should this be part of the question? – 030 Jul 23 '17 at 12:41
  • it's the answer – Peter Muryshkin Jul 23 '17 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.