Spark on a Docker swarm - workers won't accept jobs

Question

IMO the dashboard does not match the log error, or?

Spark master dashboard says (similar for memory as well):

Alive Workers: 2 Cores in use: 2 Total, 0 Used

Log:

17/07/21 20:48:45 WARN TaskSchedulerImpl: Initial job has not accepted any 
resources; 
check your cluster UI to ensure that workers are registered 
and have sufficient resources

Assuming there was e.g. not enough memory, how to determine that? Maybe a symptom, the workers get here and now "dissociated" but then reconnect.

Answer 1

TLDR; set workers' memory to 4G.

I reduced the test to the build-in Pi calculation:

./bin/run-example SparkPi 10

And got the same result; apparently 1G memory is insufficient even for simplest experiments. Increasing both master and workers memory to 4G and the problematic behaviour was not reproducible. However before the systems start computation the message still could appear but just once, seems it is printed in a waiting loop.