Apache Yarn 2.4 version is bundled up with cryptocurrency miner. The miner once activated maxes out the CPU and memory of the machines.
Ways to solve this problem
- Do not use Yarn. Run Spark in standalone mode.
- Cordon off the system from public internet so that the miners won't be able to communicate with the cryptocurrency networks and will self destruct.
What doesn't work
- Killing the miner process(es). The processes resurrect themselves through cron jobs.
- Killing spark jobs and reissuing them. The miner code continues to exist on the cluster, so nothing short of killing the cluster gets rid of them.
How to identify the problem
- Do a htop, and you will see a line like the following -
/var/tmp/java -c /var/tmp/w.conf
2. Use Cluster monitoring tools like Ganglia, to validate the resource usage.