Tuesday, January 5, 2016

Calculate total amount of Physical RAM on all Active TaskTrackers in MR1 using Unix tools

The following should accomplish this assuming the interest is in identifying how much total physical RAM is available on all Active TaskTrackers (i.e. not blacklisted) in MR1 using common Unix commands:
for i in $(curl -s http://jobtracker.company.com:50030/machines.jsp?type=active | grep 'href="http://' | cut -d '=' -f2 | cut -d '"' -f2); do curl -s $i"jmx" | grep  'TotalPhysicalMemorySize' | grep -Po [0-9]+ ; done | paste -sd+ - | bc; 
NOTE:  Replace the http://jobtracker.company.com with the appropriate hostname of the JobTracker.
The above command will retrieve the list of active TaskTrackers from the JobTracker, iterate through the list, summing the 
TotalPhysicalMemorySize
 metric from each TaskTracker's JMX and display the end result, in bytes.

Example output:
[user@node10 ~]$ for i in $(curl -s http://node1.com:50030/machines.jsp?type=active | grep 'href="http://' | cut -d '=' -f2 | cut -d '"' -f2); do curl -s $i"jmx" | grep  'TotalPhysicalMemorySize' | grep -Po [0-9]+ ; done | paste -sd+ - | bc;

23488339968
[user@node10 ~]$



As replied here: https://www.quora.com/How-can-I-check-total-RAM-in-Hadoop-cluster-running-MR1-not-YARN