Uncompress file to HDFS without unzipping on local FS

Sourced from: http://bigdatanoob.blogspot.com/2011/07/copy-and-uncompress-file-to-hdfs.html


Quick and dirty method to be able to uncompress a large file directly into HDFS without having to uncompress locally:

Syntax:
       $ gunzip -c localfile.gz | hadoop fs -put - /user/user1/localfile


Explanation of options:
       gunzip -c  = The -c option causes the output of the gunzip operation to be written to the
       console.

       The '-' specified in the hadoop fs -put operation points the source file to be originated
       from the console.


So with this example:
       $ gunzip -c 3GB_json.gz | hadoop fs -put - /user/cloudera/3GB_json

The shell will run gunzip using the a compressed 3GB Json file (3GB_json.gz) sending its output to the console, which is then piped into the hadoop fs -put operation, which will then place the payload into the file /user/cloudera/3GB_json.

Calculate total amount of Physical RAM on all Active TaskTrackers in MR1 using Unix tools

The following should accomplish this assuming the interest is in identifying how much total physical RAM is available on all Active TaskTrackers (i.e. not blacklisted) in MR1 using common Unix commands:

for i in $(curl -s http://jobtracker.company.com:50030/machines.jsp?type=active | grep 'href="http://' | cut -d '=' -f2 | cut -d '"' -f2); do curl -s $i"jmx" | grep  'TotalPhysicalMemorySize' | grep -Po [0-9]+ ; done | paste -sd+ - | bc; 

NOTE:  Replace the http://jobtracker.company.com with the appropriate hostname of the JobTracker.

The above command will retrieve the list of active TaskTrackers from the JobTracker, iterate through the list, summing the 

TotalPhysicalMemorySize

 metric from each TaskTracker's JMX and display the end result, in bytes.

Example output:
[user@node10 ~]$ for i in $(curl -s http://node1.com:50030/machines.jsp?type=active | grep 'href="http://' | cut -d '=' -f2 | cut -d '"' -f2); do curl -s $i"jmx" | grep  'TotalPhysicalMemorySize' | grep -Po [0-9]+ ; done | paste -sd+ - | bc;

23488339968
[user@node10 ~]$



As replied here: https://www.quora.com/How-can-I-check-total-RAM-in-Hadoop-cluster-running-MR1-not-YARN

Collecting MR1 task logs from TTs in a single command

$tar cvfzh /tmp/`hostname`-job_201407311402-tasklogs.tar.gz /var/log/hadoop-0.20-mapreduce/userlogs/job_201407311402_0001/*

Reason why we include the -h option in tar:

The following location typically includes symlinks to the actual location of the task logs, which typically resides in the working space of the TT:
/var/log/hadoop-0.20-mapreduce/userlogs/job_201407311402_0001/ 

$ls -l
lrwxrwxrwx 1 mapred mapred  81 Aug  8 07:54 attempt_201407311402_0001_m_000000_0 -> /mapred/local/userlogs/job_201407311402_0001/attempt_201407311402_0001_m_000000_0

Collect multiple commands’ output to a single file

Syntax to run a few commands to gather some user / group ID info, as well as recursively list a folder’s contents and permissions all into a single file:

{ grep "^hadoop" /etc/group; id mapred; id hdfs; id mapred; id yarn; ls -ltrRa /var/log/hadoop-0.20-mapreduce/ } > /tmp/`hostname`-mr-info.txt


grep “^hadoop” /etc/group = list the accounts that are associated with the group hadoop
id mapred = display account ID info about the user mapred
id hdfs = display account ID info about the user hdfs
id yarn = display account ID info about the user yarn
ls -ltrRa /var/log/hadoop-0.20-mapreduce/ = Provide a recursive (-R) long list (-l)  of all (-a) directory and files from /var/log/hadoop-0.20-mapreduce, that’s sorted based on file timestamp (-t) in reverse order (-r)

{ ; } > /tmp/`hostname`-mr-info.txt  =   Run the commands first within the curly braces { }, then provide all the output to a file called (hostname of the machine)-mr-info.txt.  Important to have the final semicolon before terminating the end (right-most) curly brace otherwise a piping error will result.