Collecting MR1 task logs from TTs in a single command

$tar cvfzh /tmp/`hostname`-job_201407311402-tasklogs.tar.gz /var/log/hadoop-0.20-mapreduce/userlogs/job_201407311402_0001/*

Reason why we include the -h option in tar:

The following location typically includes symlinks to the actual location of the task logs, which typically resides in the working space of the TT:
/var/log/hadoop-0.20-mapreduce/userlogs/job_201407311402_0001/ 

$ls -l
lrwxrwxrwx 1 mapred mapred  81 Aug  8 07:54 attempt_201407311402_0001_m_000000_0 -> /mapred/local/userlogs/job_201407311402_0001/attempt_201407311402_0001_m_000000_0

Collect multiple commands’ output to a single file

Syntax to run a few commands to gather some user / group ID info, as well as recursively list a folder’s contents and permissions all into a single file:

{ grep "^hadoop" /etc/group; id mapred; id hdfs; id mapred; id yarn; ls -ltrRa /var/log/hadoop-0.20-mapreduce/ } > /tmp/`hostname`-mr-info.txt


grep “^hadoop” /etc/group = list the accounts that are associated with the group hadoop
id mapred = display account ID info about the user mapred
id hdfs = display account ID info about the user hdfs
id yarn = display account ID info about the user yarn
ls -ltrRa /var/log/hadoop-0.20-mapreduce/ = Provide a recursive (-R) long list (-l)  of all (-a) directory and files from /var/log/hadoop-0.20-mapreduce, that’s sorted based on file timestamp (-t) in reverse order (-r)

{ ; } > /tmp/`hostname`-mr-info.txt  =   Run the commands first within the curly braces { }, then provide all the output to a file called (hostname of the machine)-mr-info.txt.  Important to have the final semicolon before terminating the end (right-most) curly brace otherwise a piping error will result.