Random sleep time in Bash

Recently, I needed to setup a script that would sleep a certain number of seconds between 1 and 8.  Stumbled across this particular item, which was adapted to my need:


$ sleep $[ ( $RANDOM % 8 ) + 1 ]s

This would sleep anywhere from 1 second to 8 seconds before continuing.  If the need comes up to set to a different limits, you can change the 8 to the max number and the 1 to the minimum.  Note that increasing the minimum value will affect the max number as well if min > 1.

Sourced from: http://blog.buberel.org/2010/07/howto-random-sleep-duration-in-bash.html

Uncompress file to HDFS without unzipping on local FS

Sourced from: http://bigdatanoob.blogspot.com/2011/07/copy-and-uncompress-file-to-hdfs.html


Quick and dirty method to be able to uncompress a large file directly into HDFS without having to uncompress locally:

Syntax:
       $ gunzip -c localfile.gz | hadoop fs -put - /user/user1/localfile


Explanation of options:
       gunzip -c  = The -c option causes the output of the gunzip operation to be written to the
       console.

       The '-' specified in the hadoop fs -put operation points the source file to be originated
       from the console.


So with this example:
       $ gunzip -c 3GB_json.gz | hadoop fs -put - /user/cloudera/3GB_json

The shell will run gunzip using the a compressed 3GB Json file (3GB_json.gz) sending its output to the console, which is then piped into the hadoop fs -put operation, which will then place the payload into the file /user/cloudera/3GB_json.

Sort a group of log4j-based log files by timestamp

Assuming that each log file consists of complete timestamps on the beginning and end (i.e. it's not abruptly truncated), here's a quick command to list first and last timestamps of a set of log4j log files, sorted in ascending order by first timestamp:

$ (for i in *.log.out*; do echo -n $i'\t'$(sed '1p;$!d' $i | cut -d ' ' -f-2 | tr '\n' '\t')'\n'; done) | sort -k2

To sort by last timestamp on each file, change sort argument from -k2 to -k4

Example:

$ (for i in *.log.out*; do echo -n $i'\t'$(sed '1p;$!d' $i | cut -d ' ' -f-2 | tr '\n' '\t')'\n'; done) | sort -k2

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.30 2016-08-12 13:09:30,297 2016-08-12 13:53:27,785

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.29 2016-08-12 13:53:27,791 2016-08-12 16:02:24,046

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.28 2016-08-12 16:02:24,051 2016-08-12 16:13:18,553

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.27 2016-08-12 16:13:18,555 2016-08-12 16:40:21,115

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.26 2016-08-12 16:40:21,123 2016-08-12 17:26:27,145

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.25 2016-08-12 17:26:27,153 2016-08-12 17:27:35,976

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.24 2016-08-12 17:27:37,404 2016-08-12 17:56:26,459

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.23 2016-08-12 17:56:26,463 2016-08-12 18:25:09,816

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.22 2016-08-12 18:25:09,822 2016-08-12 19:10:34,036

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.21 2016-08-12 19:10:34,081 2016-08-12 19:44:30,899

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.20 2016-08-12 19:44:30,996 2016-08-12 20:01:21,222

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.19 2016-08-12 20:01:21,363 2016-08-12 21:23:20,933

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.18 2016-08-12 21:23:21,183 2016-08-12 23:14:29,238

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.17 2016-08-12 23:14:29,429 2016-08-13 00:47:05,370

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.16 2016-08-13 00:47:05,376 2016-08-13 01:01:45,803

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.15 2016-08-13 01:01:45,808 2016-08-13 02:23:24,499

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.14 2016-08-13 02:23:24,499 2016-08-13 09:41:18,893

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.13 2016-08-13 09:41:18,898 2016-08-13 11:05:50,145

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.12 2016-08-13 11:05:50,149 2016-08-13 11:58:56,914

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.11 2016-08-13 11:58:56,919 2016-08-13 13:58:17,794

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.10 2016-08-13 13:58:17,800 2016-08-13 15:55:48,996

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.9 2016-08-13 15:55:49,001 2016-08-13 17:05:04,935

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.8 2016-08-13 17:05:04,939 2016-08-13 17:58:42,547

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.7 2016-08-13 17:58:42,552 2016-08-13 18:13:34,622

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.6 2016-08-13 18:13:34,627 2016-08-13 19:41:18,039

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.5 2016-08-13 19:41:18,045 2016-08-13 21:13:34,207

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.4 2016-08-13 21:13:34,209 2016-08-13 23:13:13,734

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.3 2016-08-13 23:13:13,737 2016-08-14 00:04:13,013

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.2 2016-08-14 00:04:13,017 2016-08-14 00:58:07,933

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.1 2016-08-14 00:58:07,937 2016-08-14 01:43:06,945

hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out 2016-08-14 01:43:07,107 2016-08-14 01:56:02,070

Mac OSX (El Capitan) Software Update tool via CLI

List available software updates via CLI (but don't install them):

$ sudo softwareupdate -l
Software Update Tool
Copyright 2002-2015 Apple Inc.

Finding available software
Software Update found the following new or updated software:
   * OS X El Capitan Update-10.11.5
OS X El Capitan Update (10.11.5), 740450K [recommended] [restart]
   * RAWCameraUpdate6.19-6.19
Digital Camera RAW Compatibility Update (6.19), 7575K [recommended]
   * iTunesXPatch-12.4
iTunes (12.4), 144804K [recommended]



Install all pending software updates via CLI:

$ sudo softwareupdate -i -a
Software Update Tool
Copyright 2002-2015 Apple Inc.

Finding available software

Downloading OS X El Capitan Update
Downloading Digital Camera RAW Compatibility Update
Downloading iTunes
Downloaded Digital Camera RAW Compatibility Update
Downloaded iTunes
Downloaded OS X El Capitan Update
Installing OS X El Capitan Update, Digital Camera RAW Compatibility Update, iTunes
Done with OS X El Capitan Update
Done with Digital Camera RAW Compatibility Update
Done with iTunes
Done.

You have installed one or more updates that requires that you restart your

computer.  Please restart immediately.

Calculate total amount of Physical RAM on all Active TaskTrackers in MR1 using Unix tools

The following should accomplish this assuming the interest is in identifying how much total physical RAM is available on all Active TaskTrackers (i.e. not blacklisted) in MR1 using common Unix commands:

for i in $(curl -s http://jobtracker.company.com:50030/machines.jsp?type=active | grep 'href="http://' | cut -d '=' -f2 | cut -d '"' -f2); do curl -s $i"jmx" | grep  'TotalPhysicalMemorySize' | grep -Po [0-9]+ ; done | paste -sd+ - | bc; 

NOTE:  Replace the http://jobtracker.company.com with the appropriate hostname of the JobTracker.

The above command will retrieve the list of active TaskTrackers from the JobTracker, iterate through the list, summing the 

TotalPhysicalMemorySize

 metric from each TaskTracker's JMX and display the end result, in bytes.

Example output:
[user@node10 ~]$ for i in $(curl -s http://node1.com:50030/machines.jsp?type=active | grep 'href="http://' | cut -d '=' -f2 | cut -d '"' -f2); do curl -s $i"jmx" | grep  'TotalPhysicalMemorySize' | grep -Po [0-9]+ ; done | paste -sd+ - | bc;

23488339968
[user@node10 ~]$



As replied here: https://www.quora.com/How-can-I-check-total-RAM-in-Hadoop-cluster-running-MR1-not-YARN

Read a file, extract contents encapsulated in parentheses

Sourced from http://objectmix.com/awk/26995-retrieve-string-between-parentheses.html

$ cat some_file | gawk '{if (match($0,/\((.*)\)/,f)) print f[1]}' | cut -d ':' -f1 | sort | uniq

The above command will be helpful when reviewing stack trace output from command line to filter out and identify all the referenced .java files listed.

For example:

$ cat stacktrace_1.txt | gawk '{if (match($0,/\((.*)\)/,f)) print f[1]}' | cut -d ':' -f1 | sort | uniq
FilterFileSystem.java
Job.java
JobSubmissionFiles.java
JobSubmitter.java
Native Method
ProcessBuilder.java
RawLocalFileSystem.java
Shell.java
Subject.java
ToolRunner.java
UserGroupInformation.java

Classic multi-platform Text Editor

I’ve been looking for a good classic CLI-version of a text editor that’s multi platform.

Apart from a heavily-customized version of Vim, I found this to be pretty awesome stuff:

http://triptico.com/software/mp.html

Some screenshots from the referring link above:


PS:  This is not my work, but the awesome work of Angel Ortega.  If you find this useful, please do support him and let him know!


Commands used to build on Ubuntu 14.04:

git clone https://github.com/angelortega/mp-5.x

git clone https://github.com/angelortega/mpdm

git clone https://github.com/angelortega/mpsl

sudo apt-get install libncursesw5-dev gnome-core-devel build-essential flex byacc lib ncursesw5 ncurses-dev ncurses-bin

cd mp-5.x
./config.sh --disable-gtk
make
sudo make-install