Filter log file from common entries of no interest, along with blank lines

egrep -v "Trying to set|at "  hadoop-cmf-mapreduce1-JOBTRACKER-node.company.com.log.out | sed -e '/^\s*$/d' | less

Classic multi-platform Text Editor

I’ve been looking for a good classic CLI-version of a text editor that’s multi platform.

Apart from a heavily-customized version of Vim, I found this to be pretty awesome stuff:

http://triptico.com/software/mp.html

Some screenshots from the referring link above:


PS:  This is not my work, but the awesome work of Angel Ortega.  If you find this useful, please do support him and let him know!


Commands used to build on Ubuntu 14.04:

git clone https://github.com/angelortega/mp-5.x

git clone https://github.com/angelortega/mpdm

git clone https://github.com/angelortega/mpsl

sudo apt-get install libncursesw5-dev gnome-core-devel build-essential flex byacc lib ncursesw5 ncurses-dev ncurses-bin

cd mp-5.x
./config.sh --disable-gtk
make
sudo make-install

Multiply / Sum column data in awk/sed

Sourced from here:
http://unix.stackexchange.com/questions/115998/how-do-i-multiply-and-sum-column-data-using-awk-and-or-sed

I have several columns of data. I always have the same number of rows (say 5). In the 2nd column, I want to multipy the first value by 5, then the second value by 4, the third value by 3, etc. I then want to sum these values, and divide by the sum of the values in the second column. How would I do this in sed and/or awk?

Example:

4 5 7 1 2 3
5 1 2 3 1 2
4 2 3 6 1 2
3 4 1 6 3 3
2 3 1 2 1 6

Answer: (5*5 + 4*1 + 3*2 + 2*4 + 1*3)/(5 + 1 + 2 + 4 + 3) = 3.067

Solution 1:

Replace 6 with total (number of lines + 1) if needed:

awk '{mult+=$2*(6-NR); sum+=$2;} END {print mult/sum;}' yourfile.txt 

Displays: 3.06667

Solution 2:

If in case total number of lines are not known

$ cat file
4 5 7 1 2 3
5 1 2 3 1 2
4 2 3 6 1 2
3 4 1 6 3 3
2 3 1 2 1 6

$ awk 'FNR==NR{t = NR+1;next}{mult+=$2*(t-FNR);sum+=$2} 
       END{print mult/sum}' file{,}
3.06667

Explanation

awk 'FNR == NR{t=NR+1;next}

FNR variable --> Current line number in the current file

NR variable-->The total number of lines seen so far

The condition FNR == NR can only be true while awk is reading the first file of argument

t=NR+1 --> variable t holds line count +1 extra since it's required in our current context

next--> Stops the processing of the current input record and proceeds with the next input record

mult+=$2*(t-FNR) --> variable mult holds some of (5*(t-FNR) + ..... + 4*1 + 3*2 + 2*4 + 1*3)

sum+=$2 --> variable sum holds sum of column2

END--> AnEND rule is executed, once, after all the input has been read

print mult/sum --> Finally print mult/sum

file{,} --> you can also file file file but i used shortcut this bash trick you can say, try echo file{,} and echo file{,,} on your terminal.

Collect multiple commands’ output to a single file

Syntax to run a few commands to gather some user / group ID info, as well as recursively list a folder’s contents and permissions all into a single file:

{ grep "^hadoop" /etc/group; id mapred; id hdfs; id mapred; id yarn; ls -ltrRa /var/log/hadoop-0.20-mapreduce/ } > /tmp/`hostname`-mr-info.txt


grep “^hadoop” /etc/group = list the accounts that are associated with the group hadoop
id mapred = display account ID info about the user mapred
id hdfs = display account ID info about the user hdfs
id yarn = display account ID info about the user yarn
ls -ltrRa /var/log/hadoop-0.20-mapreduce/ = Provide a recursive (-R) long list (-l)  of all (-a) directory and files from /var/log/hadoop-0.20-mapreduce, that’s sorted based on file timestamp (-t) in reverse order (-r)

{ ; } > /tmp/`hostname`-mr-info.txt  =   Run the commands first within the curly braces { }, then provide all the output to a file called (hostname of the machine)-mr-info.txt.  Important to have the final semicolon before terminating the end (right-most) curly brace otherwise a piping error will result.

Found a relatively user-friendly Ubuntu snapshot / system restore utility.  So far works rather well, we’ll see how handy it is for backups.  Currently my go-to solution for ubuntu is to just do a disk-to-disk copy.  I’m open to suggestions for alternatives.

http://www.unixmen.com/systemback-restore-linux-system-previous-state/

How to re-enable keypad keys after installing Zsh

Sourced from http://superuser.com/questions/742171/zsh-z-shell-numpad-numlock-doesnt-work

1) Edit ~/.zshrc
2) Add the following entries:
bindkey -s “” “intended replacement keystroke”

To insert the correct keystroke above, hit Ctrl-V then press they key to replace.


My section for .zshrc for a standard full-key Mac KB is below:

——— Begin copy/paste ————

# Setting Keypad info:
bindkey -s "^[Oq" "1"
bindkey -s "^[Or" "2"
bindkey -s "^[Os" "3"
bindkey -s "^[Ot" "4"
bindkey -s "^[Ou" "5"
bindkey -s "^[Ov" "6"
bindkey -s "^[Ow" "7"
bindkey -s "^[Ox" "8"
bindkey -s "^[Oy" "9"
bindkey -s "^[Op" "0"
bindkey -s "^[OX" "="
bindkey -s "^[Oo" "/"
bindkey -s "^[Oj" "*"
bindkey -s "^[Om" "-"
bindkey -s "^[Ok" “+”
# NOTE-- The ^M below was added by the following key combinations: Ctrl-V, then pressing Enter
bindkey -s "^[OM" "^M”      

——— End copy/paste ————


Enjoy!

PS:  If you don’t know what Zsh is, take a look at this site:

http://ohmyz.sh/

#Awesomesauce

Hello again!

This post will serve as a welcome back! It's been a long time since I've posted on this blog and for good reason. Since my last few posts, I've moved into the BigData space and left the storage industry to get a start on BigData. I've since enjoyed my time at my current employer learning something new every day. As such, there was a recent interviewee that help inspired me to rekindle the interest of sharing information and what I've learned with the interwebs. This post is the beginning and start of just that. While I'm not going to make the promise of updating daily, I will make the promise of sharing all things tech that I use daily and will post here to help serve as a resource to stumble upon-- Essentially sharing my virtual shoebox of post-it notes with the hopes that you will also make use of the tidbits of shared info too. As much as I love my notes kept on Evernote, what's the use of hoarding all that information if I'm not able to share it? Enjoy! ~ Ant