Showing posts with label for. Show all posts
Showing posts with label for. Show all posts

Monday, August 10, 2020

Show count and instance IDs of existing EC2 instances in all regions of AWS:

I'm kicking myself for not posting this earlier in my past life-- but here's a handy shell script that can list the number of instances in each region in AWS.


Show count and instance IDs of existing EC2 instances in all regions of AWS:


Single line:

Set my AWS profile to my personal account rather than work.  
To setup, use aws --profile personal configure   

$ export aws_profile=personal


If the buffer space can handle long entries, you can run this in a single line, otherwise there's a multi line below to run as a script.

    $ for i in $(aws --profile $aws_profile ec2 describe-regions --output json | grep "RegionName" | cut -d '"' -f4 | sort); do export buf=$(aws --profile $aws_profile ec2 --region $i describe-instances --output json | grep InstanceId | cut -d '"' -f4); export buf_num=$(echo $buf | grep -c .); echo "=== $i -> $buf_num instance(s) ===";echo $buf ; done

    Multi Line:

    Good for placing into a script to run, also listed this way for readability:
    1. #!/bin/bash export aws_profile=personal for i in $(aws --profile $aws_profile ec2 describe-regions --output json| grep "RegionName" | cut -d '"' -f4 | sort); do export buf=$(aws ec2 describe-instances \ --profile $aws_profile \ --region $i \ --output json | \ grep InstanceId | \ cut -d '"' -f4); export buf_num=$(echo $buf | grep -c .); echo "=== $i -> $buf_num instance(s) ==="; echo $buf; done

    Sample Output:

    1. === ap-northeast-1 -> 0 instances === === ap-northeast-2 -> 0 instances === === ap-south-1 -> 0 instances === === ap-southeast-1 -> 0 instances === === ap-southeast-2 -> 0 instances === === ca-central-1 -> 0 instances === === eu-central-1 -> 0 instances === === eu-north-1 -> 0 instances === === eu-west-1 -> 0 instances === === eu-west-2 -> 0 instances === === eu-west-3 -> 0 instances === === sa-east-1 -> 0 instances === === us-east-1 -> 0 instances === === us-east-2 -> 0 instances === === us-west-1 -> 0 instances === === us-west-2 -> 1 instances === i-012345678901234ab

    Tuesday, September 17, 2019

    Testing Gzipped Tarball in Bash

    Originally sourced from stack exchange. modified a bit:
    https://unix.stackexchange.com/questions/129599/test-tar-file-integrity-in-bash

    Modified from the source link above, the following command will iteratively test each .tgz file within the current directory for errors, and if any are detected, will report an error.

    $ for i in *.tgz; do echo "Testing $i..."; if tar xOfz $i &> /dev/null;  then echo "Error with tarball $i"; fi; done


    Options used:
     x  - Extract from archive
     O  - Write output to standard out instead of disk
     f  - Read input from a file
     z  - Tarball is compressed with Gzip (Some OSes may automatically detect Gzip and not need this option)
     &> /dev/null  -   Redirect both stdout and stderr to /dev/null

    Tuesday, August 16, 2016

    Sort a group of log4j-based log files by timestamp

    Assuming that each log file consists of complete timestamps on the beginning and end (i.e. it's not abruptly truncated), here's a quick command to list first and last timestamps of a set of log4j log files, sorted in ascending order by first timestamp:
    $ (for i in *.log.out*; do echo -n $i'\t'$(sed '1p;$!d' $i | cut -d ' ' -f-2 | tr '\n' '\t')'\n'; done) | sort -k2

    To sort by last timestamp on each file, change sort argument from -k2 to -k4

    Example:
    $ (for i in *.log.out*; do echo -n $i'\t'$(sed '1p;$!d' $i | cut -d ' ' -f-2 | tr '\n' '\t')'\n'; done) | sort -k2
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.30 2016-08-12 13:09:30,297 2016-08-12 13:53:27,785
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.29 2016-08-12 13:53:27,791 2016-08-12 16:02:24,046
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.28 2016-08-12 16:02:24,051 2016-08-12 16:13:18,553
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.27 2016-08-12 16:13:18,555 2016-08-12 16:40:21,115
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.26 2016-08-12 16:40:21,123 2016-08-12 17:26:27,145
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.25 2016-08-12 17:26:27,153 2016-08-12 17:27:35,976
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.24 2016-08-12 17:27:37,404 2016-08-12 17:56:26,459
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.23 2016-08-12 17:56:26,463 2016-08-12 18:25:09,816
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.22 2016-08-12 18:25:09,822 2016-08-12 19:10:34,036
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.21 2016-08-12 19:10:34,081 2016-08-12 19:44:30,899
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.20 2016-08-12 19:44:30,996 2016-08-12 20:01:21,222
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.19 2016-08-12 20:01:21,363 2016-08-12 21:23:20,933
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.18 2016-08-12 21:23:21,183 2016-08-12 23:14:29,238
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.17 2016-08-12 23:14:29,429 2016-08-13 00:47:05,370
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.16 2016-08-13 00:47:05,376 2016-08-13 01:01:45,803
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.15 2016-08-13 01:01:45,808 2016-08-13 02:23:24,499
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.14 2016-08-13 02:23:24,499 2016-08-13 09:41:18,893
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.13 2016-08-13 09:41:18,898 2016-08-13 11:05:50,145
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.12 2016-08-13 11:05:50,149 2016-08-13 11:58:56,914
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.11 2016-08-13 11:58:56,919 2016-08-13 13:58:17,794
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.10 2016-08-13 13:58:17,800 2016-08-13 15:55:48,996
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.9 2016-08-13 15:55:49,001 2016-08-13 17:05:04,935
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.8 2016-08-13 17:05:04,939 2016-08-13 17:58:42,547
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.7 2016-08-13 17:58:42,552 2016-08-13 18:13:34,622
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.6 2016-08-13 18:13:34,627 2016-08-13 19:41:18,039
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.5 2016-08-13 19:41:18,045 2016-08-13 21:13:34,207
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.4 2016-08-13 21:13:34,209 2016-08-13 23:13:13,734
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.3 2016-08-13 23:13:13,737 2016-08-14 00:04:13,013
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.2 2016-08-14 00:04:13,017 2016-08-14 00:58:07,933
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out.1 2016-08-14 00:58:07,937 2016-08-14 01:43:06,945
    hadoop-cmf-hdfs2-NAMENODE-namenode01.company.com.log.out 2016-08-14 01:43:07,107 2016-08-14 01:56:02,070