Posts Tagged ‘shell-tricks’

2012w09

Sunday, March 4th, 2012

Ohai!

This week has been rather productive. I’ve both gotten work done AND learnt a crapload of stuff AND gotten to hack away on some scripts, leading to some personal programming revelations :D

When it comes to shell scripting, printf has become a new friend (leaving echo pretty much out in the cold). It is a continuation from last weeks post about shell tricks and I actually got to use it helping a colleague at work to better format the output of a script.

Something along the lines of:

printf "There were %s connections from %s\n" `some-counting-command` $tmpIP

I also wrote a small demonstrator for another colleague:

for i in `seq 1 21`; do printf "obase=16; $i\n" | bc; done

(Yes, I know about printf’s ability to convert/print hexadecimal on the fly)

for i in `seq 1 21`; do printf "%0.2x\n" $i; done

The for loop piping to bc was mostly for fun and to spark ideas about loops and pipes.

In another script I found myself needing two things: a reliable way to shut the script down (it was running a loop which would only stop when certain things appeared in a log) and a way to debug certain parts of the loop.

I know there is nothing special at all about it, but coming up with the solution instead of trying to google myself to a solution left me feeling like a rocket-scientist ;D

If you have a loop, and you want the ability to controllably get out of said loop, do something along the lines of this in your script:

touch /tmp/someUniqueName
while [ ... && -f /tmp/someUniqueName ]; do 
    ...
done

My first thought was to use $$ or $! to have a unique name but since I wouldn’t (couldn’t) be running more than one instance of this script at a time, I didn’t need to worry about that, and it would have made it a tiny bit harder to stop the script, so I finally (thanks razor) opted for a static, known, filename.

While that file exists, and you other loop conditions are normal, the loop will … loop on, but the second either condition becomes false, like someone removing the file ;) the loop doesn’t do another iteration.

Problem two was that I wanted a quick way to switch between running the script live, or in debug mode. Since running it live calls on some other stuff which then takes a while to reset, debugging the script using these calls would have been painfully slow, but I found a neat way around that:

DEBUG="echo" # (should be "" (debug off) or "echo" (debug on)
...
$DEBUG some-slow-command
...

With debug “on” it will print the command and any parameters, instead of executing it. It doesn’t look all that impressive in this shortened example, but instead imagine if you had more than ten of those places you wanted to debug.

What would you rather do? Edit the code in ten+ places, perhaps missing one, or just change in one place, and have it applied in all the places at once?

This script, once in place and running, did however bring with it another effect, namely a whole lot of cleanup. Cleanup which could only be performed by running a command and giving it some parameters, which could be found in the output of yet another command.

To make matters worse, not all lines of that output were things I wanted to remove. The format of that output was along the lines of:

<headline1>,<date>
<subheader1-1>,<date>
<subheader1-2>,<date>
<subheader1-3>,<date>
<subheader1-4>,<date>
...

<headline2>,<date>
<subheader2-1>,<date>
<subheader2-2>,<date>
<subheader2-3>,<date>
<subheader2-4>,<date>
...

<headline3>,<date>
<subheader3-1>,<date>
<subheader3-2>,<date>
<subheader3-3>,<date>
<subheader3-4>,<date>
...

Again, these seem like small amounts, but for the “headline” I needed to clean up, there were about 70 subheaders, out of which I wanted to clean up all but one. Thankfully, that one subheader I wanted to preserve was not named in the same way as the other 69 subheaders (which had been created programmatically using the loop above).

Also, it was rather important not to delete any other headlines or subheaders. awk to the rescue! But first, here are some facts going into this problem:

  • the subheaders I wanted removed all shared a common part of name between each section is an empty line
  • I knew the name of the section heading containing the subheaders to remove
  • To remove one such subheader I’d need to execute a command giving the subheader as an argument

And this is what I did:

list-generating-command | awk -F, 
    '/headline2/ { m = 1 }
    /subheader/ && m == 1 { print $1 }
    /^$/ { m = 0 }' | while read name;
do
    delete-subheader-command $name
done

Basically, what is going on here is that first of all, setting the field separator to “,” and then, once we come across the unique headline string, set a little flag telling awk that we are within the matching area, and if it can only match the subheader pattern, it gets to print the first column of that line. Finally, when we reach any line containing nothing but a newline, unset the flag, so that there will be no more printouts

And another thing I’ve stumbled upon, and which I already know where I can use this, is this post and more specifically:

diff -u .bashrc <(ssh remote cat .bashrc)

(although it is not .bashrc files I am going to compare).

And finally, some links on assorted topics:

2011w50

Sunday, December 18th, 2011

tmux

tmux is a terminal multiplexer, resembling screen and seemingly straight-forward to configure.

Now, those of you paying attention will know that I use wmii, a tiling window manager, and you may ask what the difference is between creating one big tmux window and laying out a couple of terminals in that, or letting wmii place those terminals beside each other itself.

The answer is that for most instances, wmii will be enough, but just a little while ago I discovered a killer feature (one which makes me wish that tmux was available at work), namely the abililty to perform:

C^b:setw synchronize-panes

(demonstrated here) which simply outputs whatever you type into one of the terminals, into all the other terminals in this tmux instance as well.

How is this useful? If you have a couple of servers, on which you need to execute the exact same command, you simply start tmux, create a terminal for each server (and log in to that server) and then ask tmux to synchronize the panes, and then you type in your commands.

(Yes, this could probably be easily solved with a bash for-loop as well, depending on the amount of commands and their complexity)

Stupid Shell Tricks

I’ve known about ^foo^bar for a while (i.e. you type
$ some-command wif a typo
and you then do
$ ^wif^with
to have the shell replace the first instance of that typo with the correct spelling (hopefully ;D)

But, this is really only good for typos or when there is ONE instance to replace. ^foo^bar won’t replace EVERY foo with bar, only the first occurrance. Which is sometimes now what you wanted.

Enter !!:gs/foo/bar which replaces ALL instances of foo in the previous command, with bar, and re-executes it. Thanks to http://blog.urfix.com/25-linux-commands/ for that.

less

I think I have touched upon this before, but here we go anyway: it is possible to export an environment variable called “LESS” and less will read this and determine any runtime special behaviour based on the contents of the variable.

I am currently trying out export LESS='FiX' where F makes less exit if the contents are short enough to all fit on the screen, i is for case-insensitive search and X for stopping less from sending the termcap initialization and deinitialization strings.

This means that when less exits, it won’t clear the screen (which would be a bummer if using F and less:ing short files…)

Links

A pretty interesting read about how one could “work in the cloud.” I would have chosen other hardware/software (except for vim of course) but to all his/her own, right?

This sounds as if it could be useful for making sure that your logs are really your real logs. Makes sense, right? ;)

From the reptyr readme: reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don’t want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

I am apparantly not the only one to get the idea of describing their software stack.

A pretty cool more-utils command, ifne, which continues execution of the rest of the command, iff data was coming into ifne’s stdin.

:wq