Archive for the ‘Documentation’ Category

2012w01

Sunday, January 8th, 2012

column

The other day I wanted some prettier (tabularized) output and of course someone has already wanted this and of course there are tools for that :)

bash_completion

This is so frakking cool! I’ve built this little shellscript “vault.sh” which is a simple wrapper script for mounting and unmounting encfs mounts.

It takes two parameters: operation and target, where operation can be one of “lock” and “unlock”, and target—at present—resolves to “thunderbird” (signifying my .thunderbird directory).

Since I intend to expand this with more encrypted directories as I see fit, I don’t want to hard-code that.

What I did want, however, was to be able to auto complete operation and target. So I looked around, and found this post, and although I couldn’t derive enough knowledge from it to solve my particular problem, having multiple levels of completion, the author was gracious enough to provide references to where s/he had found the knowledge (here, here and here). That second link was what did it for me.

My /etc/bash_completion.d/vault.sh now looks like this:

_vault()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    first="lock unlock"
    second="thunderbird"

    if [[ ${cur} == * && ${COMP_CWORD} -eq 2 ]] ; then
        COMPREPLY=( $(compgen -W "${second}" -- ${cur}) )
        return 0
    fi

    if [[ ${cur} == * && ${COMP_CWORD} -eq 1 ]] ; then
        COMPREPLY=( $(compgen -W "${first}" -- ${cur}) )
        return 0
    fi
}
complete -F _vault vault.sh

And all the magic is happening in the two if-statements. Essentially: if current word (presently half typed and tabbed) is whatever, and this is the second argument to the command, respond with suggestions taken from the variable $second.

Otherwise, if current word is whatever, and this is the first parameter, take suggestions from the variable $first.

Awsum!

awk for great good

Another great use for awk: viewing selected portions of source code. For instance, in Perl, if you just want to view a specific subroutine, without getting distracted by all the other crud, you could do: $ awk '/sub SomeSubName/,/}/' somePerlModule.pm

Links

If PHP were British, perhaps it’s just me, but I find it hilarious.

PayPal just keeps working their charm…

Belarus just… wait what?

Why we need version control

Preserving space, neat!

Fuzzy string matching in Python

If you aren’t embarrassed by v1.0 you didn’t release it early enough

The makers schedule, oldie but goldie

CSS Media Queries are pretty cool

Static site generator using the shell and awk

A netstat companion

Reducing code nesting

Comparing images using perceptual hashes

Microsofts GPS “avoid ghetto” routing algorithm patent…

2011w52

Sunday, January 1st, 2012

Merry belated christmas greetings everyone! And by the time this post is published I could extend it with Happy belated new years greetings as well ;)

vim + html5 syntax

I’ve been tinkering a lot with html5 during my vacation and vim just didn’t want to play nicely with the new html-tags.

Namely, as it wouldn’t recognise the new semantic structural tags (footer, header, article, section, nav, aside) it wouldn’t indent the source properly and it was a cause for both distraction, and the resulting frustration.

I was not the first to feel this frustration, and a quick search turned up this result which solved both the html and css syntax issues (check the comments for the css solution). Very elegant solution, and now I’ve also learned about vim’s .vim/after/ directory… That was pretty cool.

Learning html5

I’ve actually shied away from doing stuff with html5, as whenever I tried to wrap my head around the new tags and how they should be used, there were just a myriad of different sites interpreting the usage in subtle but differing ways, but I finally found a resource which makes sense to me, so until a definitive interpretation has been hammered out, that’s the one I’m going to stick with.

Also, for sticky footers using css, and html5, check out this page. I had no trouble getting that to work.

Links

This question pretty much sums up why I like the command line so much

This looks interesting for synching (and deleting) without having to worry about doing “the right thing”

Nice list of things one could do with a home server

Doing it for teh lulz, 1903 style

EA, Nintendo and Sony now only covertly support SOPA (through their membership in various interest organizations). Wanting to eat the cake and still have it huh?

Tom’s Hardware not being amused by SOPA

Oh how I so hope that Wikipedia, Google, et al, will go down this path. (I do think there is a difference between companies lobbying, writing laws, and pressuring governments, and companies urging people to put pressure on governments, so yes, I think this is ok)

An interesting theory about why cinemas are having such a rough time

Haven’t had a chance to try this, but creating art using a written grammar does sound pretty neat, especially if you could get a script and /dev/random involved as well ;)

German police tracking people via silent SMS. I am beginning to think that rms is correct in his cellphone “usage”

Too much reading and constant information overload makes us pretty little passive consumers

My Software Stack 2011 edition

Saturday, December 31st, 2011

I realize that I haven’t written my customary “software stack” post for this year yet. But hey, from where I’m sitting, I still have … 36 minutes to spare ;)

I’ll be using the same categories as last year; system, communications, web, development, office suite, server, organization, and entertainment.

System

The OS of choice is still Archlinux, my window manager is still wmii, my terminal emulator is rxvt-unicode, upgraded by also installing urxvt-tabbedex.

My shell is still bash, my cron daemon is still fcron, and my network manager is wicd.

To this configuration I’ve added the terminal multiplexer tmux, and have lately found out just how useful mc can be. Oh, and qmv from the renameutils package is now a given part of the stack.

Communications

Not much change here, Thunderbird for email, Pidgin for instant messaging, irssi for IRC.

Heybuddy has been replaced by identicurse as my micro-blogging (identi.ca) client. Heybuddy is very nice, but I can use identicurse from the commandline, and it has vim-like bindings.

For Pidgin I use OTR to encrypt conversations. For Thunderbird I use the enigmail addon along with GnuPG.

This means that Thunderbird still hasn’t been replaced by the “mutt-stack” (mutt, msmtp, offlineimap and mairix) and this is mostly due to me not having the energy to learn how to configure mutt.

I also considered trying to replace Pidgin with irssi and bitlbee but Pidgin + OTR works so well, and I have no idea about how well OTR works with bitlbee/irssi (well, actually, I’ve found irssi + OTR to be flaky at best.

Web

Not much changed here either, Firefox dominates, and I haven’t looked further into uzbl although that is still on the TODO list, for some day.

I do some times also use w3m, elinks, wget, curl and perl-libwww.

My Firefox is customized with NoScript, RequestPolicy, some other stuff, and Pentadactyl.

Privoxy is nowadays also part of the loadout, to filter out ads and other undesirable web “resources”.

Development

In this category there has actually been some changes:

  • gvim has been completely dropped
  • eclipse has been dropped, using vim instead
  • mercurial has been replaced by git

Thanks in no small part to my job, I have gotten more intimate knowledge of awk and expect, as well as beginning to learn Perl.

I still do some Python hacking, a whole lot of shell scripting, and for many of these hacks, SQLite is a faithful companion.

Doh! I completely forgot that I’ve been dabbling around with Erlang as well, and that mscgen has been immensely helpful in helping me visualize communication paths between various modules.

“Office suite”

I still use LaTeX for PDF creation (sorry hook, still haven’t gotten around to checking out ConTeXt), I haven’t really used sc at all, it was just too hard to learn the controls, and I had too few spreadsheets in need of creating. I use qalculate almost on a weekly basis, but for shell scripts I’ve started using bc instead.

A potential replacement for sc could be teapot, but again, I usually don’t create spreadsheets…

Server

Since I’ve dropped mercurial, and since the mercurial-server package suddenly stopped working after a system update, I couldn’t be bothered to fix it, and it is now dropped.

screen and irssi is of course always a winning combination.

nginx and uwsgi has not been used to any extent, I haven’t tried setting up a VPN service, but I have a couple of ideas for the coming year (mumble, some VPN service, some nginx + Python/Perl thingies, bitlbee) and maybe replace the Ubuntu installation with Debian.

Organization

I still use both vimwiki and vim outliner, and my Important Dates Notifier script.

Still no TaskJuggler, and I haven’t gotten much use out of abook.

remind has completely replaced when, while I haven’t gotten any use what so ever out of wyrd.

Entertainment

For consuming stuff I use evince (PDF), mplayer (video), while for music, moc has had to step down from the throne, to leave place for mpd and ncmpcpp.

eog along with gthumb (replacing geeqie) handles viewing images.

For manipulation/creation needs I use LaTeX, or possibly Scribus, ffmpeg, audacity, imagemagick, inkscape, and gimp.

Bonus: Security

I thought I’d add another category, security, since I finally have something worthwhile to report here.

I’ve begun encrypting selected parts of my hard drive (mostly my email directory) using EncFS, and I use my passtore script for password management.

And sometimes (this was mostly relevant for when debugging passtore after having begun actively using it) when I have a sensitive file which I for a session need to store on the hard drive, in clear text, I use quixand to create an encrypted directory with a session key only stored in RAM. So once the session has ended, there is little chance of retrieving the key and decrypting the encrypted directory.

Ending notes

That’s about it. Some new stuff, mostly old stuff, only a few things getting kicked off the list. My stack is pretty stable for now. I wonder what cool stuff I will find in 2012 :D

:wq

2011w51

Sunday, December 25th, 2011

Bash variable string operators

I had a file filled with URLs to files I needed to download. Some of the files on the list, however, had already been downloaded, so no need to do it all again.

Should be fairly easy, right? cat the file to a while loop, reading the lines one by one, extracting the filename from the URL, check that it isn’t existing already, and if it isn’t, download it with wget.

So… how do you go about extracting the filename? You could certainly use sed and store the extracted filename in a separate variable, but that seems kindof wasteful, especially in a one-liner while loop. This article provided me with another option.

${line##*/} which deletes the longest possible match from the left (which in this case means up to (including) the last “/”) i.e. everything up to the name of the file.

No can haz censorship plz

If you’d like to make it clear that you too oppose SOPA (which, fittingly, means “garbage” in Swedish) then head over to Github, pick up your very own copy of stopcensorship.js, embed it on your site, and you’re set :)

I am also noting, with some glee, that GoDaddy is catching a whole lot of flak for their support of SOPA.

The only thing companies truly understand is when you hit them where it hurts, and that is their wallets (or as some brilliant person jokingly expressed it: “stop hitting us in our quarterly reports!”), and the only way to do that, is by voting with your own wallet.

I’m so happy about the fact that more and more people are catching on to this realization that I could… shit rainbows :)

Japanese Whaling + Tsunami disaster relief funds = disgusting

Just when I didn’t believe it possible for the Japanese whaling industry to appear as bigger scumbags than they already appear (yes, it is a quite one-sided story we’re getting from “Whale Wars” but according to National Geographic, the whalers have gotten the chance to tell their side of the story, and it would seem likely that they decline because they know full well just what type of scumbags they are… but hey, that’s just my opinion…) they go and do even more disgusting stuff, like using money from the tsunami relief donations to hire security ships to keep the Sea Shepherd Conservation Society away from their dirty business…

:wq

2011w50

Sunday, December 18th, 2011

tmux

tmux is a terminal multiplexer, resembling screen and seemingly straight-forward to configure.

Now, those of you paying attention will know that I use wmii, a tiling window manager, and you may ask what the difference is between creating one big tmux window and laying out a couple of terminals in that, or letting wmii place those terminals beside each other itself.

The answer is that for most instances, wmii will be enough, but just a little while ago I discovered a killer feature (one which makes me wish that tmux was available at work), namely the abililty to perform:

C^b:setw synchronize-panes

(demonstrated here) which simply outputs whatever you type into one of the terminals, into all the other terminals in this tmux instance as well.

How is this useful? If you have a couple of servers, on which you need to execute the exact same command, you simply start tmux, create a terminal for each server (and log in to that server) and then ask tmux to synchronize the panes, and then you type in your commands.

(Yes, this could probably be easily solved with a bash for-loop as well, depending on the amount of commands and their complexity)

Stupid Shell Tricks

I’ve known about ^foo^bar for a while (i.e. you type
$ some-command wif a typo
and you then do
$ ^wif^with
to have the shell replace the first instance of that typo with the correct spelling (hopefully ;D)

But, this is really only good for typos or when there is ONE instance to replace. ^foo^bar won’t replace EVERY foo with bar, only the first occurrance. Which is sometimes now what you wanted.

Enter !!:gs/foo/bar which replaces ALL instances of foo in the previous command, with bar, and re-executes it. Thanks to http://blog.urfix.com/25-linux-commands/ for that.

less

I think I have touched upon this before, but here we go anyway: it is possible to export an environment variable called “LESS” and less will read this and determine any runtime special behaviour based on the contents of the variable.

I am currently trying out export LESS='FiX' where F makes less exit if the contents are short enough to all fit on the screen, i is for case-insensitive search and X for stopping less from sending the termcap initialization and deinitialization strings.

This means that when less exits, it won’t clear the screen (which would be a bummer if using F and less:ing short files…)

Links

A pretty interesting read about how one could “work in the cloud.” I would have chosen other hardware/software (except for vim of course) but to all his/her own, right?

This sounds as if it could be useful for making sure that your logs are really your real logs. Makes sense, right? ;)

From the reptyr readme: reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don’t want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

I am apparantly not the only one to get the idea of describing their software stack.

A pretty cool more-utils command, ifne, which continues execution of the rest of the command, iff data was coming into ifne’s stdin.

:wq

2011w49

Sunday, December 11th, 2011

IFS and for loops

I needed to iterate over lines in a file, and I needed to use a for loop (well, I probably could have solved it in a myriad other ways, but that’s not the point).

Thanks Luke, updating for clarification: I simplified this problem somewhat to make the post shorter, but the problem in need of solving involved doing some addition across lines, and have the result available after the loop, and for this, I have learned, pipes and the “while read line; do”-pattern isn’t of much help.

So I tell the for loop to do

for line in `cat myfile`; do echo $lines; done

And obviously this doesn’t work, as the IFS variable is set to space, and thus prints out each word on a separate line, instead of printing all the words in one line on lines.

So I think “oh I know, I’ll just change the IFS variable” and try:

IFS="\n"

and this turns out poorly, with the for loop now believing every “n” (and “\”, thanks Luke :) ) to be the separator and breaking words on that instead… So I try with single quotes, no joy…

Having approached and passed the point where it is taking me more time to solve this problem rather than solving the problem I was using the loop for, I stop trying and start googling, finding this post.

The solution is rather nifty actually:

IFS='
'

There you have it.

Cube

I haven’t tried it out, but this seems like it could be useful. From that page one could also make their way to one of the projects powering Cube, namely D3, and on that page you can find one or two (or more) interesting diagram types.

And filed under “oh frak! how glad I am that I never got a paypal account!”:

  1. http://www.regretsy.com/2011/12/05/cats-1-kids-0/
  2. https://www.thepaypalblog.com/2011/12/regretsy-issue-resolution/
  3. http://thegreengeeks.wordpress.com/2011/12/06/why-paypal-is-wrong-regarding-regretsy-according-to-their-own-policies/

:wq

awk, filtering and counting

Monday, December 5th, 2011

Suppose that you have a file containing some structured data, something perhaps along the lines of this, highly fictive but yet remarkably common, syntax:

<id><separator><somestring><separator><integer>

Now, let’s say that there were 99999 lines of this to go through, and the file is unsorted, and you wanted to find all the lines where SOMESTRING is foo, and then sum up the INTEGER field of those lines.

I almost had this problem at work, except my file probably didn’t contain more than a hundred or so lines.

For this I wrote a Perl script, which worked well, with the small inconvenience that I’d have to move that script onto each system where I’d want to use it.

Pontus, never the one to berate anyones efforts, but still finding room for improvements, both in the fact that my approach, the script, carried that inconvenience, and that is was very verbose when compared to the solution he ultimately suggested, he showed me a better way, the awk way.

$ awk -F<separatorGoesHere> 'BEGIN { SUM = 0 } /<someStringGoesHERE/ { SUM += $3 } END { print SUM }' <fileToBeParsedGoesHere>

I said before that my real file, at work, was small, so awk crunched through it at lightning speed. I also suggested a file containing 99.999 lines, and I did that to prove a point, namely:

Using this script:

#!/usr/bin/env python2

import random

filename = "awk.example.txt"
index = 0
iterations = 100000
choices = ['foo', 'bar', 'baz']
fh = open(filename, 'w')

for index in range(1, iterations):
    fh.write("%d, %s, %d\n" % (index,
                               random.choice(choices),
                               random.randint(0, 100)))
fh.close()

I generated a file (~1.5Mb) with a couple of lines ;) and let awk loose on it:

$ time awk -F, 'BEGIN { SUM = 0 } /foo/ { SUM += $3 } END { print SUM }' awk.example.txt

Which on my netbook took 0.241 seconds to complete.

real	0m0.241s
user	0m0.237s
sys	0m0.000s

Or in other words: awk if pretty frakking fast!

Now, let’s break it down:

awk

obviously, is the command, and it rocks, ‘nuf said.

-F,

means “change the field separator (from whitespace) to commas”

And then it gets tricky, but not as tricky as at least I was lead to believe.

There are two single-quotes, and between these we place all the things we want awk to do for us.

One good thing to note is that the syntax for awk is quite simple, something I didn’t grasp at first. It goes like this:

<somePattern> { <someAction> }

And that’s it. You can chain several <pattern>{<actions>} after each other.

In my, well Pontus’, command above, there are three such pairs:

BEGIN { SUM = 0 }

which is just another way of saying “before we start executing, create a variable SUM and set its value to 0″

/foo/ { SUM += $3 }

If you’re familiar with regular expressions you might have stumbled upon the pattern in which you enclose an expression between two slashes, and that pattern is used to search (or match) contents of lines or files. That’s what we’re doing here. So we’re basically saying “find lines containing foo, and from these lines extract column number three ($3), and increment the variable SUM by the value stored in column three.”

If instead, you’d wanted to count all the lines containing foo, SUM += 1 would have done that job.

Finally:

END { print SUM }

which should be pretty obvious: “When all is said and done, print whatever is stored in the variable SUM”

And last but not least, outside the single-quotes, we give awk the name of the file we wish it to process.

This is just a fantastic tool which I regret not having taking the time to learn the basics of earlier. Thank you Pontus for making me see the light (again) ;)

:wq

2011w47

Sunday, November 27th, 2011

top

So I have been playing around some more with top, and I have to say that I no longer feel any reason to install htop.

Perhaps if I dig into the manpage of htop, I’ll yet again revert to thinking it is better, but for now there’s no need.

I can get coloring (z), I can filter on users (u<username><enter>), I can control how many processes I list (n<int><enter>), and I can have the current sort field highlighted (x), and when I am happy with the configuration, W lets me save it to ${HOME}/.toprc

grep

Pontus showed me a new shiny flag for grep the other day: -s which, to quote the grep manpage, says Suppress error messages about nonexistent or unreadable files.

And this is awesome for when your are doing directory-wide recursive greps in places where you might not have the credentials to look through all the files.

Beware though as there are some differences between GNU grep and UNIX grep.

RabbitMQ

I’ve many times read about RabbitMQ and how that is good to know and if you don’t know what it is you’ve been hiding under a rock (apparantly I have), because it wasn’t until this week I actually found a blogpost that could adequately explain to me what it is and what it’s good for.

And thanks to that blogpost I now have yet one more thing pushed onto the “toLearn” stack…

blockdiag

This next thing I found is a more or less graphviz, wrapped around a python(2) module which helps create block diagrams.

There are actually four modules, blockdiag, seqdiag, actdiag, and finally nwdiag, and I could imagine all four having their use under certain circumstances.

Links

GNU source highlight — For most of your sourcecode highlighting needs

2011w46

Sunday, November 20th, 2011

First of all: this is really disturbing.


Commands and flags

I think I’ve already mentioned watch, and how that could be useful at time (e.g. $ watch -n 10 -d 'ls -l')

I just found out about a value which can optionally be appended to the -d flag: -d=cumulative

It has its own flag as well --cumulative, and quoting the man-page it makes highlighting “sticky”, presenting a running display of all positions that have ever changed.

Also, this week I learnt about sdiff, which seems neat if you’re on a system which doesn’t have vim (and thus vimdiff) installed.

Anoter nice flag I just found for grep is -m <int> which tells grep to stop looking after the INT first matches.

Scripting Vim

Ok, so I’ve been running into this problem where I am using my own .vimrc configuration in other places, in systems where the vim version isn’t the same as the one I use myself.

This has proven problematic as some of the configuration options I use (most notably set cul (which gives me a better indication about which line the cursor is on)) doesn’t exist in … say a vim version less than 7.

Which meant that if I loaded the same .vimrc config on a system running a vim version earlier than 7, I’d get a warning at startup, which I’d have to press enter to pass by. Irritating.

As luck would have it, it isn’t all that difficult to make a little conditional to check which version is currently loading the config and just ignore the settings which won’t work for that version, such as:

if v:version >= 700
    set cul
endif

Links

Finally, at this years FSCONS I was introduced to the site renderfarm.fi where people can go to either contribute CPU-cycles, or get CPU-cycles, (or both) to help speed up rendering.

:wq

2011w44

Sunday, November 6th, 2011

FSCONS

T minus 5 days and counting :D

It’s gonna be great to see all the familiar faces again and hopefully get to know some new people as well.

(h)top

top (and it’s more colorful sibling htop) can be a great tool for keeping track of processes, and to some extent users, in a system. If, for instance, you’re sharing a system with a couple other people, and the systems starts feeling sluggish, top will help you out by displaying the process id (and the username which started the process) of the process consuming the most CPU cycles.

If you’d rather sort on the processes consuming the most system memory that would be On<ESC> (i.e. “<shift>o” followed by “n” followed by <ESC>)
Note that this is done from inside a running top.

You could even single out a specific user for monitoring with

$ top -u <username>

bc

This one took me a while to figure out… I had this problem during the week, which I for the life of me cannot remember now, but it manifested itself in that my attempts to do

$ echo "$part / $total" | bc

didn’t do the trick as it just truncated the value down to 0.

And the solution to this, which I found at Linux By Examples was rather simply to add a scale command to bc:

$ echo "scale=3; $part / $total" | bc

Perl

Named capture groups are great (I already knew of them from Python (Django, urls.py) but they took me quite some time to hunt down in Perl.

Links

The Geek Stuff has some really great posts about various more or less admin-oriented commands (indeed, when I was in a hurry to learn what I needed from top this week, The Geek Stuff was my primary tutor.