Archive for the ‘Tools’ Category

2012w3

Sunday, January 22nd, 2012

mitmproxy

Granted, there shouldn’t be all that many use cases for a software such as this for a non-pentesting, non-criminal, but the fact that it can record and replay previous interactions, which can be useful, for instance, to automate login on access protected networks (hopefully ones that we already have permission to enter, but find the actual logging in part a real hassle).

Procedural City, Part 1

I thoroughly enjoyed reading this entire series of blog posts, in which the author had the “simple” goal of generating an entire city, digitally, in a procedural manner.

Fedora Friendfinder

Ok, so this is just humor, but you know what? It is good humor :)

XXXTerm

This sounds like something kinky, but is in reality a minimalist web browser with sophisticated security features designed-in.

So, a bit like luakit but with a funnier name ;)

Pipeviewer

pipeviewer really is something I could have more use for, if I just ever remebered to use it ;)

The next SOPA

You know what? This guy is on the right track. And I think Joel is as well.

MPAA shows us just why they are not a part of the solution. Mostly it is because they, and the rest of the abusive copyright-holders are their own worst enemy. Of course, they won’t go down peacefully, so it really is time for us to start fighting back. And there are plenty of targets to chose from.

In related news, it seems the Polish internet community is “unhappy” with ACTA… very nice :)

2012w02

Sunday, January 15th, 2012

Update: Corrected link, thanks Ulf

This has been quite the busy week. Oodles and oodles of stuff happening, both nice and… less nice.

UEFI + SecureBoot

Microsoft up to no good again. Basically, on non-ARM systems Microsoft requires that a user can disable SecureBoot, but not on ARM systems (i.e. smartphones, tablets, and the upcoming ultrabooks). Anyone surprised?

Music Production

While I have no real interest in producing music myself—code and, to some extent, graphics have always come easier to me—I do have an interest in seeing tools like this come to GNU+Linux as well, since it means that’s one less category of creators not having the alternative to be creative in a free software environment :)

sshuttle

This project seems pretty cool, I haven’t tried it out yet, and the thing about uploading code to the server is something I’ll definitively look into before actually considering executing it, but all in all this looks like a pretty easy SSH tunneling/VPN mimicing proxy thingy solution which could be useful at times.

Especially if it means I can sit at an internet café or some such, and have all my traffic routed över SSH through my server at home, not having to worry about someone in that café sniffing it up.

tmux

I installed a local copy of tmux at work, and so far it has been a complement rather than a detriment to the way I work.

The one thing that I wasn’t pleased with at first, but which was trivially easy to fix, once I read a blogpost (also, don’t miss the second post), was that I wanted 11 shells all stacked on top / below eachother, with an even size (i.e. each should take up 1/11th of the tmux window height.

When splitting the window, to make room for another shell, it just divides the current shell height by two, and makes the one part the new shell, and the other part the old shell. For multiples of two I suppose this would work out fine, but with 11 shells?

So I went about it, and the tenth and eleventh shell were small.
But there are different preconfigured layouts, and you loop through them by repeatedly hitting the control sequence (I’ve mapped this to C^a) and space. One of those layouts proved to be just what I wanted :)

Raspberry Pi

Now this is a pretty cool project! For the condensed summary, read the wikipedia page. It is making the dream of a $100 computer a reality, and there are some pretty cool ideas already about how to put it to good use.

SOPA

Reddit doesn’t like SOPA, and Tim O’Reilly isn’t all that pleased either.

If you’re an Android user, and you don’t like SOPA either, there is an app for letting you know (by scanning barcodes) if a product is made by a pro-SOPA company so you can avoid supporting them.

There have been some advances which means that making a fuzz about it can pay off. Of course, it would be better to scrap those bills completely.

So, 2012-01-18 is still SOPA Blackout day and a whole lot of sites are participating, and you could join in as well (and if you want to join in, please be smart about it and host the javascript your own damn self so that the hosting server doesn’t go down… (which also means, get that javascript now, and not on tuesday evening when everyone else is going to try to get it))

Links

  • Privacy in social networks — not sure I understand how it is done, not sure that this implementation is optimal, but nice idea none the less
  • I read a post the other day, and the author of that post, while being in the right, just came off … I don’t know, but his post was a rant, and not the passionate kind, but the whiney kind, so I won’t be linking to his post, I have no wish to drive traffic to him. However, another person, with reasoning and values more aligned to my own, wrote a reaction post to his, which I feel was more constructive, and nicer, so here is the link to that post
  • Unfair advantages grow from irrational habits
  • Rikard tipped me off to a thoughtful TED talk video, which I liked alot, and through the speakers website I found, among others, this game—EVOKE—which seems to be pretty cool
  • I had the idea of building an image gallery a while ago, so when I came across this link I was a little interested in seeing how they’d approached it, but what I really took away from this site, is how much I liked their rather user-friendly step-by-step manual for getting it up and running
  • I wonder what he will create? :)
  • I don’t know if it’s just me, but non-flashy, low-requirements games make me all warm and fuzzy inside
  • I seem to recall that I wasn’t all that impressed with the unhosted project some time ago. This post (specifically the verification section) is exactly why I hesitate

2012w01

Sunday, January 8th, 2012

column

The other day I wanted some prettier (tabularized) output and of course someone has already wanted this and of course there are tools for that :)

bash_completion

This is so frakking cool! I’ve built this little shellscript “vault.sh” which is a simple wrapper script for mounting and unmounting encfs mounts.

It takes two parameters: operation and target, where operation can be one of “lock” and “unlock”, and target—at present—resolves to “thunderbird” (signifying my .thunderbird directory).

Since I intend to expand this with more encrypted directories as I see fit, I don’t want to hard-code that.

What I did want, however, was to be able to auto complete operation and target. So I looked around, and found this post, and although I couldn’t derive enough knowledge from it to solve my particular problem, having multiple levels of completion, the author was gracious enough to provide references to where s/he had found the knowledge (here, here and here). That second link was what did it for me.

My /etc/bash_completion.d/vault.sh now looks like this:

_vault()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    first="lock unlock"
    second="thunderbird"

    if [[ ${cur} == * && ${COMP_CWORD} -eq 2 ]] ; then
        COMPREPLY=( $(compgen -W "${second}" -- ${cur}) )
        return 0
    fi

    if [[ ${cur} == * && ${COMP_CWORD} -eq 1 ]] ; then
        COMPREPLY=( $(compgen -W "${first}" -- ${cur}) )
        return 0
    fi
}
complete -F _vault vault.sh

And all the magic is happening in the two if-statements. Essentially: if current word (presently half typed and tabbed) is whatever, and this is the second argument to the command, respond with suggestions taken from the variable $second.

Otherwise, if current word is whatever, and this is the first parameter, take suggestions from the variable $first.

Awsum!

awk for great good

Another great use for awk: viewing selected portions of source code. For instance, in Perl, if you just want to view a specific subroutine, without getting distracted by all the other crud, you could do: $ awk '/sub SomeSubName/,/}/' somePerlModule.pm

Links

If PHP were British, perhaps it’s just me, but I find it hilarious.

PayPal just keeps working their charm…

Belarus just… wait what?

Why we need version control

Preserving space, neat!

Fuzzy string matching in Python

If you aren’t embarrassed by v1.0 you didn’t release it early enough

The makers schedule, oldie but goldie

CSS Media Queries are pretty cool

Static site generator using the shell and awk

A netstat companion

Reducing code nesting

Comparing images using perceptual hashes

Microsofts GPS “avoid ghetto” routing algorithm patent…

2011w52

Sunday, January 1st, 2012

Merry belated christmas greetings everyone! And by the time this post is published I could extend it with Happy belated new years greetings as well ;)

vim + html5 syntax

I’ve been tinkering a lot with html5 during my vacation and vim just didn’t want to play nicely with the new html-tags.

Namely, as it wouldn’t recognise the new semantic structural tags (footer, header, article, section, nav, aside) it wouldn’t indent the source properly and it was a cause for both distraction, and the resulting frustration.

I was not the first to feel this frustration, and a quick search turned up this result which solved both the html and css syntax issues (check the comments for the css solution). Very elegant solution, and now I’ve also learned about vim’s .vim/after/ directory… That was pretty cool.

Learning html5

I’ve actually shied away from doing stuff with html5, as whenever I tried to wrap my head around the new tags and how they should be used, there were just a myriad of different sites interpreting the usage in subtle but differing ways, but I finally found a resource which makes sense to me, so until a definitive interpretation has been hammered out, that’s the one I’m going to stick with.

Also, for sticky footers using css, and html5, check out this page. I had no trouble getting that to work.

Links

This question pretty much sums up why I like the command line so much

This looks interesting for synching (and deleting) without having to worry about doing “the right thing”

Nice list of things one could do with a home server

Doing it for teh lulz, 1903 style

EA, Nintendo and Sony now only covertly support SOPA (through their membership in various interest organizations). Wanting to eat the cake and still have it huh?

Tom’s Hardware not being amused by SOPA

Oh how I so hope that Wikipedia, Google, et al, will go down this path. (I do think there is a difference between companies lobbying, writing laws, and pressuring governments, and companies urging people to put pressure on governments, so yes, I think this is ok)

An interesting theory about why cinemas are having such a rough time

Haven’t had a chance to try this, but creating art using a written grammar does sound pretty neat, especially if you could get a script and /dev/random involved as well ;)

German police tracking people via silent SMS. I am beginning to think that rms is correct in his cellphone “usage”

Too much reading and constant information overload makes us pretty little passive consumers

My Software Stack 2011 edition

Saturday, December 31st, 2011

I realize that I haven’t written my customary “software stack” post for this year yet. But hey, from where I’m sitting, I still have … 36 minutes to spare ;)

I’ll be using the same categories as last year; system, communications, web, development, office suite, server, organization, and entertainment.

System

The OS of choice is still Archlinux, my window manager is still wmii, my terminal emulator is rxvt-unicode, upgraded by also installing urxvt-tabbedex.

My shell is still bash, my cron daemon is still fcron, and my network manager is wicd.

To this configuration I’ve added the terminal multiplexer tmux, and have lately found out just how useful mc can be. Oh, and qmv from the renameutils package is now a given part of the stack.

Communications

Not much change here, Thunderbird for email, Pidgin for instant messaging, irssi for IRC.

Heybuddy has been replaced by identicurse as my micro-blogging (identi.ca) client. Heybuddy is very nice, but I can use identicurse from the commandline, and it has vim-like bindings.

For Pidgin I use OTR to encrypt conversations. For Thunderbird I use the enigmail addon along with GnuPG.

This means that Thunderbird still hasn’t been replaced by the “mutt-stack” (mutt, msmtp, offlineimap and mairix) and this is mostly due to me not having the energy to learn how to configure mutt.

I also considered trying to replace Pidgin with irssi and bitlbee but Pidgin + OTR works so well, and I have no idea about how well OTR works with bitlbee/irssi (well, actually, I’ve found irssi + OTR to be flaky at best.

Web

Not much changed here either, Firefox dominates, and I haven’t looked further into uzbl although that is still on the TODO list, for some day.

I do some times also use w3m, elinks, wget, curl and perl-libwww.

My Firefox is customized with NoScript, RequestPolicy, some other stuff, and Pentadactyl.

Privoxy is nowadays also part of the loadout, to filter out ads and other undesirable web “resources”.

Development

In this category there has actually been some changes:

  • gvim has been completely dropped
  • eclipse has been dropped, using vim instead
  • mercurial has been replaced by git

Thanks in no small part to my job, I have gotten more intimate knowledge of awk and expect, as well as beginning to learn Perl.

I still do some Python hacking, a whole lot of shell scripting, and for many of these hacks, SQLite is a faithful companion.

Doh! I completely forgot that I’ve been dabbling around with Erlang as well, and that mscgen has been immensely helpful in helping me visualize communication paths between various modules.

“Office suite”

I still use LaTeX for PDF creation (sorry hook, still haven’t gotten around to checking out ConTeXt), I haven’t really used sc at all, it was just too hard to learn the controls, and I had too few spreadsheets in need of creating. I use qalculate almost on a weekly basis, but for shell scripts I’ve started using bc instead.

A potential replacement for sc could be teapot, but again, I usually don’t create spreadsheets…

Server

Since I’ve dropped mercurial, and since the mercurial-server package suddenly stopped working after a system update, I couldn’t be bothered to fix it, and it is now dropped.

screen and irssi is of course always a winning combination.

nginx and uwsgi has not been used to any extent, I haven’t tried setting up a VPN service, but I have a couple of ideas for the coming year (mumble, some VPN service, some nginx + Python/Perl thingies, bitlbee) and maybe replace the Ubuntu installation with Debian.

Organization

I still use both vimwiki and vim outliner, and my Important Dates Notifier script.

Still no TaskJuggler, and I haven’t gotten much use out of abook.

remind has completely replaced when, while I haven’t gotten any use what so ever out of wyrd.

Entertainment

For consuming stuff I use evince (PDF), mplayer (video), while for music, moc has had to step down from the throne, to leave place for mpd and ncmpcpp.

eog along with gthumb (replacing geeqie) handles viewing images.

For manipulation/creation needs I use LaTeX, or possibly Scribus, ffmpeg, audacity, imagemagick, inkscape, and gimp.

Bonus: Security

I thought I’d add another category, security, since I finally have something worthwhile to report here.

I’ve begun encrypting selected parts of my hard drive (mostly my email directory) using EncFS, and I use my passtore script for password management.

And sometimes (this was mostly relevant for when debugging passtore after having begun actively using it) when I have a sensitive file which I for a session need to store on the hard drive, in clear text, I use quixand to create an encrypted directory with a session key only stored in RAM. So once the session has ended, there is little chance of retrieving the key and decrypting the encrypted directory.

Ending notes

That’s about it. Some new stuff, mostly old stuff, only a few things getting kicked off the list. My stack is pretty stable for now. I wonder what cool stuff I will find in 2012 :D

:wq

2011w51

Sunday, December 25th, 2011

Bash variable string operators

I had a file filled with URLs to files I needed to download. Some of the files on the list, however, had already been downloaded, so no need to do it all again.

Should be fairly easy, right? cat the file to a while loop, reading the lines one by one, extracting the filename from the URL, check that it isn’t existing already, and if it isn’t, download it with wget.

So… how do you go about extracting the filename? You could certainly use sed and store the extracted filename in a separate variable, but that seems kindof wasteful, especially in a one-liner while loop. This article provided me with another option.

${line##*/} which deletes the longest possible match from the left (which in this case means up to (including) the last “/”) i.e. everything up to the name of the file.

No can haz censorship plz

If you’d like to make it clear that you too oppose SOPA (which, fittingly, means “garbage” in Swedish) then head over to Github, pick up your very own copy of stopcensorship.js, embed it on your site, and you’re set :)

I am also noting, with some glee, that GoDaddy is catching a whole lot of flak for their support of SOPA.

The only thing companies truly understand is when you hit them where it hurts, and that is their wallets (or as some brilliant person jokingly expressed it: “stop hitting us in our quarterly reports!”), and the only way to do that, is by voting with your own wallet.

I’m so happy about the fact that more and more people are catching on to this realization that I could… shit rainbows :)

Japanese Whaling + Tsunami disaster relief funds = disgusting

Just when I didn’t believe it possible for the Japanese whaling industry to appear as bigger scumbags than they already appear (yes, it is a quite one-sided story we’re getting from “Whale Wars” but according to National Geographic, the whalers have gotten the chance to tell their side of the story, and it would seem likely that they decline because they know full well just what type of scumbags they are… but hey, that’s just my opinion…) they go and do even more disgusting stuff, like using money from the tsunami relief donations to hire security ships to keep the Sea Shepherd Conservation Society away from their dirty business…

:wq

2011w50

Sunday, December 18th, 2011

tmux

tmux is a terminal multiplexer, resembling screen and seemingly straight-forward to configure.

Now, those of you paying attention will know that I use wmii, a tiling window manager, and you may ask what the difference is between creating one big tmux window and laying out a couple of terminals in that, or letting wmii place those terminals beside each other itself.

The answer is that for most instances, wmii will be enough, but just a little while ago I discovered a killer feature (one which makes me wish that tmux was available at work), namely the abililty to perform:

C^b:setw synchronize-panes

(demonstrated here) which simply outputs whatever you type into one of the terminals, into all the other terminals in this tmux instance as well.

How is this useful? If you have a couple of servers, on which you need to execute the exact same command, you simply start tmux, create a terminal for each server (and log in to that server) and then ask tmux to synchronize the panes, and then you type in your commands.

(Yes, this could probably be easily solved with a bash for-loop as well, depending on the amount of commands and their complexity)

Stupid Shell Tricks

I’ve known about ^foo^bar for a while (i.e. you type
$ some-command wif a typo
and you then do
$ ^wif^with
to have the shell replace the first instance of that typo with the correct spelling (hopefully ;D)

But, this is really only good for typos or when there is ONE instance to replace. ^foo^bar won’t replace EVERY foo with bar, only the first occurrance. Which is sometimes now what you wanted.

Enter !!:gs/foo/bar which replaces ALL instances of foo in the previous command, with bar, and re-executes it. Thanks to http://blog.urfix.com/25-linux-commands/ for that.

less

I think I have touched upon this before, but here we go anyway: it is possible to export an environment variable called “LESS” and less will read this and determine any runtime special behaviour based on the contents of the variable.

I am currently trying out export LESS='FiX' where F makes less exit if the contents are short enough to all fit on the screen, i is for case-insensitive search and X for stopping less from sending the termcap initialization and deinitialization strings.

This means that when less exits, it won’t clear the screen (which would be a bummer if using F and less:ing short files…)

Links

A pretty interesting read about how one could “work in the cloud.” I would have chosen other hardware/software (except for vim of course) but to all his/her own, right?

This sounds as if it could be useful for making sure that your logs are really your real logs. Makes sense, right? ;)

From the reptyr readme: reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don’t want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

I am apparantly not the only one to get the idea of describing their software stack.

A pretty cool more-utils command, ifne, which continues execution of the rest of the command, iff data was coming into ifne’s stdin.

:wq

awk, filtering and counting

Monday, December 5th, 2011

Suppose that you have a file containing some structured data, something perhaps along the lines of this, highly fictive but yet remarkably common, syntax:

<id><separator><somestring><separator><integer>

Now, let’s say that there were 99999 lines of this to go through, and the file is unsorted, and you wanted to find all the lines where SOMESTRING is foo, and then sum up the INTEGER field of those lines.

I almost had this problem at work, except my file probably didn’t contain more than a hundred or so lines.

For this I wrote a Perl script, which worked well, with the small inconvenience that I’d have to move that script onto each system where I’d want to use it.

Pontus, never the one to berate anyones efforts, but still finding room for improvements, both in the fact that my approach, the script, carried that inconvenience, and that is was very verbose when compared to the solution he ultimately suggested, he showed me a better way, the awk way.

$ awk -F<separatorGoesHere> 'BEGIN { SUM = 0 } /<someStringGoesHERE/ { SUM += $3 } END { print SUM }' <fileToBeParsedGoesHere>

I said before that my real file, at work, was small, so awk crunched through it at lightning speed. I also suggested a file containing 99.999 lines, and I did that to prove a point, namely:

Using this script:

#!/usr/bin/env python2

import random

filename = "awk.example.txt"
index = 0
iterations = 100000
choices = ['foo', 'bar', 'baz']
fh = open(filename, 'w')

for index in range(1, iterations):
    fh.write("%d, %s, %d\n" % (index,
                               random.choice(choices),
                               random.randint(0, 100)))
fh.close()

I generated a file (~1.5Mb) with a couple of lines ;) and let awk loose on it:

$ time awk -F, 'BEGIN { SUM = 0 } /foo/ { SUM += $3 } END { print SUM }' awk.example.txt

Which on my netbook took 0.241 seconds to complete.

real	0m0.241s
user	0m0.237s
sys	0m0.000s

Or in other words: awk if pretty frakking fast!

Now, let’s break it down:

awk

obviously, is the command, and it rocks, ‘nuf said.

-F,

means “change the field separator (from whitespace) to commas”

And then it gets tricky, but not as tricky as at least I was lead to believe.

There are two single-quotes, and between these we place all the things we want awk to do for us.

One good thing to note is that the syntax for awk is quite simple, something I didn’t grasp at first. It goes like this:

<somePattern> { <someAction> }

And that’s it. You can chain several <pattern>{<actions>} after each other.

In my, well Pontus’, command above, there are three such pairs:

BEGIN { SUM = 0 }

which is just another way of saying “before we start executing, create a variable SUM and set its value to 0″

/foo/ { SUM += $3 }

If you’re familiar with regular expressions you might have stumbled upon the pattern in which you enclose an expression between two slashes, and that pattern is used to search (or match) contents of lines or files. That’s what we’re doing here. So we’re basically saying “find lines containing foo, and from these lines extract column number three ($3), and increment the variable SUM by the value stored in column three.”

If instead, you’d wanted to count all the lines containing foo, SUM += 1 would have done that job.

Finally:

END { print SUM }

which should be pretty obvious: “When all is said and done, print whatever is stored in the variable SUM”

And last but not least, outside the single-quotes, we give awk the name of the file we wish it to process.

This is just a fantastic tool which I regret not having taking the time to learn the basics of earlier. Thank you Pontus for making me see the light (again) ;)

:wq

2011w48

Sunday, December 4th, 2011

Where the frakk did this week go?!?!

Work has been progressing, I can’t say that I am good at it yet, but I am better than I was just last week, which is thoroughly encouraging :)

Pontus made me realize that knowing sed is not enough, for some things you really need awk. Another thing to push to the toLearn stack…

I’ve been doing some more Perl hackery, but nothing worth showing, but I did however come across a site which I believe to be rather good and helpful regarding learning basic things about Perl.

Something which passed me by completely was that this Monday saw the release of YaCy 1.0 (in German), but as you can see on Alessandro’s blog I might have been just about the only one who didn’t get that particular news item. Congratulations on making version 1.0 guys!

I was also toying with the idea the other day of making quarterly summaries as well. One blog post a week is great as it forces me to write, thus improving my writing, but it doesn’t really do anything for discerning trends, or changes in the way I work. This could be interesting :)

Finally, I should really start planning for writing my yearly “technology stack” post by diffing what I used back then and what I’m using now.

I am already certain that I’ve disappointed myself in some aspects, and surprised myself in others…

:wq

2011w47

Sunday, November 27th, 2011

top

So I have been playing around some more with top, and I have to say that I no longer feel any reason to install htop.

Perhaps if I dig into the manpage of htop, I’ll yet again revert to thinking it is better, but for now there’s no need.

I can get coloring (z), I can filter on users (u<username><enter>), I can control how many processes I list (n<int><enter>), and I can have the current sort field highlighted (x), and when I am happy with the configuration, W lets me save it to ${HOME}/.toprc

grep

Pontus showed me a new shiny flag for grep the other day: -s which, to quote the grep manpage, says Suppress error messages about nonexistent or unreadable files.

And this is awesome for when your are doing directory-wide recursive greps in places where you might not have the credentials to look through all the files.

Beware though as there are some differences between GNU grep and UNIX grep.

RabbitMQ

I’ve many times read about RabbitMQ and how that is good to know and if you don’t know what it is you’ve been hiding under a rock (apparantly I have), because it wasn’t until this week I actually found a blogpost that could adequately explain to me what it is and what it’s good for.

And thanks to that blogpost I now have yet one more thing pushed onto the “toLearn” stack…

blockdiag

This next thing I found is a more or less graphviz, wrapped around a python(2) module which helps create block diagrams.

There are actually four modules, blockdiag, seqdiag, actdiag, and finally nwdiag, and I could imagine all four having their use under certain circumstances.

Links

GNU source highlight — For most of your sourcecode highlighting needs