Archive for the ‘Documentation’ Category

2012w25

Sunday, June 24th, 2012

Quite a while since I wrote a post now, I’ve not been sick or anything, but there has been a lot of work abound, and outside work I prioritized sleeping over writing. But now I’m back for the moment, so let’s get down to business :)

Since last time I’ve come up with new ways of abusing awk, such as having it find the highest value from a command outputting in the following syntax:

\t<characters, integers, fullstop>: <integer>\n

To make it a little more different, the command also spits out a header, as well as an additional newline after the end of output.

I just now, while writing this, came up with a different solution, which doesn’t use awk:

theCommand | grep -v '^[^ \t]\+' | tr -d ' ' | cut -d':' -f2 | sort -r | head -n 1

but what I ended up using was:

theCommand | awk 'BEGIN { highest = 0 } $0 ~ /^[ \t]/ { if ( $2 > highest ) { highest = $2 } } END { print highest }'

In this case, from what I can gather, awk is the more efficient solution. One process versus five.

Update: As Werner points out, the if statement isn’t really necessary (which also makes it possible to cut out the BEGIN statement as well):

theCommand | awk '/^[^ \t]/ && $2 > highest { highest = $2 } END { printf "%d\n", highest }'

Utilities

  • ditaa (a.k.a DIagrams Through Ascii Art) makes it easy to generate nice-looking diagram images from… rather nice-looking ASCII diagrams
  • docopt, a command-line interface description language, which also seems to support generating the parser for the CLI being described
  • Peity for generating different types of charts using jQuery and <canvas>
  • Ghost.py interacting with web pages, programmatically

As of late I have been thinking a great deal about backups and the project which seems the most interesting to me is Duplicity.

Random tech stuff

Other random not-so-techy stuff

What I pass for humour

:wq

2012w21

Sunday, May 27th, 2012

Since I always try to end things on a positive note, let’s start with the… hrmmm, “less positive” things first:

Before reading this thread please note that patents on trajectories probably DIDN’T kill this satellite, but come on… patentable trajectories?

UK police will start, or already has started, copying the contents of detained suspects’ cellphones, and will be keeping those contents regardless of if any charges are ever pressed…

With that out of the way, let’s check out these really cool things I found:

Eldy seems to be a nice project aimed at making computers more suitable for elderly. And it even has a tarball for installation on GNU/Linux :)

sipml5: The world’s first HTML5 SIP client.

If you have a dataset with subjective ratings, and some known preferences, could you use that to discover other brands of scotch you might enjoy?

Start sending dates the correct way, i.e. ISO8601.

MomentJS seems to be the JS library to handle all your datetime needs.

This was a pretty cool idea: 3D QR codes only accessible at certain times of day

Perhaps this should be spread around more to decrease the risk of misunderstandings?

And finally some programming-“related” things:

Programmers are optimists: This can be read as a quip about the deficiency in programmers with always overestimating their abilities and underestimating the problem, but the last lines paint the post in a somewhat different light ;)

Funny quotes about programming languages :D

:wq

2012w20

Sunday, May 20th, 2012

There was no summary post last week, because I was in the middle of being sick as frak. I’m better now :)

During these last two weeks I’ve had a couple of eye-openers thanks mostly to other peoples blog posts:

Although a rant it does give food for thought. Why should the database be in the centre of the system anyway?

I give the orders around here, oh wow have I gotten OOP wrong all these years… :(

I never understood how useful bash someFileContainingCommands could be until week when I had to rename a couple of files in a couple of directories and didn’t have my usual set of tools (qmv would have made this so easy), so what I ended up doing was:

ls -1 > rename.sh
vim rename.sh 
# work your :%s/// magic 
bash rename.sh 
rm rename.sh

for each of the directories. No extra step of adding a shebang, no modifying the executable bit. Just enter vim, do some regular expression search replace and execute.

SRP as applied to CSS.

There has also been a great many things written about programmers, specifically who should or shouldn’t become one:

Jeff Atwood wrote a really nice post, and while I don’t agree with everything he says he is making some good points. I do however firmly believe that there are a great many mundane tasks today, being performed manually, needlessly I might add, since with the right thinking and just a little knowledge, the tasks could be automated. Case in Point.

Anyway, Jeff’s post spawned a great many thoughtful reactions. All in all I think it was a good thing to publish that post. Lots and lots of great replies and comments.

I do personally believe that more and more of our world is being governed by digital technology, and a better everyday understanding of how programs are constructed and what the basic concepts are, could never hurt. Hell it might even make it easier to formulate in better words what is going wrong when you call tech support. (“It crashed” vs “It crashed after I instructed it to iterate over these filenames”)

If you do end up wanting to give it a shot, how should you go about it? Adjust your expectations and prepare for inevitability :)

And in any case, whatever your profession ends up being, and although I only agree with #1 and #2, you really should build something. Doing it first doesn’t matter if you do it better.

I’ll end this topic with a single word: SHUN!

I also found some cool/interesting/potentially useful stuff:

Pykka seems rather interesting, I’ve often wanted something like Erlang, but with just a tiny bit easier way to launch it and interoperate with the system. I guess now I can :D

git-playback for when you wish to visualize the changes in files over time.

Focus.py the productivity guarding firewall ;)

Compleat: Bash completion for human beings.

Last but not least, the miscellaneous category:

The Dictator’s practical internet guide to power retention.

Plenty of rather interesting ideas about gamification to increase user contribution in this thread.

I don’t know how I feel about OSHackers.org. And what’s worse is I can’t put my finger on why I don’t know what I feel about it.

Timeline of the far future, this sounds like something I’ve read on xkcd.

2012w18

Sunday, May 6th, 2012

The scripty stuff

This week I finally managed to crack a problem I’d been trying to solve for a couple of weeks, namely how to only print the foobar errors, and the ensuing stack trace of these errors from a logfile:

awk 'BEGIN { section = 0 } /foobar/ { section = 1; print; next } /^[A-Z]/ && section == 1 { section = 0; next } section == 1 { print; next }' logfile 

Looking at the solution, I am kindof ashamed that it took me that long to get a workable solution…

I also found this neat little oneliner in a comment on reddit: echo "something long and space separated of which you want the last word" | rev | cut -d ' ' -f 1 | rev. Then again, I’m sure that awk could have done this with a little $(NF-1) magic or something like that.

The headache-inducing stuff

All since my netbook broke down, I’ve thought about two things: restoring the netbook/replacing it, and how to create some form of backup infrastructure which should be better than what I have in place today.

As for the backups, the “system” I have today is couple of USB-disks which I at times plug in and sync files to. That and most of my projects and config-files are in various git repositories all synced to the laptop/server-in-the-wardrobe which I made sure to backup after the netbook died, especially since the laptop/server disk is much older than the netbook disk was.

Another thing which bothers me with the current solution is that I have no off-site storage. And that would be nice to have. Belt AND suspenders of course, and off-site storage comes with its own set of problems such as trust in the offsite storage maintainer.

I think the solution will take the shape of a GNU+Linux box and Unison and possibly aided by incron. Not sure yet, will have to think more about it.

There are some other requirements which I have just barely scratched the surface of or not even begun thinking about yet, for instance it would be nice to be able to backup my parents stuff as well on a regular basis as to keep their stuff safer as well.

And as for the netbook, although it was a nice little machine, the keyboard was getting a bit worn out, and at times it was rather underpowered with its single core 1.6GHz atom processor, so the direction I am looking in now is towards something like this.

The stuff screwing over society

Now there’s truly no way in hell I’ll ever use Skype again.

Nothing new under the sun I guess, but it lends credibility to the Skype quip above.

This sure is some level-A grade retarded society we are constructing for ourselves…

Samsung Galaxy S3: The first smartphone designed entirely by lawyers, a great read about a truly depressing matter which probably is closer to the truth than we imagine. On the other hand, my personal opinion is that the midnight blue version looks pretty damn sweet.

SaaS and other crap where someone else is in control sure is a honking good idea, isn’t? Well, I guess it is if you’re the one in control, but I guess you won’t ever get my business…

The cool stuff

And I also managed to find some posts which touched the hacker in me, such as this post about how one could go about generating pseudo-random numbers (don’t use the algorithms, just be inspired by them) or how this guy started shaving bytes off of his “hello, world!” binary.

I immediately thought about FSCONS when I read this, and I didn’t feel at all worried about people thinking the same about our conf :)

Until the other day, when I read about its inclusion into git, I’d never even heard about git subtree, but this post makes a compelling case for looking into it.

I also came across a, to me, new data structure: the XOR linked list. Now, it has a couple of drawbacks, and I don’t think I’ll find much use for it, ever, but as a concept it is a very interesting idea, and just goes to show that XOR is frakking awesome.

I thought this was a pretty cool thing.

While I don’t have any problems with my ISP hijacking DNS requests right now, it is nice to know for posterity that there are ways around it ;)

If you are going to use JSON, and need comments, this seems like a reasonable way to go about it.

While I haven’t decided what I think about Go I really liked this blog post on how to create a grass mowing agent which derives the most optimal way to cut the digital grass in a simulated world.

Hopefully I ain’t the only one who finds this hilarious ;)

This is actually quite neat: Instead of adding “lorem ipsum” paragraphs all over your design, tweak the word list in the script, include it in the mockup, and markup all places which need filler content. Done.

In the latest issue of DatorMagaZin there was an article about FUSE which caught my eye, and having read the article my interest was piqued, so I just had to go look at the list myself, and truly, have you seen all the cool filesystems people have come up with? Frakkin’ awesome!

The food for thought stuff

Oh yeah, finally remember to treat everyone the way you’d like people to treat your own mother

:wq

Pacman and cleaning out old packages

Tuesday, May 1st, 2012

Just found this out, and thought it may benefit someone else, so here you go :)

In my netbook install of Archlinux I was running out of disk space on / because the package cache (/var/cache/pacman/pkg/) was always filling up with old versions of packages. The reason for this obviously being that whenever I would upgrade a package, it wouldn’t be until next reboot I’d know if something was amiss or not, so pacman -Sc wasn’t really an option. And at next reboot, did I remember to run pacman -Sc? Of course not.

But, as it has been said before, and will be said again countless times: the arch wiki is fantastic!

Take for instance the pacman page, where it gives a hint that if you don’t really like pacman -Sc, you could try cacheclean (found in the AUR).

It takes at least one parameter, or, I guess, two at the most. The required one is a number, indicating how many previous versions you wish to keep. And on top of that you could add -p for preview, in which case it will only simulate removing the packages, and instead printing their names, so you have a chance to spot any mistake you might have made. With -v, cacheclean will perform the task, and tell you what it has done.

Since it will operate on /var/ you’ll need to execute it as root.

Simple as that. The only gotcha is that it is a python3 script, but since that is the standard in arch these days anyway, it shouldn’t make much of a difference anyhow.

:wq

2012w16

Sunday, April 22nd, 2012

I ought to dedicate this blog post to git and rsync: The hard drive on my netbook died this week. I haven’t attempted to recover anything from the disk yet, but of that which is most important I figure I haven’t lost anything at all. And that’s due in no small part to git and rsync.

All of my configuration files, at least those I care about, had been added to a git repository. And most of the binaries I wanted to preserve had been rsynced to my server.
Not all of it though, which is a shame, but it shouldn’t be hard to replace what I’ve lost. Especially if I can get the old hard drive to function just one more time, just long enough to at least make a list of what it is I’m missing. The rest of the disk, well, it’s spring, perhaps a spring cleaning was in order.

So all is not lost, and looking beyond this setback, I did learn some other things this week (except for the fact that I need to become better at performing backups) such as:

Also, quite some time ago, I went around thinking about how to automatically track my working time, and while this isn’t exactly like what I had in mind (I would probably just have created a daemon which somehow fetched the window title of the currently active window from X, and did so randomly 6 times per hour (not deterministically enough to be able to cheat the system).

And some assorted links which may or may not be of any particular use for anyone:

:wq

2012w15

Sunday, April 15th, 2012

This has been a pretty rough week, but I guess there is nothing less to expect when deadlines are drawing near.

This week I found myself wanting to count all the occurrences of “foo”, but ONLY if they occurred BEFORE “bar”:

awk 'BEGIN { fooCount=0; stopCounting=0 } /bar/ { stopCounting=1 } /foo/ && stopCounting=0 { fooCount = fooCount + 1 } END { print fooCount }' <myfile>

And despite the quite hectic schedule, I did manage to help a colleague with a little scripting, and those are two things which almost always sets me in a better mood: scripting (problem solving), and helping others (of course, if I don’t manage to be of any help, that kindof defeats any positive mood change I get from scripting, but in this particular case it all worked out really well in the end) :)

And now for the mandatory collection of links from this week:

  • This must be a joke right? The US can’t really, for real, be irritated with Australia for preferring national service providers over American ones, right? Especially when it could come down to storing data about Australian citizens, or in other ways vital to the government. This has to be a joke right?
  • I wonder if this is something most programmers can relate to or if it’s just me
  • This post could have been written by me… well, not as articulate, but the spirit of it. What’s even more interesting is the response this triggered on HackerNews.
  • QArt Codes is where QR codes, Solomon-Reed error correction, some extra calculations and your imagination mix together ;)
  • Hilarious post making fun about certain governments and their want for even more snooping laws, especially about conducting surveillance in in-game chats…

:wq

2012w12

Sunday, March 25th, 2012

Woho! Not only a great week at work, but I also managed to squeeze in some other activities as well.

Managed to execute two test cases (yes, that’s a little slow, but there were mitigating circumstances, so all in all I am not too worried, and I did learn from it so it’s all good), got a small lecture in a subsystem (which made me want to tinker with Erlang and mnesia again), went swimming after work one day, and spent Thursday evening at an OWASP event, listening to a very entertaining dude named Jim Manico

There was a change in plans, so the talk wasn’t about Web Application Access Control Design but instead about the ten most critical web application security flaws found on the “OWASP Top Ten Project”-list.

Learning interesting stuff AND having fun at the same time? Oh wait, when I put it like that it just sounds like any other (work)day in my life, but you get the point. Great event, looking forward to the next :)

And Friday evening was spent hanging out with Rikard, Zara and Alfred. All in all, a rather good week.

I did have some trouble with tmux, not all of which I managed to solve.

There were actually two issues, and only one of them have been “solved”, and I use that word pretty lightly because I don’t find the solution particularly good, although it is probably the solution.

  1. Scrolling backwards (up) in an terminal inside tmux is painful. C^b [ sets you in the mode you need to be to enable PgUp to work, but that is not nearly as easy as my muscle-memory-bound Shift-PgUp (plus this also means I need to exit that “cut-mode” or whatever it is called when I’m done scrolling
  2. tmux doesn’t seem to interact all that well with a mouse. I admit, that probably wasn’t high on the priorities, but if I don’t have vim bindings (visual-mode, yank) the mouse is by far the easiest way to copy text from a terminal. Click and hold mouse1, drag over the area to copy, release mouse1, DONE!

The mouse issue is probably easy enough to fix, I suspect I just need to read the man-page better and fiddle some more with the configuration. But I am not so sure about #1. That’s the built-in way to do it… getting something better working there is probably not straightforward at all.

Finally, this week I also “rediscovered” zodiac and I am now pondering whether or not to just use that instead of building my own “makefile blog”-type of thing. I’d need to hack it a bit, there are some things I don’t want to make do without (RSS, prev/next-links, tags) and it would be pretty neat (albeit useless) to have post signing using GPG.

:wq

2012w11

Sunday, March 18th, 2012

This week has been filled with all sorts of fun and challenging stuff.

For one, I used Wireshark to (successfully) diagnose a problem for the first time :D

The problem was an overly restrictive iptables rule on the system I was trying to connect to, but boy did it feel good to identify and solve that problem on my own.

I also managed to squeeze in time to write scripts to two work buddies so that they didn’t need to perform overly repetitive and very error-prone tasks by hand. The one script even needed a little bit of Expect-scripting, which I’ve been aching to hack some more.

I also found out about vim‘s :e # (to try it out, open a file in vim, then :e someOtherFile followed by :e #)

I found a very pedagogic explanation about how quicksort works, which made me start thinking and wondering about algorithms in general again, even to the point of investigating quicksort further.

I also found this post which makes a fair point: IFF you are going to rely on a web service (in other words, a SaaS you are probably better off getting hooked up with one charging money for it.

Not that I could ever conscionable recommend anyone using a SaaS as this is bound to lock you into dependence of that provider, but sure, I get that some (most?) would rather throw money at it than learning (learning is scary, and you don’t have the time right?) to set up your own service.

Furthermore, I can’t really decide whether the author is being disingenuous or illiterate in naming this stance an “anti-free-software movement”. It is unclear if s/he meant “anti-gratis-software movement” or if s/he is actually feeling threatened by free software. Had there been a comment field on that “blog” I probably would have spent the two minutes required to ask, but since there isn’t one, I won’t bother, and on pure principle, if I ever need a bookmarking-service, I’ll just make sure to stay clear of that one, just in case s/he actually was taking a jab against free software.

Yes, I am petty and vindictive at times, shit happens.

Whilst toying with the idea of modifying my homepage a bit, a site wide change, I realized that it might make a lot of sense to rewrite it so that it is, if not template-based, at least modularized. But I guess I’ll continue thinking about that before making any changes.

A funny thing I just thought of out of the blue the other day was that with some sites, like LinkedIn and probably Facebook and others, there are options for giving these services your email account username and password, so that they could log in, check your contact list, and hook you up with other people you know.

But I am pretty sure that most email providers license agreements state (no, I haven’t checked, but it would be foolish of these mail providers not to) that it would be a violation of said agreement for the user to disclose their account password to anyone else.

Then again, I don’t really understand people who would use those tools, but I guess there are even more people who wouldn’t understand why I chose not to…

A very interesting read about the Batman and the Joker. Those who know me can probably guess what character I voted for.

I’ve also been thinking about the prospect of storing binaries in some sort of version control, mostly because I have a friend who is into game development/modding, and doing so without version control. And that has bitten him in the ass more than once now.

And since I am nowadays a git convert I naturally started looking there, and found this, which lead me, among other places, here. Haven’t tried it yet, but it is definitively an interesting idea.

And now that I have begun working with Wireshark, tcpdump seems to be a good next step.

Finally, yet another reason why the patent industry needs to change.

:wq

2012w08

Sunday, February 26th, 2012

Hacks

A capture the flag game where the objective is to break into a computer system.

Commandline

I found myself needing to remove a couple (three) columns from a file containing about 15 columnts per line. And sure, I could have done something like awk '{ print $1 " " $2 " " $3 " " }' for the 12 columns I wanted, but that would have been tedious.

There just had to be a better way. And of course there was ;)

* * * * * *

I’ve been entertaining an idea which would need version controlled updates, and they’d also need to be trusted. So I’d need signed commits, and since I’m mostly using git nowadays, I needed to find out if this was possible. It is.

* * * * * *

Since starting my new job I’ve realized just how important it can be to write portable scripts (especially echo has bitten me in the ass a couple of times already) so this post was pretty useful to me.

Society

Now this was a pretty inspiring post.

* * * * * *

A pretty funny post about how truly sorry a state the TV is in.