Archive for the ‘Development’ Category

2012w25

Sunday, June 24th, 2012

Quite a while since I wrote a post now, I’ve not been sick or anything, but there has been a lot of work abound, and outside work I prioritized sleeping over writing. But now I’m back for the moment, so let’s get down to business :)

Since last time I’ve come up with new ways of abusing awk, such as having it find the highest value from a command outputting in the following syntax:

\t<characters, integers, fullstop>: <integer>\n

To make it a little more different, the command also spits out a header, as well as an additional newline after the end of output.

I just now, while writing this, came up with a different solution, which doesn’t use awk:

theCommand | grep -v '^[^ \t]\+' | tr -d ' ' | cut -d':' -f2 | sort -r | head -n 1

but what I ended up using was:

theCommand | awk 'BEGIN { highest = 0 } $0 ~ /^[ \t]/ { if ( $2 > highest ) { highest = $2 } } END { print highest }'

In this case, from what I can gather, awk is the more efficient solution. One process versus five.

Update: As Werner points out, the if statement isn’t really necessary (which also makes it possible to cut out the BEGIN statement as well):

theCommand | awk '/^[^ \t]/ && $2 > highest { highest = $2 } END { printf "%d\n", highest }'

Utilities

  • ditaa (a.k.a DIagrams Through Ascii Art) makes it easy to generate nice-looking diagram images from… rather nice-looking ASCII diagrams
  • docopt, a command-line interface description language, which also seems to support generating the parser for the CLI being described
  • Peity for generating different types of charts using jQuery and <canvas>
  • Ghost.py interacting with web pages, programmatically

As of late I have been thinking a great deal about backups and the project which seems the most interesting to me is Duplicity.

Random tech stuff

Other random not-so-techy stuff

What I pass for humour

:wq

2012w21

Sunday, May 27th, 2012

Since I always try to end things on a positive note, let’s start with the… hrmmm, “less positive” things first:

Before reading this thread please note that patents on trajectories probably DIDN’T kill this satellite, but come on… patentable trajectories?

UK police will start, or already has started, copying the contents of detained suspects’ cellphones, and will be keeping those contents regardless of if any charges are ever pressed…

With that out of the way, let’s check out these really cool things I found:

Eldy seems to be a nice project aimed at making computers more suitable for elderly. And it even has a tarball for installation on GNU/Linux :)

sipml5: The world’s first HTML5 SIP client.

If you have a dataset with subjective ratings, and some known preferences, could you use that to discover other brands of scotch you might enjoy?

Start sending dates the correct way, i.e. ISO8601.

MomentJS seems to be the JS library to handle all your datetime needs.

This was a pretty cool idea: 3D QR codes only accessible at certain times of day

Perhaps this should be spread around more to decrease the risk of misunderstandings?

And finally some programming-”related” things:

Programmers are optimists: This can be read as a quip about the deficiency in programmers with always overestimating their abilities and underestimating the problem, but the last lines paint the post in a somewhat different light ;)

Funny quotes about programming languages :D

:wq

2012w20

Sunday, May 20th, 2012

There was no summary post last week, because I was in the middle of being sick as frak. I’m better now :)

During these last two weeks I’ve had a couple of eye-openers thanks mostly to other peoples blog posts:

Although a rant it does give food for thought. Why should the database be in the centre of the system anyway?

I give the orders around here, oh wow have I gotten OOP wrong all these years… :(

I never understood how useful bash someFileContainingCommands could be until week when I had to rename a couple of files in a couple of directories and didn’t have my usual set of tools (qmv would have made this so easy), so what I ended up doing was:

ls -1 > rename.sh
vim rename.sh 
# work your :%s/// magic 
bash rename.sh 
rm rename.sh

for each of the directories. No extra step of adding a shebang, no modifying the executable bit. Just enter vim, do some regular expression search replace and execute.

SRP as applied to CSS.

There has also been a great many things written about programmers, specifically who should or shouldn’t become one:

Jeff Atwood wrote a really nice post, and while I don’t agree with everything he says he is making some good points. I do however firmly believe that there are a great many mundane tasks today, being performed manually, needlessly I might add, since with the right thinking and just a little knowledge, the tasks could be automated. Case in Point.

Anyway, Jeff’s post spawned a great many thoughtful reactions. All in all I think it was a good thing to publish that post. Lots and lots of great replies and comments.

I do personally believe that more and more of our world is being governed by digital technology, and a better everyday understanding of how programs are constructed and what the basic concepts are, could never hurt. Hell it might even make it easier to formulate in better words what is going wrong when you call tech support. (“It crashed” vs “It crashed after I instructed it to iterate over these filenames”)

If you do end up wanting to give it a shot, how should you go about it? Adjust your expectations and prepare for inevitability :)

And in any case, whatever your profession ends up being, and although I only agree with #1 and #2, you really should build something. Doing it first doesn’t matter if you do it better.

I’ll end this topic with a single word: SHUN!

I also found some cool/interesting/potentially useful stuff:

Pykka seems rather interesting, I’ve often wanted something like Erlang, but with just a tiny bit easier way to launch it and interoperate with the system. I guess now I can :D

git-playback for when you wish to visualize the changes in files over time.

Focus.py the productivity guarding firewall ;)

Compleat: Bash completion for human beings.

Last but not least, the miscellaneous category:

The Dictator’s practical internet guide to power retention.

Plenty of rather interesting ideas about gamification to increase user contribution in this thread.

I don’t know how I feel about OSHackers.org. And what’s worse is I can’t put my finger on why I don’t know what I feel about it.

Timeline of the far future, this sounds like something I’ve read on xkcd.

2012w18

Sunday, May 6th, 2012

The scripty stuff

This week I finally managed to crack a problem I’d been trying to solve for a couple of weeks, namely how to only print the foobar errors, and the ensuing stack trace of these errors from a logfile:

awk 'BEGIN { section = 0 } /foobar/ { section = 1; print; next } /^[A-Z]/ && section == 1 { section = 0; next } section == 1 { print; next }' logfile 

Looking at the solution, I am kindof ashamed that it took me that long to get a workable solution…

I also found this neat little oneliner in a comment on reddit: echo "something long and space separated of which you want the last word" | rev | cut -d ' ' -f 1 | rev. Then again, I’m sure that awk could have done this with a little $(NF-1) magic or something like that.

The headache-inducing stuff

All since my netbook broke down, I’ve thought about two things: restoring the netbook/replacing it, and how to create some form of backup infrastructure which should be better than what I have in place today.

As for the backups, the “system” I have today is couple of USB-disks which I at times plug in and sync files to. That and most of my projects and config-files are in various git repositories all synced to the laptop/server-in-the-wardrobe which I made sure to backup after the netbook died, especially since the laptop/server disk is much older than the netbook disk was.

Another thing which bothers me with the current solution is that I have no off-site storage. And that would be nice to have. Belt AND suspenders of course, and off-site storage comes with its own set of problems such as trust in the offsite storage maintainer.

I think the solution will take the shape of a GNU+Linux box and Unison and possibly aided by incron. Not sure yet, will have to think more about it.

There are some other requirements which I have just barely scratched the surface of or not even begun thinking about yet, for instance it would be nice to be able to backup my parents stuff as well on a regular basis as to keep their stuff safer as well.

And as for the netbook, although it was a nice little machine, the keyboard was getting a bit worn out, and at times it was rather underpowered with its single core 1.6GHz atom processor, so the direction I am looking in now is towards something like this.

The stuff screwing over society

Now there’s truly no way in hell I’ll ever use Skype again.

Nothing new under the sun I guess, but it lends credibility to the Skype quip above.

This sure is some level-A grade retarded society we are constructing for ourselves…

Samsung Galaxy S3: The first smartphone designed entirely by lawyers, a great read about a truly depressing matter which probably is closer to the truth than we imagine. On the other hand, my personal opinion is that the midnight blue version looks pretty damn sweet.

SaaS and other crap where someone else is in control sure is a honking good idea, isn’t? Well, I guess it is if you’re the one in control, but I guess you won’t ever get my business…

The cool stuff

And I also managed to find some posts which touched the hacker in me, such as this post about how one could go about generating pseudo-random numbers (don’t use the algorithms, just be inspired by them) or how this guy started shaving bytes off of his “hello, world!” binary.

I immediately thought about FSCONS when I read this, and I didn’t feel at all worried about people thinking the same about our conf :)

Until the other day, when I read about its inclusion into git, I’d never even heard about git subtree, but this post makes a compelling case for looking into it.

I also came across a, to me, new data structure: the XOR linked list. Now, it has a couple of drawbacks, and I don’t think I’ll find much use for it, ever, but as a concept it is a very interesting idea, and just goes to show that XOR is frakking awesome.

I thought this was a pretty cool thing.

While I don’t have any problems with my ISP hijacking DNS requests right now, it is nice to know for posterity that there are ways around it ;)

If you are going to use JSON, and need comments, this seems like a reasonable way to go about it.

While I haven’t decided what I think about Go I really liked this blog post on how to create a grass mowing agent which derives the most optimal way to cut the digital grass in a simulated world.

Hopefully I ain’t the only one who finds this hilarious ;)

This is actually quite neat: Instead of adding “lorem ipsum” paragraphs all over your design, tweak the word list in the script, include it in the mockup, and markup all places which need filler content. Done.

In the latest issue of DatorMagaZin there was an article about FUSE which caught my eye, and having read the article my interest was piqued, so I just had to go look at the list myself, and truly, have you seen all the cool filesystems people have come up with? Frakkin’ awesome!

The food for thought stuff

Oh yeah, finally remember to treat everyone the way you’d like people to treat your own mother

:wq

Pacman and cleaning out old packages

Tuesday, May 1st, 2012

Just found this out, and thought it may benefit someone else, so here you go :)

In my netbook install of Archlinux I was running out of disk space on / because the package cache (/var/cache/pacman/pkg/) was always filling up with old versions of packages. The reason for this obviously being that whenever I would upgrade a package, it wouldn’t be until next reboot I’d know if something was amiss or not, so pacman -Sc wasn’t really an option. And at next reboot, did I remember to run pacman -Sc? Of course not.

But, as it has been said before, and will be said again countless times: the arch wiki is fantastic!

Take for instance the pacman page, where it gives a hint that if you don’t really like pacman -Sc, you could try cacheclean (found in the AUR).

It takes at least one parameter, or, I guess, two at the most. The required one is a number, indicating how many previous versions you wish to keep. And on top of that you could add -p for preview, in which case it will only simulate removing the packages, and instead printing their names, so you have a chance to spot any mistake you might have made. With -v, cacheclean will perform the task, and tell you what it has done.

Since it will operate on /var/ you’ll need to execute it as root.

Simple as that. The only gotcha is that it is a python3 script, but since that is the standard in arch these days anyway, it shouldn’t make much of a difference anyhow.

:wq

2012w16

Sunday, April 22nd, 2012

I ought to dedicate this blog post to git and rsync: The hard drive on my netbook died this week. I haven’t attempted to recover anything from the disk yet, but of that which is most important I figure I haven’t lost anything at all. And that’s due in no small part to git and rsync.

All of my configuration files, at least those I care about, had been added to a git repository. And most of the binaries I wanted to preserve had been rsynced to my server.
Not all of it though, which is a shame, but it shouldn’t be hard to replace what I’ve lost. Especially if I can get the old hard drive to function just one more time, just long enough to at least make a list of what it is I’m missing. The rest of the disk, well, it’s spring, perhaps a spring cleaning was in order.

So all is not lost, and looking beyond this setback, I did learn some other things this week (except for the fact that I need to become better at performing backups) such as:

Also, quite some time ago, I went around thinking about how to automatically track my working time, and while this isn’t exactly like what I had in mind (I would probably just have created a daemon which somehow fetched the window title of the currently active window from X, and did so randomly 6 times per hour (not deterministically enough to be able to cheat the system).

And some assorted links which may or may not be of any particular use for anyone:

:wq

2012w15

Sunday, April 15th, 2012

This has been a pretty rough week, but I guess there is nothing less to expect when deadlines are drawing near.

This week I found myself wanting to count all the occurrences of “foo”, but ONLY if they occurred BEFORE “bar”:

awk 'BEGIN { fooCount=0; stopCounting=0 } /bar/ { stopCounting=1 } /foo/ && stopCounting=0 { fooCount = fooCount + 1 } END { print fooCount }' <myfile>

And despite the quite hectic schedule, I did manage to help a colleague with a little scripting, and those are two things which almost always sets me in a better mood: scripting (problem solving), and helping others (of course, if I don’t manage to be of any help, that kindof defeats any positive mood change I get from scripting, but in this particular case it all worked out really well in the end) :)

And now for the mandatory collection of links from this week:

  • This must be a joke right? The US can’t really, for real, be irritated with Australia for preferring national service providers over American ones, right? Especially when it could come down to storing data about Australian citizens, or in other ways vital to the government. This has to be a joke right?
  • I wonder if this is something most programmers can relate to or if it’s just me
  • This post could have been written by me… well, not as articulate, but the spirit of it. What’s even more interesting is the response this triggered on HackerNews.
  • QArt Codes is where QR codes, Solomon-Reed error correction, some extra calculations and your imagination mix together ;)
  • Hilarious post making fun about certain governments and their want for even more snooping laws, especially about conducting surveillance in in-game chats…

:wq

2012w14

Sunday, April 8th, 2012

Not a whole lot to say this week, it has mostly been work, sleep, work, sleep, … well you get the picture.

Some noteworthy things however:

Just how, in their infinite wisdom, does the EU expect to test the security of their own servers and services if they are going to outlaw so-called “cyber-attack tools”. For that matter, how do they propose ANY manufacturer of ANY type of digital system perform ANY type of actual security testing worth a damn?

Social AND Private? Well… not quite yet, but if they get the p2p and encryption stuff working, then we’re in business :)

ENCRYPT ALL THE THINGS! ;)

And it wouldn’t be one of my hallmark blagposts if it didn’t have some random links which may or may not be of any value, no would it?

Dunno just what it might be useful for, but creating 3D graphics procedurally using Lua, like with Fugu seems like just the right approach for me. If I were to do 3D models that is.

Now this promises to be an interesting game.

And a rather interesting programming language.

:wq

2012w13

Sunday, April 1st, 2012

This week has been stressful. Mostly due to the fact that we have an upcoming deadline and problems have a tendency to appear just at that time to exacerbate things further…

But with problems resolved, and tests underway, I found some time to help a colleague out with a script to automate some pretty tricky measurement tests.
This included doing some expect-scripting.

In this particular instance I needed to get a bunch of parameters sent to expect, and treat them as a single string. Ordinarily something like ./foo.exp "some space separated parameters" would have worked, but quoting would be a hassle, so if I could avoid that it would be great.

So I learnt about Tcl’s lrange (up to that point I’d only worked with lindex) and constructed something along the lines of set foo [lrange $argv 3 end] (the first three parameters were other things).

This week I’ve also had the pleasure to read about some pretty cool people this week:

Of course, there are people in the other part of that spectrum, people who doesn’t seem to like the ugly truth or who’d just like to kill off the Internet in its present form.

Now, if you’re still a Firefox user (these days it seems people have all jumped on the Chrome bandwagon) and you still don’t understand why it could be good idea to use NoScript, RequestPolicy or BetterPrivacy, Collusion, from Mozilla might visualize it for you. For my part, with those add-ons activated, there weren’t any surprises, but just for fun I turned them off, only leaving collusion on. Frak me! A whole lot of stuff happening behind the scenes.

I also sniffed out a new timeline library which uses jQuery and is powered by JSON.

:wq

2012w12

Sunday, March 25th, 2012

Woho! Not only a great week at work, but I also managed to squeeze in some other activities as well.

Managed to execute two test cases (yes, that’s a little slow, but there were mitigating circumstances, so all in all I am not too worried, and I did learn from it so it’s all good), got a small lecture in a subsystem (which made me want to tinker with Erlang and mnesia again), went swimming after work one day, and spent Thursday evening at an OWASP event, listening to a very entertaining dude named Jim Manico

There was a change in plans, so the talk wasn’t about Web Application Access Control Design but instead about the ten most critical web application security flaws found on the “OWASP Top Ten Project”-list.

Learning interesting stuff AND having fun at the same time? Oh wait, when I put it like that it just sounds like any other (work)day in my life, but you get the point. Great event, looking forward to the next :)

And Friday evening was spent hanging out with Rikard, Zara and Alfred. All in all, a rather good week.

I did have some trouble with tmux, not all of which I managed to solve.

There were actually two issues, and only one of them have been “solved”, and I use that word pretty lightly because I don’t find the solution particularly good, although it is probably the solution.

  1. Scrolling backwards (up) in an terminal inside tmux is painful. C^b [ sets you in the mode you need to be to enable PgUp to work, but that is not nearly as easy as my muscle-memory-bound Shift-PgUp (plus this also means I need to exit that “cut-mode” or whatever it is called when I’m done scrolling
  2. tmux doesn’t seem to interact all that well with a mouse. I admit, that probably wasn’t high on the priorities, but if I don’t have vim bindings (visual-mode, yank) the mouse is by far the easiest way to copy text from a terminal. Click and hold mouse1, drag over the area to copy, release mouse1, DONE!

The mouse issue is probably easy enough to fix, I suspect I just need to read the man-page better and fiddle some more with the configuration. But I am not so sure about #1. That’s the built-in way to do it… getting something better working there is probably not straightforward at all.

Finally, this week I also “rediscovered” zodiac and I am now pondering whether or not to just use that instead of building my own “makefile blog”-type of thing. I’d need to hack it a bit, there are some things I don’t want to make do without (RSS, prev/next-links, tags) and it would be pretty neat (albeit useless) to have post signing using GPG.

:wq