Posts Tagged ‘Python’

2012w39

Sunday, September 30th, 2012

The bad

EU is hard at work making a complete ass of itself… again… further reading

The good

California passes groundbreaking open textbook legislation :D

The unsorted thoughtful and/or cool

Pretty cool idea although probably not very useful for anything ;D

The lack of a good manual to explain both a) how it works, b) what it will actually install and where (and if it will set up services or not), c) and how I would actually operate it once installed, are the things (i.e., all results of lacking the necessary documentation for me to read up on BEFORE trying it) which keep me from even considering installing and testing prophet. Which is a real shame as it sounds like an interesting project.

Yet another nifty AWK tutorial

I just heard about Dancer, which seems to be to Perl, what Flask is to Python

:wq

2012w25

Sunday, June 24th, 2012

Quite a while since I wrote a post now, I’ve not been sick or anything, but there has been a lot of work abound, and outside work I prioritized sleeping over writing. But now I’m back for the moment, so let’s get down to business :)

Since last time I’ve come up with new ways of abusing awk, such as having it find the highest value from a command outputting in the following syntax:

\t<characters, integers, fullstop>: <integer>\n

To make it a little more different, the command also spits out a header, as well as an additional newline after the end of output.

I just now, while writing this, came up with a different solution, which doesn’t use awk:

theCommand | grep -v '^[^ \t]\+' | tr -d ' ' | cut -d':' -f2 | sort -r | head -n 1

but what I ended up using was:

theCommand | awk 'BEGIN { highest = 0 } $0 ~ /^[ \t]/ { if ( $2 > highest ) { highest = $2 } } END { print highest }'

In this case, from what I can gather, awk is the more efficient solution. One process versus five.

Update: As Werner points out, the if statement isn’t really necessary (which also makes it possible to cut out the BEGIN statement as well):

theCommand | awk '/^[^ \t]/ && $2 > highest { highest = $2 } END { printf "%d\n", highest }'

Utilities

  • ditaa (a.k.a DIagrams Through Ascii Art) makes it easy to generate nice-looking diagram images from… rather nice-looking ASCII diagrams
  • docopt, a command-line interface description language, which also seems to support generating the parser for the CLI being described
  • Peity for generating different types of charts using jQuery and <canvas>
  • Ghost.py interacting with web pages, programmatically

As of late I have been thinking a great deal about backups and the project which seems the most interesting to me is Duplicity.

Random tech stuff

Other random not-so-techy stuff

What I pass for humour

:wq

2012w01

Sunday, January 8th, 2012

column

The other day I wanted some prettier (tabularized) output and of course someone has already wanted this and of course there are tools for that :)

bash_completion

This is so frakking cool! I’ve built this little shellscript “vault.sh” which is a simple wrapper script for mounting and unmounting encfs mounts.

It takes two parameters: operation and target, where operation can be one of “lock” and “unlock”, and target—at present—resolves to “thunderbird” (signifying my .thunderbird directory).

Since I intend to expand this with more encrypted directories as I see fit, I don’t want to hard-code that.

What I did want, however, was to be able to auto complete operation and target. So I looked around, and found this post, and although I couldn’t derive enough knowledge from it to solve my particular problem, having multiple levels of completion, the author was gracious enough to provide references to where s/he had found the knowledge (here, here and here). That second link was what did it for me.

My /etc/bash_completion.d/vault.sh now looks like this:

_vault()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    first="lock unlock"
    second="thunderbird"

    if [[ ${cur} == * && ${COMP_CWORD} -eq 2 ]] ; then
        COMPREPLY=( $(compgen -W "${second}" -- ${cur}) )
        return 0
    fi

    if [[ ${cur} == * && ${COMP_CWORD} -eq 1 ]] ; then
        COMPREPLY=( $(compgen -W "${first}" -- ${cur}) )
        return 0
    fi
}
complete -F _vault vault.sh

And all the magic is happening in the two if-statements. Essentially: if current word (presently half typed and tabbed) is whatever, and this is the second argument to the command, respond with suggestions taken from the variable $second.

Otherwise, if current word is whatever, and this is the first parameter, take suggestions from the variable $first.

Awsum!

awk for great good

Another great use for awk: viewing selected portions of source code. For instance, in Perl, if you just want to view a specific subroutine, without getting distracted by all the other crud, you could do: $ awk '/sub SomeSubName/,/}/' somePerlModule.pm

Links

If PHP were British, perhaps it’s just me, but I find it hilarious.

PayPal just keeps working their charm…

Belarus just… wait what?

Why we need version control

Preserving space, neat!

Fuzzy string matching in Python

If you aren’t embarrassed by v1.0 you didn’t release it early enough

The makers schedule, oldie but goldie

CSS Media Queries are pretty cool

Static site generator using the shell and awk

A netstat companion

Reducing code nesting

Comparing images using perceptual hashes

Microsofts GPS “avoid ghetto” routing algorithm patent…

2011w31

Sunday, August 7th, 2011

(python) mechanize and BeautifulSoup

I’m slowly making preparations to replace WordPress with, as it currently stands, fugitive.

But I have a couple of posts in WordPress, 150—counting this one—to be more precise… which I don’t particularly feel like just abandoning, well… at least not all of them ;)

This means that they need to be preserved, and that’s where python’s two excellent modules mechanize and BeautifulSoup come into play.

Yes, I know relying on Beautiful Soup is discouraged nowadays, but as long as it works, I’ll be using it.

There is this great post about the basics of mechanize which, and since I only wanted to scrape a couple of posts, that was all I needed.

I needed Beautiful Soup to soup.find('link', {'rel': 'next'}) (i.e. get the URL for the “next” post) and then there was the little problem of retrieving the href-attribute from the found link-element.

StackOverflow to the rescue!

rsync

I found myself needing to synchronize a folder between two systems. I.e. new files added in the “source” system needed to be added to the “destination” which is a simple
$ rsync -av /path/to/source/directory/ user@remote:/path/to/destination/directory/
(please note the trailing slashes in BOTH paths, these in the source path, which tell rsync to copy the files INSIDE the directory, not including the directory itself).

However, for the first time I also needed that all removed files in source should be removed at the destination as well. I found this blogpost which gave me the information I needed.

This is… not trickier, but… not something you’d want to frakk up.

So, the flag which will delete on the removed files is simply --delete i.e.:

$ rsync -av --delete /path/to/source/directory/ user@remote:/path/to/destination/directory/

BEFORE you do this you REALLY should check that the operation will perform correctly by also attaching the flag --dry-run (which will simulate the real deal, without doing any changes on the remote end). Very nice.

$ rsync --dry-run -av --delete /path/to/source/directory/ user@remote:/path/to/destination/directory/

Musings

@shiny posted a notice about “open surface” being the perfect term for software that should be avoided.

@webmink, in what feels like true hacker fashion, cleverly played around with the words of the term and came up with “superficially open”.

Links

The Longest joke in the world

:wq

2011w26

Sunday, July 3rd, 2011

How silly of me… I totally forgot to publish last weeks summary yesterday. So without further ado, only one day late:

bashPhotoGallery

I don’t know how I get myself into these things… all of a sudden I found myself needing a, reproducible, way of setting up a photo gallery, complete with thumbnails and affixing a content license to the images.

When it comes to creating a batch of thumbnails, imagemagick is the tool to use. Accept no substitutes!

I am also going through a little crush on markdown.

Getting rid of UTM variables

I am not all that fond of those UTM variables that some “services” tack onto their links in order to better track where people are coming from (I understand why they’d do it, but I have no interest in being tracked, even if all they want to know is whether or not their push to be visible on $SOCIAL_MEDIA_SITE_OF_THE_MONTH is successful.
I know that I by accident stumbled upon a blog post outlining how to get rid of them programmatically, and I also know that I for some reason or other couldn’t find it, but without being too paranoid, I can understand why Google might not want to help people find the knowledge to do that ;)
In any case, I stumbled upon two resources for doing just that, but I think that was more due to dumb luck than any concentrated effort on my part, actually, it was quite by accident while looking for something else.

Bash is so cool!

I already knew about echo {A,B,C,D} (great in conjunction with mkdir -p, but I have realized that bash is cooler than that.
echo {A..D} delivers the same result, without the need to explicitly specify all of the chars I want expanded. Nice!

makefile blogging :: comments

psquid had a rather interesting solution to blog comments. I’ll have to think more about this. I don’t know how I feel about letting some other party (even if it is as nice a party as identi.ca) host “my” comments, but it is totally worth considering.

Wednesday

All in all a pretty good day. Got to assist razor with both vim and LaTeX skills (the student has become the master, yay!), got some writing out of my head, and ended up doing a little Test-Driven Python hacking.

And although I was a bit sceptical about OlofB‘s pyTDDmon, especially about it blinking continuously, which could get really old really fast, at first, I have to say that it has kindof grown on me since.

2011w22

Sunday, June 5th, 2011

SSH tunnelling

This Friday I finally got a valid reason to dig into how one sets up an SSH tunnel between two machines. The reason was that I was sitting at the new FFKP office lulzing about with razor, and found myself needing to test some PHP I had been working on.

So I was not at home, I am not stuffing Apache and PHP and MySQL onto my netbook just to do web dev stuff, so I needed contact with my “server” back home.

The slight problem being that since it is for development use only, I don’t expose its Apache to the Internet, only to the local network. SSH is another matter altogether.

So I thought that it shouldn’t be impossible to set up what I wanted, i.e. from my netbook, type in localhost:8080 (or whatever port number floats your boat) and be routed through the tunnel to the server.

It turns out there was this really neat write up on how to do it, already available, and even better, it was really simple:

ssh -f <user>@<remote-host> -L <local-port>:<remote-host>:<remote-port> -N

-f makes it a background process, -L tells SSH we want a tunnel, -N tells SSH that we aren’t trying to execute a command.

This is something I might even begin to remember :)

wmfs

wmfs, or window manager from scratch, seems like an awfully nice little tiling window manager.

Unfortunately I haven’t gotten the time this weekend to play with it, but perhaps a little later today?

I was warned on diaspora that there is a pretty huge-ish performance-related bug in the code right now, and since I haven’t tested it yet I can’t make any recommendations, but from the very superficial (and that will hopefully soon change) observation of the documentation, it seems like it could be a wmii-replacer.

Links

As I am attempting to learn git, I soon came to the conclusion that the best way to learn it would be to use it.

After all, the basics are not extremely different from mercurial, and while most of my projects remain single-person-projects, the basics are more than enough for me.

So I hereby solemnly swear that the next little project I start, whatever it may be, will be versioned using git instead of mercurial.

My foray into the git world also meant that I started looking at the two largest git hosting solutions (both of them free as in beer, and one free as in freedom, github and gitorious.

In doing so, I ran through the list of hosted projects, and there is so much cool stuff out there to test, and try, and learn, and play with… so many things, so little time.

  • z – jump around, which studies your usage of the various directories on your machine, learns, and then makes it easy to jump to “popular” directories easily
  • yajl, Yet Another JSON Library, and I likes me sum JSON
  • Underscore.js, a javascript “utility-belt” library
  • wormhole a jQuery plugin, which for most parts would probably be more fun than useful, but it seems to have been initially written to scratch an itch
  • Backbone.js, which I had already read about here and been wanting to play with it ever since
  • Microjs, a site for finding the javascript framework you need, in order to do what you want
  • Raphaël, a javascript library for working with SVG
  • TMS, a Temporary Mail Server, written in Python, this just must be useful for testing and debugging mail-stuff
  • tablib, another Python module, this one for parsing and converting tabular datasets between various formats, (i.e. csv, html, json, ods, xls(x), yaml)
  • twotwodo, a personal 2do list in the browser, using jQuery (javscript) for logic, and cookies for storage (so it is local to the machine, with no shared storage back-end, which can be good sometimes
  • protovis, nice javascript visualization toolkit

Two other links I find worth mentioning are:

FamFamFam has a rather nice-looking icon set (Silk), licensed using CC By 2.5 or 3.0, which I might finally get to use in a project of mine (haven’t really gotten a chance since it was a long time since I did any serious web programming.)

Finally, a blog which has helped me greatly in understanding various things is BetterExplained which has now released something new which they call aha.betterexplained.com, and I will follow this with great interest.

Revelation of the week

The one real “aha” moment this week was Friday afternoon, when Grégoire showed me that it was possible to add people from other diaspora seeds by searching for their <username>@<other-seed>.
That was good news since I was rather bummed out about razor and greg had set up shop at diasp.org leaving me with relatively few contacts on “my” seed.

Other than that, to be honest, this week has been pretty boring. I did get a whole lot done in timetrack yesterday, and found a deeper love for grep‘s -A flag, and I have been doing some serious thinking about writing my own “makefile blogging”-thingy.

Friday night right before falling asleep I got the idea to write a little script which would pick a wallpaper at random and set that as the active wallpaper at startup. Since I use wmii, and wmii uses xloadimage, given a path, I could simply put all my wallpapers in a directory and have the script symlink one at random on start up.

2011w21

Sunday, May 29th, 2011

timetrack

I have come up with a way to achieve the changes I want, but without introducing sqlite3 as a dependency, and a big part of the solution is to use bash arrays.

Furthermore, I have been thinking about how to, if possible, get timetrack to automagically start a new session when a file in the project is opened.

This won’t help anyone to start the timetracker when thinking about the project, but at least when physically transferring code from brain to hard drive, and the lead I am working off of is inotify.

ticket

During this weeks FSCONS meeting jonaso jokingly suggested that I’d try to write an issue tracker in bash. (damn you! ;))

Of course my mind started wandering and although there is no code to back it up, I have a couple of rather interesting ideas about how to pull it off.

For this project, sqlite is the way to go, but I was somewhat worried about concurrent access which I probably shouldn’t be.

My tests indicate (oh yeah, so there exist code, just not any actual issue tracking code) that the sqlite3 library is intelligent enough to lock the file, and thus doesn’t allow concurrent access.

I’ll still need to devise a way of detecting these locks, and have the second script stand in line and try again later, but that should be trivial.

Links

Turning Vim into a modern Python IDE

Learning styles
From what I can gather, I am an assimilator. resistance is futile!

Cheat sheets!

The 9 secret burdens of being a Linux user

Big businesses acting out like this might very well get me to start boycotting them again…

String manipulation in bash

Seemingly nice way of doing HTTP requests in Python

2011w20

Sunday, May 22nd, 2011

Update: I suck! I forgot to make the URLs in the links section actually links… updated now.

Thunderbird / Lightning / iCal

I recently started receiving emails including iCal invitations that needed to be answered, but I had no idea how, and Thunderbird does not come with any such functionality out of the box.

The “Lightning” add-on, however, does give Thunderbird features to handle that, and it works really well (if one makes one configuration change in Thunderbird: go to “View” and ensure that “Display Attachments Inline” is checked)

On my netbook, I  must have already set this, as it just worked there, whilst I was dumbfounded and needed the guidance from this thread to get it to work on the desktop.

Now every email with an iCal thingy that needs to be responded to will display a question at the top of the email window/tab

chromebooks

I pretty much agree with everything said in this article and for those reasons, a computer running a “cloud OS” will not become a consideration for me, until the computer is running my cloud which I and only I have full control over.

Most people won’t, but I’ll take privacy over simplicity/ease of use (or whatever other selling point is being made about these products) any day of the week.

And while we are speaking about the cloud, and why I dislike it so much, it is convenient that the next topic is related:

dropbox

So, Dropbox, the simple cloud storage and file synchronization service turned out to have a rather huge security flaw: Their employees can (to my knowledge it hasn’t happened, but how would anyone except a potential offender know, and it is this uncertainty which makes me shy away from such services) access their users encrypted shares, since Dropbox stores the encryption keys.

Dropbox is probably just as legit as they have always been, and they have probably never done anything wrong, but I can’t say that incidents like these strengthen my confidence in “the cloud”, at least not clouds operated by third parties, or actually, anyone except for the individual herself.

And that is why it is good to see that alternatives are beginning to crop up.

Jsoup

A friend of mine is doing some Java (Android) hacking, and asked me if I knew of any good web scraping libraries. For Java, my answer was no. For Python I would have instantly responded “Beautiful Soup”. So my answer became: “If I were you, I’d Google for beautiful soup for Java”. And then I did that myself, finding this post which inevitably lead me here.

Links

If you aren’t embarrassed by v 1.0 you didn’t release it early enough
Why we need version control
Interesting ways of making the most of a small living space

Revelation of the week

Learning a programming language by using an IDE can be damaging almost beyond repair.

This might perhaps just be me, but I learned html (albeit not strictly a programming language) in notepad, and have had no problems with html ever.

The same is true for bash, Python, javascript and Erlang. C would be the exception, those damn pointers continues to elude my understanding (not the concept of them, the syntax).

And then there is Java. We were taught to interact with it through an IDE, Eclipse as it was. That was 2005. This Wednesday was the first time I managed to write some barely non-trivial Java, compile it, and execute it, outside the “safety” of an IDE.

The reason for this was that Eclipse, for reasons beyond my understanding, keep crashing a couple of seconds after starting up, and that I had a friend in need of a little technology demonstrator.

And that’s when it really dawned on me. Outside Eclipse… it’s not that I am all that lost, it’s just that everything takes longer, is more tedious, and I have thusly shied away from it, thereby reinforcing that exact pattern.

And that is surely detrimental. I don’t want to be tied to a specific tool in order to be able to perform above average. A specialized tool might perhaps increase the effectiveness further, but being lost without it… that’s just wrong.

My software stack revisited – Programming

Friday, December 24th, 2010

Programming is one of my primary interests, mainly because it allows me to stimulate my brain with solving problems, but also force it to think in new ways.

Languages

I started programming in PHP, picked up Java and Erlang during classes at ITU, picked up Python on my own during my studies at ITU, and my latest addition would be shell scripting.

Slightly tangent to the topic are the markup languages I have picked up as well, html and css in high-school and LaTeX at ITU. I dabbled around for a while with both creole and markdown, but that didn’t last long.

Editor / IDE

My first and foremost tool of choice given nearly any situation will be (g)vim. The only two exceptions I can think of off the bat is Java (for which I use Eclipse and if I need to write a whole lot of text, with minimal distraction (more on that later).

The pragmatic programmers recommend learning one text-editor, and learn it well. If the name of that editor is vim, emacs, kate, gedit, or whatever, I really don’t care. Just pick up one that fits you, and LEARN IT WELL!

I have extended vim with a couple of plugins, the most prominent being NERD Commenter, matchit, snipMate and sparkup. There are at least two more plugins, but I will write more about those later.

And for Python, I usually install the IPython interactive prompt as it is a fair bit more useful than the standard python-prompt.

Version Control

While studying at ITU I had my eyes opened about the wonderful concept of version control.

I was first exposed to SVN, and while quite capable, I figured it was too much of a hassle to set it up myself, since that would require the presence of a server somewhere to host the SVN repositories.

But then mercurial entered the stage. Git or bazaar would have done the job just as good, but the people orchestrating the fourth term settled on mercurial, and it is so dead simple and still powerful enough for what I need that I haven’t had a reason to look elsewhere.

Issue tracking

For a course at ITU I tried using Mantis, a web-based bug tracker written in PHP, and while it worked well, it was a hassle to manipulate bug reports since it meant I’d have to go online and log in to yet another system.

I have however found a different solution which I am currently trying out: a plugin to mercurial called b with the tagline “distributed bug tracking”. It is a bit too early to tell if it will do, but for the time being it solves the immediate problem of having to go online somewhere to handle bugs.

Next post in line: “Office Suite” software

:wq

Vim + snipMate

Sunday, March 14th, 2010

As I sat down in front of the computer today and started reading the identi.ca backlog I came across this gem, which lead here, about how to turn Vim into a Python IDE. The blogpost itself contained many useful tips/tricks (the code folding script being on the top of that list), but as is often the case, the really cool stuff gets unearthed in the comments, like this one about snipMate. Also, excellent video presentation is excellent!

That post also lead me to another post on that blog which inspired me to finally start looking into ctags, which is a two step installation, one being installing the package exuberant-ctags from the repository, and the other fetching this script.

All I need to do now is try to learn the key mappings M^e, f, M^f, and TT/F4 ;D oh, and to tab a lot ;D