Posts Tagged ‘Python’

2012w01

Sunday, January 8th, 2012

column

The other day I wanted some prettier (tabularized) output and of course someone has already wanted this and of course there are tools for that :)

bash_completion

This is so frakking cool! I’ve built this little shellscript “vault.sh” which is a simple wrapper script for mounting and unmounting encfs mounts.

It takes two parameters: operation and target, where operation can be one of “lock” and “unlock”, and target—at present—resolves to “thunderbird” (signifying my .thunderbird directory).

Since I intend to expand this with more encrypted directories as I see fit, I don’t want to hard-code that.

What I did want, however, was to be able to auto complete operation and target. So I looked around, and found this post, and although I couldn’t derive enough knowledge from it to solve my particular problem, having multiple levels of completion, the author was gracious enough to provide references to where s/he had found the knowledge (here, here and here). That second link was what did it for me.

My /etc/bash_completion.d/vault.sh now looks like this:

_vault()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    first="lock unlock"
    second="thunderbird"

    if [[ ${cur} == * && ${COMP_CWORD} -eq 2 ]] ; then
        COMPREPLY=( $(compgen -W "${second}" -- ${cur}) )
        return 0
    fi

    if [[ ${cur} == * && ${COMP_CWORD} -eq 1 ]] ; then
        COMPREPLY=( $(compgen -W "${first}" -- ${cur}) )
        return 0
    fi
}
complete -F _vault vault.sh

And all the magic is happening in the two if-statements. Essentially: if current word (presently half typed and tabbed) is whatever, and this is the second argument to the command, respond with suggestions taken from the variable $second.

Otherwise, if current word is whatever, and this is the first parameter, take suggestions from the variable $first.

Awsum!

awk for great good

Another great use for awk: viewing selected portions of source code. For instance, in Perl, if you just want to view a specific subroutine, without getting distracted by all the other crud, you could do: $ awk '/sub SomeSubName/,/}/' somePerlModule.pm

Links

If PHP were British, perhaps it’s just me, but I find it hilarious.

PayPal just keeps working their charm…

Belarus just… wait what?

Why we need version control

Preserving space, neat!

Fuzzy string matching in Python

If you aren’t embarrassed by v1.0 you didn’t release it early enough

The makers schedule, oldie but goldie

CSS Media Queries are pretty cool

Static site generator using the shell and awk

A netstat companion

Reducing code nesting

Comparing images using perceptual hashes

Microsofts GPS “avoid ghetto” routing algorithm patent…

2011w31

Sunday, August 7th, 2011

(python) mechanize and BeautifulSoup

I’m slowly making preparations to replace WordPress with, as it currently stands, fugitive.

But I have a couple of posts in WordPress, 150—counting this one—to be more precise… which I don’t particularly feel like just abandoning, well… at least not all of them ;)

This means that they need to be preserved, and that’s where python’s two excellent modules mechanize and BeautifulSoup come into play.

Yes, I know relying on Beautiful Soup is discouraged nowadays, but as long as it works, I’ll be using it.

There is this great post about the basics of mechanize which, and since I only wanted to scrape a couple of posts, that was all I needed.

I needed Beautiful Soup to soup.find('link', {'rel': 'next'}) (i.e. get the URL for the “next” post) and then there was the little problem of retrieving the href-attribute from the found link-element.

StackOverflow to the rescue!

rsync

I found myself needing to synchronize a folder between two systems. I.e. new files added in the “source” system needed to be added to the “destination” which is a simple
$ rsync -av /path/to/source/directory/ user@remote:/path/to/destination/directory/
(please note the trailing slashes in BOTH paths, these in the source path, which tell rsync to copy the files INSIDE the directory, not including the directory itself).

However, for the first time I also needed that all removed files in source should be removed at the destination as well. I found this blogpost which gave me the information I needed.

This is… not trickier, but… not something you’d want to frakk up.

So, the flag which will delete on the removed files is simply --delete i.e.:

$ rsync -av --delete /path/to/source/directory/ user@remote:/path/to/destination/directory/

BEFORE you do this you REALLY should check that the operation will perform correctly by also attaching the flag --dry-run (which will simulate the real deal, without doing any changes on the remote end). Very nice.

$ rsync --dry-run -av --delete /path/to/source/directory/ user@remote:/path/to/destination/directory/

Musings

@shiny posted a notice about “open surface” being the perfect term for software that should be avoided.

@webmink, in what feels like true hacker fashion, cleverly played around with the words of the term and came up with “superficially open”.

Links

The Longest joke in the world

:wq

2011w26

Sunday, July 3rd, 2011

How silly of me… I totally forgot to publish last weeks summary yesterday. So without further ado, only one day late:

bashPhotoGallery

I don’t know how I get myself into these things… all of a sudden I found myself needing a, reproducible, way of setting up a photo gallery, complete with thumbnails and affixing a content license to the images.

When it comes to creating a batch of thumbnails, imagemagick is the tool to use. Accept no substitutes!

I am also going through a little crush on markdown.

Getting rid of UTM variables

I am not all that fond of those UTM variables that some “services” tack onto their links in order to better track where people are coming from (I understand why they’d do it, but I have no interest in being tracked, even if all they want to know is whether or not their push to be visible on $SOCIAL_MEDIA_SITE_OF_THE_MONTH is successful.
I know that I by accident stumbled upon a blog post outlining how to get rid of them programmatically, and I also know that I for some reason or other couldn’t find it, but without being too paranoid, I can understand why Google might not want to help people find the knowledge to do that ;)
In any case, I stumbled upon two resources for doing just that, but I think that was more due to dumb luck than any concentrated effort on my part, actually, it was quite by accident while looking for something else.

Bash is so cool!

I already knew about echo {A,B,C,D} (great in conjunction with mkdir -p, but I have realized that bash is cooler than that.
echo {A..D} delivers the same result, without the need to explicitly specify all of the chars I want expanded. Nice!

makefile blogging :: comments

psquid had a rather interesting solution to blog comments. I’ll have to think more about this. I don’t know how I feel about letting some other party (even if it is as nice a party as identi.ca) host “my” comments, but it is totally worth considering.

Wednesday

All in all a pretty good day. Got to assist razor with both vim and LaTeX skills (the student has become the master, yay!), got some writing out of my head, and ended up doing a little Test-Driven Python hacking.

And although I was a bit sceptical about OlofB‘s pyTDDmon, especially about it blinking continuously, which could get really old really fast, at first, I have to say that it has kindof grown on me since.

2011w22

Sunday, June 5th, 2011

SSH tunnelling

This Friday I finally got a valid reason to dig into how one sets up an SSH tunnel between two machines. The reason was that I was sitting at the new FFKP office lulzing about with razor, and found myself needing to test some PHP I had been working on.

So I was not at home, I am not stuffing Apache and PHP and MySQL onto my netbook just to do web dev stuff, so I needed contact with my “server” back home.

The slight problem being that since it is for development use only, I don’t expose its Apache to the Internet, only to the local network. SSH is another matter altogether.

So I thought that it shouldn’t be impossible to set up what I wanted, i.e. from my netbook, type in localhost:8080 (or whatever port number floats your boat) and be routed through the tunnel to the server.

It turns out there was this really neat write up on how to do it, already available, and even better, it was really simple:

ssh -f <user>@<remote-host> -L <local-port>:<remote-host>:<remote-port> -N

-f makes it a background process, -L tells SSH we want a tunnel, -N tells SSH that we aren’t trying to execute a command.

This is something I might even begin to remember :)

wmfs

wmfs, or window manager from scratch, seems like an awfully nice little tiling window manager.

Unfortunately I haven’t gotten the time this weekend to play with it, but perhaps a little later today?

I was warned on diaspora that there is a pretty huge-ish performance-related bug in the code right now, and since I haven’t tested it yet I can’t make any recommendations, but from the very superficial (and that will hopefully soon change) observation of the documentation, it seems like it could be a wmii-replacer.

Links

As I am attempting to learn git, I soon came to the conclusion that the best way to learn it would be to use it.

After all, the basics are not extremely different from mercurial, and while most of my projects remain single-person-projects, the basics are more than enough for me.

So I hereby solemnly swear that the next little project I start, whatever it may be, will be versioned using git instead of mercurial.

My foray into the git world also meant that I started looking at the two largest git hosting solutions (both of them free as in beer, and one free as in freedom, github and gitorious.

In doing so, I ran through the list of hosted projects, and there is so much cool stuff out there to test, and try, and learn, and play with… so many things, so little time.

  • z – jump around, which studies your usage of the various directories on your machine, learns, and then makes it easy to jump to “popular” directories easily
  • yajl, Yet Another JSON Library, and I likes me sum JSON
  • Underscore.js, a javascript “utility-belt” library
  • wormhole a jQuery plugin, which for most parts would probably be more fun than useful, but it seems to have been initially written to scratch an itch
  • Backbone.js, which I had already read about here and been wanting to play with it ever since
  • Microjs, a site for finding the javascript framework you need, in order to do what you want
  • Raphaël, a javascript library for working with SVG
  • TMS, a Temporary Mail Server, written in Python, this just must be useful for testing and debugging mail-stuff
  • tablib, another Python module, this one for parsing and converting tabular datasets between various formats, (i.e. csv, html, json, ods, xls(x), yaml)
  • twotwodo, a personal 2do list in the browser, using jQuery (javscript) for logic, and cookies for storage (so it is local to the machine, with no shared storage back-end, which can be good sometimes
  • protovis, nice javascript visualization toolkit

Two other links I find worth mentioning are:

FamFamFam has a rather nice-looking icon set (Silk), licensed using CC By 2.5 or 3.0, which I might finally get to use in a project of mine (haven’t really gotten a chance since it was a long time since I did any serious web programming.)

Finally, a blog which has helped me greatly in understanding various things is BetterExplained which has now released something new which they call aha.betterexplained.com, and I will follow this with great interest.

Revelation of the week

The one real “aha” moment this week was Friday afternoon, when Grégoire showed me that it was possible to add people from other diaspora seeds by searching for their <username>@<other-seed>.
That was good news since I was rather bummed out about razor and greg had set up shop at diasp.org leaving me with relatively few contacts on “my” seed.

Other than that, to be honest, this week has been pretty boring. I did get a whole lot done in timetrack yesterday, and found a deeper love for grep‘s -A flag, and I have been doing some serious thinking about writing my own “makefile blogging”-thingy.

Friday night right before falling asleep I got the idea to write a little script which would pick a wallpaper at random and set that as the active wallpaper at startup. Since I use wmii, and wmii uses xloadimage, given a path, I could simply put all my wallpapers in a directory and have the script symlink one at random on start up.

2011w21

Sunday, May 29th, 2011

timetrack

I have come up with a way to achieve the changes I want, but without introducing sqlite3 as a dependency, and a big part of the solution is to use bash arrays.

Furthermore, I have been thinking about how to, if possible, get timetrack to automagically start a new session when a file in the project is opened.

This won’t help anyone to start the timetracker when thinking about the project, but at least when physically transferring code from brain to hard drive, and the lead I am working off of is inotify.

ticket

During this weeks FSCONS meeting jonaso jokingly suggested that I’d try to write an issue tracker in bash. (damn you! ;) )

Of course my mind started wandering and although there is no code to back it up, I have a couple of rather interesting ideas about how to pull it off.

For this project, sqlite is the way to go, but I was somewhat worried about concurrent access which I probably shouldn’t be.

My tests indicate (oh yeah, so there exist code, just not any actual issue tracking code) that the sqlite3 library is intelligent enough to lock the file, and thus doesn’t allow concurrent access.

I’ll still need to devise a way of detecting these locks, and have the second script stand in line and try again later, but that should be trivial.

Links

Turning Vim into a modern Python IDE

Learning styles
From what I can gather, I am an assimilator. resistance is futile!

Cheat sheets!

The 9 secret burdens of being a Linux user

Big businesses acting out like this might very well get me to start boycotting them again…

String manipulation in bash

Seemingly nice way of doing HTTP requests in Python

2011w20

Sunday, May 22nd, 2011

Update: I suck! I forgot to make the URLs in the links section actually links… updated now.

Thunderbird / Lightning / iCal

I recently started receiving emails including iCal invitations that needed to be answered, but I had no idea how, and Thunderbird does not come with any such functionality out of the box.

The “Lightning” add-on, however, does give Thunderbird features to handle that, and it works really well (if one makes one configuration change in Thunderbird: go to “View” and ensure that “Display Attachments Inline” is checked)

On my netbook, I  must have already set this, as it just worked there, whilst I was dumbfounded and needed the guidance from this thread to get it to work on the desktop.

Now every email with an iCal thingy that needs to be responded to will display a question at the top of the email window/tab

chromebooks

I pretty much agree with everything said in this article and for those reasons, a computer running a “cloud OS” will not become a consideration for me, until the computer is running my cloud which I and only I have full control over.

Most people won’t, but I’ll take privacy over simplicity/ease of use (or whatever other selling point is being made about these products) any day of the week.

And while we are speaking about the cloud, and why I dislike it so much, it is convenient that the next topic is related:

dropbox

So, Dropbox, the simple cloud storage and file synchronization service turned out to have a rather huge security flaw: Their employees can (to my knowledge it hasn’t happened, but how would anyone except a potential offender know, and it is this uncertainty which makes me shy away from such services) access their users encrypted shares, since Dropbox stores the encryption keys.

Dropbox is probably just as legit as they have always been, and they have probably never done anything wrong, but I can’t say that incidents like these strengthen my confidence in “the cloud”, at least not clouds operated by third parties, or actually, anyone except for the individual herself.

And that is why it is good to see that alternatives are beginning to crop up.

Jsoup

A friend of mine is doing some Java (Android) hacking, and asked me if I knew of any good web scraping libraries. For Java, my answer was no. For Python I would have instantly responded “Beautiful Soup”. So my answer became: “If I were you, I’d Google for beautiful soup for Java”. And then I did that myself, finding this post which inevitably lead me here.

Links

If you aren’t embarrassed by v 1.0 you didn’t release it early enough
Why we need version control
Interesting ways of making the most of a small living space

Revelation of the week

Learning a programming language by using an IDE can be damaging almost beyond repair.

This might perhaps just be me, but I learned html (albeit not strictly a programming language) in notepad, and have had no problems with html ever.

The same is true for bash, Python, javascript and Erlang. C would be the exception, those damn pointers continues to elude my understanding (not the concept of them, the syntax).

And then there is Java. We were taught to interact with it through an IDE, Eclipse as it was. That was 2005. This Wednesday was the first time I managed to write some barely non-trivial Java, compile it, and execute it, outside the “safety” of an IDE.

The reason for this was that Eclipse, for reasons beyond my understanding, keep crashing a couple of seconds after starting up, and that I had a friend in need of a little technology demonstrator.

And that’s when it really dawned on me. Outside Eclipse… it’s not that I am all that lost, it’s just that everything takes longer, is more tedious, and I have thusly shied away from it, thereby reinforcing that exact pattern.

And that is surely detrimental. I don’t want to be tied to a specific tool in order to be able to perform above average. A specialized tool might perhaps increase the effectiveness further, but being lost without it… that’s just wrong.

My software stack revisited – Programming

Friday, December 24th, 2010

Programming is one of my primary interests, mainly because it allows me to stimulate my brain with solving problems, but also force it to think in new ways.

Languages

I started programming in PHP, picked up Java and Erlang during classes at ITU, picked up Python on my own during my studies at ITU, and my latest addition would be shell scripting.

Slightly tangent to the topic are the markup languages I have picked up as well, html and css in high-school and LaTeX at ITU. I dabbled around for a while with both creole and markdown, but that didn’t last long.

Editor / IDE

My first and foremost tool of choice given nearly any situation will be (g)vim. The only two exceptions I can think of off the bat is Java (for which I use Eclipse and if I need to write a whole lot of text, with minimal distraction (more on that later).

The pragmatic programmers recommend learning one text-editor, and learn it well. If the name of that editor is vim, emacs, kate, gedit, or whatever, I really don’t care. Just pick up one that fits you, and LEARN IT WELL!

I have extended vim with a couple of plugins, the most prominent being NERD Commenter, matchit, snipMate and sparkup. There are at least two more plugins, but I will write more about those later.

And for Python, I usually install the IPython interactive prompt as it is a fair bit more useful than the standard python-prompt.

Version Control

While studying at ITU I had my eyes opened about the wonderful concept of version control.

I was first exposed to SVN, and while quite capable, I figured it was too much of a hassle to set it up myself, since that would require the presence of a server somewhere to host the SVN repositories.

But then mercurial entered the stage. Git or bazaar would have done the job just as good, but the people orchestrating the fourth term settled on mercurial, and it is so dead simple and still powerful enough for what I need that I haven’t had a reason to look elsewhere.

Issue tracking

For a course at ITU I tried using Mantis, a web-based bug tracker written in PHP, and while it worked well, it was a hassle to manipulate bug reports since it meant I’d have to go online and log in to yet another system.

I have however found a different solution which I am currently trying out: a plugin to mercurial called b with the tagline “distributed bug tracking”. It is a bit too early to tell if it will do, but for the time being it solves the immediate problem of having to go online somewhere to handle bugs.

Next post in line: “Office Suite” software

:wq

Vim + snipMate

Sunday, March 14th, 2010

As I sat down in front of the computer today and started reading the identi.ca backlog I came across this gem, which lead here, about how to turn Vim into a Python IDE. The blogpost itself contained many useful tips/tricks (the code folding script being on the top of that list), but as is often the case, the really cool stuff gets unearthed in the comments, like this one about snipMate. Also, excellent video presentation is excellent!

That post also lead me to another post on that blog which inspired me to finally start looking into ctags, which is a two step installation, one being installing the package exuberant-ctags from the repository, and the other fetching this script.

All I need to do now is try to learn the key mappings M^e, f, M^f, and TT/F4 ;D oh, and to tab a lot ;D

pyflakes-vim

Saturday, July 18th, 2009

It all started with this notice on identi.ca:

@jamesw thanks! pyflakes-vim is a welcome addition to my !vim !python setup

Being the curious person I am, I sought and found what it is and does. It adds wavy red underlines in Vim to Python code which is determined incorrect. I tried it out just for a second, seeing if it could keep all it promised, and it indeed seem like it. It identifies unnecessary (unused) imports, it identified when I tried to append stuff to a variable I hadn’t declared as a list yet. All in all pretty nice.

I would have to say that I didn’t care much for the packaging of the vim-scripts, when I extracted them they spread over my ~/src directory instead of confining themselves to a pyflakes-vim/ directory inside the ~/src directory, but it was only three items, a readme, a script and a directory. Easily moved. Installation from that point was even simpler.

$ mkdir -p ~/ .vim/ftplugins/python
$ cp -R pyflakes* ~/.vim/ftplugins/python

And ensure that your ~/.vimrc has the following settings:

filetype on
filetype plugin on

With that done, every time you leave insert mode, pyflakes will scan the code and tell you about any* errors in the code

*small notice here: with “any errors” I mean any errors which pyflakes can reasonably find without actually executing the code.

I do believe I have been bitten by Python

Saturday, May 30th, 2009

Lately I have found myself writing short texts in Swedish, destined to end up at a friends computer. A Windows-using friend, with all the UTF-8 / ISO-8859-1 hassles this entails. For the first file, I simply copied it onto a memory stick and rebooted into the Windows partition, and search/replaced all the offending characters (å, ä, ö and the odd é). Then rebooted again (since I don’t have my emails set up in Windows) and fired off the mail.

I simply figured that this file would be kindof a one-shot deal and nothing more. About two weeks later, I wrote a second file, and re-did the entire reboot-procedure. I found myself writing a third file yesterday… I can’t for the life of me remember the saying, or where I read it, but it was something along the lines of if you do the same thing more than twice, automate the shit out of it.

An audience with the great oracle lead me to this blog post and after trying it out manually (which required me to reboot one more time just to verify that the converted file had in fact been converted) I was all set to write a little shell script. I came so far as to write the first lines of error handling in the script (make sure that the script had recieved a filename) before I realized that I really didn’t want to write a shell script. Not when I could piece together a Python script in half that time, which would have better error checking. And yes, that time estimate included researching how to have Python execute a system call. (subprocess.call() is what I settled on, as per advise from StackOverflow. It took me a minute or so of reading the manual to figure out how to redirect the output from that command (the full text, in ISO-8859-1 encoding) to a new file (getting a file pointer to the new file, and redirecting stdout from the subprocess.call() to that file pointer)

Something along these lines:

fp = open('myfile.iso.txt', 'w')
args = ['iconv', '--from-code=UTF-8', '--to-code=ISO-8859-1', 'myfile.txt']
subprocess.call(args, stdout=fp)
fp.close()

No more silly rebooting to convert plaintext files for me :D