Posts Tagged ‘mercurial’

Important Dates Notifier

Saturday, January 29th, 2011

I have never been especially good at remembering dates or appointments. My memory just doesn’t seem constructed to handle things like “on YYYY-MM-DD do X” (peculiarly enough, “on Friday next week, have X done” works much better).

I guess things relative to “now” works better for me (at least in the short term) than some abstract date some time into the future.

Luckily enough, I don’t seem to be the only one suffering from having an “appointment-impaired memory” so others have created calendar applications and what not.

These work great, the one I presently use, remind is awesome. But sometimes it isn’t “intrusive” enough.

There are some appointments/dates that are important enough that I would like for remind to hunt me down an alley and smack me over the head with the notification, as opposed to presenting it to me when I query it for today’s appointments.

So I put together a little shell script which push notifications (the ones I feel are the most important) to me via jabber.

The solution involves cron, remind, grep and a nifty little program called sendxmpp.

This script I have placed on my “laptop-made-server” which coincidentally also happens to be the central mercurial server, through which I synchronize the repositories on my desktop and netbook.

Which means that if I just take care to clone the specific repository containing my remind files to some place local on the server, I could have a cronjob pull and update that repository and it would thus always (as long as I have pushed changes made from the desktop/netbook) have the most up to date files available.

If setting up a repository server seems to big of a hassle, one could of course (at least with remind) have a master ~/.reminders file, which then invokes an INCLUDE expression.

This makes remind look in the specified directory for other files, and in that directory have one file for each computer (along the lines of .rem) and have each individual machine scp (and overwrite) their individual file every now and then.

In any case, once the server have fresh and updated sources of data, all it need do is execute the idn.sh script once every day (preferably early, this will work pretty well as the jabber-server I use caches all messages sent to me while I was offline, so once I log in again, I’ll get the notifications then).

As sendxmpp takes the message from STDIN, recipient-addresses as parameters, and parses a configuration file to divine what account should be used to send the message (I set up a “bot” account to send from, and then just authorized that bot in my primary jabber-account), I see no reason why someone couldn’t modify the script to instead (or in addition) use something like msmtp to send out and email instead.

The script itself, in its current form, is rather straightforward, although I’m sure there are still room for optimizations.

#!/bin/sh
TAGS="#Birthday #ATTENTION"

for t in `echo "$TAGS"`;
do
    rem | grep -i "$t" | while read line;
    do
        echo "$line" | sendxmpp recipient@jabber.example.org
    done
done
exit 0

Relevant parts of my crontab:

0   7   *   *   *       cd /home/patrik/remind-repo; /usr/bin/hg pull -u 2>&1
5   7   *   *   *       cd /home/patrik/idn-repo; /usr/bin/hg pull -u 2>&1
10  7   *   *   *       /bin/bash /home/patrik/bin/idn.sh 2>&1

In /home/patrik I have created symlinks /home/patrik/.remind -> /home/patrik/remind-repo/.remind and /home/patrik/.reminders -> /home/patrik/remind-repo/.reminders

And in /home/patrik/bin/ I have a symlink (idn.sh) to /home/patrik/idn-repo/idn.sh. So in case I change the script, like add a tag to look for or something (ok, that should be moved out to a configuration file, that will be part of the next version) that will be picked up as well, before the notifications goes out.

And that’s about it. Risk of forgetting something important: mitigated.

:wq

Tags: , , , , , , , , ,
Posted in Documentation, English, GNU/Linux, Tools | 2 Comments »

My software stack revisited – Programming

Friday, December 24th, 2010

Programming is one of my primary interests, mainly because it allows me to stimulate my brain with solving problems, but also force it to think in new ways.

Languages

I started programming in PHP, picked up Java and Erlang during classes at ITU, picked up Python on my own during my studies at ITU, and my latest addition would be shell scripting.

Slightly tangent to the topic are the markup languages I have picked up as well, html and css in high-school and LaTeX at ITU. I dabbled around for a while with both creole and markdown, but that didn’t last long.

Editor / IDE

My first and foremost tool of choice given nearly any situation will be (g)vim. The only two exceptions I can think of off the bat is Java (for which I use Eclipse and if I need to write a whole lot of text, with minimal distraction (more on that later).

The pragmatic programmers recommend learning one text-editor, and learn it well. If the name of that editor is vim, emacs, kate, gedit, or whatever, I really don’t care. Just pick up one that fits you, and LEARN IT WELL!

I have extended vim with a couple of plugins, the most prominent being NERD Commenter, matchit, snipMate and sparkup. There are at least two more plugins, but I will write more about those later.

And for Python, I usually install the IPython interactive prompt as it is a fair bit more useful than the standard python-prompt.

Version Control

While studying at ITU I had my eyes opened about the wonderful concept of version control.

I was first exposed to SVN, and while quite capable, I figured it was too much of a hassle to set it up myself, since that would require the presence of a server somewhere to host the SVN repositories.

But then mercurial entered the stage. Git or bazaar would have done the job just as good, but the people orchestrating the fourth term settled on mercurial, and it is so dead simple and still powerful enough for what I need that I haven’t had a reason to look elsewhere.

Issue tracking

For a course at ITU I tried using Mantis, a web-based bug tracker written in PHP, and while it worked well, it was a hassle to manipulate bug reports since it meant I’d have to go online and log in to yet another system.

I have however found a different solution which I am currently trying out: a plugin to mercurial called b with the tagline “distributed bug tracking”. It is a bit too early to tell if it will do, but for the time being it solves the immediate problem of having to go online somewhere to handle bugs.

Next post in line: “Office Suite” software

:wq

Tags: , , , , , , , , , , , , , ,
Posted in English, GNU/Linux, Tools | Comments Off

Scripting NetworkManager

Sunday, May 2nd, 2010

At home I have a laptop acting as a central mercurial server (from here on referenced as myserver). This is great, being able to synchronize my desktop and netbook through an intermediary gives me three hard drives on which my data is stored. The setup was, however, not especially useful since my netbook goes with me almost everywhere (read: outside the home network).

This presents a problem as when I clone a repository and my .hg/hgrc file now contain a link to the server using at best a name defined in my /etc/hosts file, and at worst a local IP address (192.168.0.xyz).

I had long since done away with the IP address links, replacing them with hostname links (i.e. ssh://hg@myserver/repo instead of ssh://hg@192.168.0.xyz/repo) which was the first step. Now all I needed to do was make myserver accessible through the firewall (forwarding the appropriate port (since we are talking ssh here that would be 22, unless you are paranoid and have switched the port which sshd listen on) in the router, make the server listen on that port, and finally come up with a way to dynamically switch what IP address myserver should point to.

There probably is a better solution for this, configuring the router to make all this would be one way, but I’m not sure that my router has that capability, and even if it did, I’m not sure I would have the skills to configure it. Software and shell scripting on the other hand, those are things I know how to work.

My solution, was simply to create two new hosts-files in /etc :

# touch /etc/hosts-{home,not-home}

and then insert appropriate information into them such as

# echo "myserver    192.168.0.xyz" > hosts-home
# echo "myserver    aaa.bbb.ccc.ddd" > hosts-not-home

(where aaa.bbb.ccc.ddd is my external IP as seen on http://whatismyip.org/).

I also made a backup of my current hosts file just to be on the safe side

# cp /etc/hosts /etc/hosts.bak

So, the overall plan is that once I’ve identified if I am at home, or somewhere else (i.e. not home) I will overwrite (copy) /etc/hosts with the “correct” hosts-file (-home or -not-home).

And this is where NetworkManager comes into play. Every time there is a change in the connection status of any identified network interface, NetworkManager reacts by executing every script found in /etc/NetworkManager/dispatcher.d/.

Every script is called with two parameters, the first being which interface changed (e.g. “eth0″, “wlan0″, etc.) and the second being how it changed (e.g. “up”, “down”). What I wanted to do was, every time a new interface went up, check if it had connected to a device in my home network (wlan or ethernet).

So, how do you distinguish different devices in a simple manner? For me, distinguishing different MAC addresses (two known addresses at home (wlan and ethernet), and every other address was essentially not home) would do nicely.

Enter the “arp” command. From it, using flag “-i” (for interface) I can discover the IP and MAC address  of the device I’m connected to. When I first discovered the example usage used flag “-a” as well which, according to the man-page, outputs results in an alternate BSD style format, using no fixed columns. This had some benefits, but as I read it again now, the “no fixed columns” remark begin to worry me. The fact that it has worked so far isn’t good enough proof that it will always work. Needless to say, ymmv.

Putting it all together:

#!/bin/sh
# filename: /etc/NetworkManager/dispatcher.d/99switchHosts.sh

# Unless the status is "up" I'm not interested
if [ "$2" != "up" ];
then
    exit 0;
fi

# I am interested in two interfaces, eth0 and wlan0
# each connect through different IP addresses and
# these devices have separate MAC addresses
if [ "$1" == "eth0" ];
then
    TARGETIP="192.168.0.1"
    TARGETMAC="aa:bb:cc:dd:ee:ff"
else
    TARGETIP="192.168.1.1"
    TARGETMAC="12:34:56:78:90:ab"
fi

# Sometimes arp hasn't gotten hold of any IP/MAC addresses
# pinging and waiting does wonders for this
ping -c 1 "$TARGETIP" 1>/dev/null 2>&1
sleep 5
CURMAC=$(arp -ai "$1" | grep "$TARGETIP" | awk '{ print $4 }')

while [ -z "$CURMAC" ];
do
    # Again, arp might not have gotten hold of IP/MAC
    sleep 3
    CURMAC=$(arp -ai "$1" | grep "$TARGETIP" | awk '{ print $4 }')
done

# If MACs match, it is a known address, i.e. home
if [ "$TARGETMAC" == "$CURMAC" ];
then
    cp /etc/hosts-home /etc/hosts
else
    cp /etc/hosts-not-home /etc/hosts
fi

exit 0

This should be put in /etc/NetworkManager/dispatcher.d/99switchHosts.sh (and the script should be made execute:able)

# chmod +x /etc/NetworkManager/dispatcher.d/99switchHosts.sh

All done, not too shabby eh?

Tags: , , , , , ,
Posted in Documentation, English, GNU/Linux, Tools | Comments Off

mercurial-server

Saturday, July 18th, 2009

I have just found a pretty cool way to set up a centralized mercurial server called “mercurial-server”, but unfortunately I haven’t got it all quite working yet. Installation was dead simple:

$ cd src/tarballs
$ wget http://hg.opensource.lshift.net/mercurial-server/archive/release_0.6.tar.gz
$ cd ..
$ tar -zxvf tarballs release_0.6.tar.gz
$ cd mercurial-server-release_0.6
$ sudo ./install

With that done you need to upload your ssh public key (i.e. ~/.ssh/id_rsa.pub on your local machine, not the server on which you are installing mercurial-server) and place it in /etc/mercurial-server/keys/root/[username]/[hostname_of_local_machine], although I am assuming that the extra directory ([username]) and naming your pubkey after your local hostname is for ease of administration only.

I had set up some rather restrictive rules about what groups could enter the server by ssh, in my /etc/ssh/sshd_config file, which  effectively thwarted my first attempts at doing anything (say cloning the hgadmin repo) by ignoring my pubkeys and asking for the password to the hg user. I was rather stumped at that for a while until I remembered my sshd_config being… paranoid. ;)

With that out of the way I could attempt a hg clone ssh://hg@jupiter/hgadmin, which failed miserably with a stack trace indicating that I didn’t have the python-dispatch package installed. I tried Googling for a list of dependencies, but the name “mercurial-server” doesn’t seem to be well established with Google yet, so most results end up with questions about “mercurial” and “server” (placing mercurial-server inside quotes, it does nothing!). Anyhow, one sudo apt-get install python-dispatch later and I got a new stack trace:

$ hg clone ssh://hg@jupiter/hgadmin
remote: Traceback (most recent call last):
remote:   File "/usr/local/share/mercurial-server/hg-ssh", line 89, in <module>
remote:     try_cmd(os.environ.get('SSH_ORIGINAL_COMMAND', '?'))
remote:   File "/usr/local/share/mercurial-server/hg-ssh", line 56, in try_cmd
remote:     dispatch.dispatch(['-R', repo, 'serve', '--stdio'])
remote:   File "/var/lib/python-support/python2.5/mercurial/demandimport.py", line 75, in __getattribute__
remote:     return getattr(self._module, attr)
remote: AttributeError: 'module' object has no attribute 'dispatch'
abort: no suitable response from remote hg!

This was, as I realized when installing the mercurial-server temporarily on another much more up to date system, due to an outdated version of mercurial installed on the system. I really ought to do a dist upgrade or a complete reinstall of the entire box now while I don’t have much of value on it, but I opted for the less time consuming way of just uninstalling the old (repository version) mercurial and installing from source. And now all is well :D

All there is to it now is experiment by managing the server from its hgadmin repository and see if it is as awesome as it seems.

The one really nifty thing about this is that this way I could expose my repository to people I know and trust simply by adding their ssh pubkey, without having to worry about giving them a shell on the server. Very cool.

Tags: , , , ,
Posted in Development, Tools | Comments Off

My software stack

Friday, May 29th, 2009

A week or so ago I stumbled across this blog, which went almost instantly into my RSS feed, due not only to the name of a post which cracks me up (yes, I know my humor is off ;P) but also to the posts I found really interesting.

And then I came along this post which got me thinking about what software I ended up using towards the end of my bachelors. Or the software I have learned of since, but wish I’d known about earlier. I began to write a comment to her post, but realized that it would be too long, so I write here instead. All credit to Hazel though, since without her post I wouldn’t have been inspired to write this one.

My list, as compared to Hazels, will not be as well-rounded, it won’t necessarily fit every student the way her list do. Also, the software I list will only be guaranteed to work in GNU/Linux, as that is what I used in the final semesters, and have continued to use since.

First of all, a text editor. It doesn’t really matter which, just evaluate a bunch until you find one you feel comfortable with. Once you have found “the one” become intimate with it. Become a frakking Jedi-master at wielding it. I’m still a padawan-level user of Vim, but I’m getting there.

I say the same about web browsers, mail clients and instant messaging clients. Find a good one, learn as much as you can about it, and use it effectively. Firefox, Thunderbird and Pidgin are my preferred tools.

A bug-tracker, although often web based creating a need for a web server, can often provide more “good stuff” than just tracking bugs. Stuff like statistics, or, if you think outside the box you’d be able to track things other than bugs, which I guess it was issue-trackers does. Some of these also include a wiki-system, which makes establishing a project-specific knowledge-base kindof easy. In the one university project where we used such a system (and where I realized its potential) we used Trac.

A blogging-system with an RSS-feed capable of being filtered on tags or categories could be used to distribute status updates to other members of a group. That I’m using WordPress should be fairly obvious to all.

Use a version control system wherever and whenever possible. With the next two suggestions on the list, “wherever” will be a lot more commonplace than one might first believe, even for non-programmers. At the university we had access to SVN-servers, and also tried Mercurial, a distributed vcs. Mercurial stuck with me ever since.

From generic suggestions, let’s go specific.

I could encourage you to check out markup languages such as reStructuredText or Markdown, to find one which suits you best and to run with it. And since I’ve now written the terms you’d need to Google, you could do that, but I’ll simply recommend LaTeX. The reason for markup languages in general, and LaTeX specifically is that you can then store your information in one plaintext format (which makes it easy to manage in version control) and can then transform it to a slew of other formats as needed.

Most of the time we needed to hand in PDFs. LaTeX excels in that and manages all the typesetting stuff and (obvious) formatting. Which leaves you with more time to focus on the content. One could also either extend LaTeX with Beamer, to create presentations, or simply generate a PDF and run Impress!ve.

For diagrams, graphs and flowcharts or representations of state-machines, Graphviz would be my recommended way to go. Again using plaintext to control the content, again with the benefits of version control. Inkscape saves files in the SVG format (again, plaintext) which might be usable (especially since it can also save files as both PS and PDF)

If you need graphical representations of statistical data or other plots, matplotlib could be the way to go.

I personally don’t like managing things, or management-related stuff, but lately I have been haunted by the feeling that if I used management tools, even if I would only be managing myself and my pet projects, I could be more organized and efficient. So I have started looking at TaskJuggler. It is similar to Microsoft Project, with the largest difference being that… you guessed it, you code the project plan ;D. Plaintext yet again. And then you compile the plan and TaskJuggler attempts to verify that no resources have been double-booked.

Considering each piece in this list on their own, it might seem like a waste of time to exchange one software with another. I do find each of these softwares impressive in and on their own, but it is when they are put together, when all their strengths are combined, that you tend to get the most out of it.

The all plaintext approach I have tried, both in groupwork at the university, and later on my own, work rather well. That so many of the softwares on the list can be used to communicate and transfer information between parties is also intentional as without communication the chance of a successful project outcome diminish rapidly.

The last (bonus?) item on the list would be to recommend learning, at least superficially, a programming language which you could hack together small scripts with. Something which you could use to “glue” together the other parts. I adore Python, and many of the softwares listed above have python-bindings ready to use. Perl, Ruby and others, which elude me right now, would undoubtedly work equally well or better, but as with the text editor, pick a language you feel comfortable with, and rock on.

Thoughts? Questions?

Update: Fixed broken link

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in Blog, Documentation, GNU/Linux, Programming, Tools | 4 Comments »

Mercurial and hooks

Thursday, February 19th, 2009

I found myself today with a problem. I have a development server on which I run tests and build things. It as of today also houses a new mercurial repository. Inside it, a bunch of PHP-files. My original idea was to link the needed files from the repository into the wwwroot. This of course will not work as no complete files (to my knowledge) is stored inside the repository. So then, after having committed, I would want the repository to push the new changes out to a local clone, which I could then link to from the wwwroot.

This was actually fairly easy. Inside the repository you find a hidden directory “.hg”. Within it there should exist a file “hgrc” (it didn’t in my case so I created it).

My first attempt, following these instructions didn’t quite work out. I don’t really know why, but checking the local clone made evident that it had not updated as it should have.

What I tried was:

[hooks]
changegroup = hg push /path/to/clone

which left me with an error message on the client “abort: unexpected response: ‘pushing to /path/to/clone/[repo-name]\n’“. My next attempt was to use a shell-script instead. The second attempt failed also, this time because I stuck the shell-script inside the .hg directory, and tried to call the script with a relative path from hgrc (I guess hg isn’t executed from that directory so it fell flat on its face)

Third and final attempt, the same shell-script, moved to a directory on the $PATH, and I push from my remote (workstation) repository. The client still receive an error message: “abort: unexpected response: ‘pulling from /path/to/repository/[repo-name]\n’“, but at least this time the clone on the server has been updated.

The shell-script was a quick and dirty hack:

#!/bin/sh
cd /path/to/clone
hg pull -u
exit 0

but worked like a charm. This is in no way extensible (although I guess one could make it work iff the hook-scripts are named carefully, but it would be a much better solution to have each project specific hook located inside the project repository instead…

Anyway, my Google-Fu fails me in my searches for how to get around the client error message. It obviously isn’t aborting since the clone, pulling from the server, is getting the changes. If you know, I’d be happy to hear from you.

Update:

My Google-Fu eventually came through, and I found this conversation in which the proposed solution worked superbly. My hgrc now look like this:

[hooks]
changegroup = /path/to/shell-script > /dev/null

Tags: , , , ,
Posted in Development, Personal, Programming, Project, Tools | Comments Off