When walking back to Lars’ place Thursday evening, I spotted Holger Danskes vej, which reminded me of, well, Holger Dansk, but more importantly, Peopleware. Peopleware is an amazing book about making software projects work. The meat of the book is that most projects don’t fail because of technological reasons but sociological: unhappy people. Joel Spolsky hits the nail pretty well by calling Peopleware an “anti-Dilbert manifesto”.

I know. I’ve written about Peopleware before. But hey, I’ve read the book many times and still find it as eye-opening as ever. So if you’ve so far been able to find excuses for avoiding the purchase, give up. You won’t regret it.

If you’re already in Copenhagen for Building of Basecamp or reboot7, be sure not to miss the Blogger dinner tonight at Cafe Ketchup in Tivoli. 8 o’clock sharp, followed by some fireworks.

I could write a whole entry complaining about the (nonexis€ac support of the two Finnish [sic!] makers of heart rate monitors and sports wrist computers, Polar and Suunto. I’ll save that for later, however.

I felt a sense of hope when I found Polar Running Coach, a web-based software for maintaining the data collected by a Polar monitor. Maybe — just maybe — they would have been able to create an applet that is able to command the USB infrared port used to get the data out of my Polar S710. Unfortunately, I never got far enough to find out.

Someone at Polar seems to have thought it’d be neat to use Macromedia Flash for the registration process of the Running Coach. It’s not. It’s stupid as hell. It totally highlights the biggest usability and accessibility problems in Flash.

“But hey, 95% percent have Flash player, how could we possibly go wrong with? Besides, with it we can build as great-looking drop-down widgets as we want to. How cool is that?”

Here’s how my registration ended:

Now how cool is that? Especially after spending about ten minutes filling in information about myself. Note that there are no links I could use to back up to the previous step. And as the wizard is made in Flash, I can’t use my beloved back button because it will throw me in vaults out of the whole registration process. Complete bummer.

Oh, but we still have the cool drop-downs, don’t we? Using the standard html form widgets I can type “f”, “i” and “n” after I click on a country drop-down menu and I can be pretty sure I have Finland selected. Here, I end up to Namibia (via Falkland Islands and Iceland). You know, sometimes reinventing the wheel just results in an inferior wheel.

It goes without saying that I don’t really feel like trying again, despite the pretty please. However, the outcome is that I have a €350 heart rate monitor but am unable to download and analyze the data collected with it. The Mac OS Classic version of the old software (that I’ve never been able to launch on OS X) doesn’t really cut it.. C’mon Polar, you’ve always been a forerunner. Please start taking us other pioneers into account, too.

So here it is, the first ever Railsday 24h coding contest, starting tomorrow at 8AM Finnish time. Unfortunately I won’t be participating since I will be fighting for Finnish championship in sprint orienteering at the same time. Luckily the organisers have already announced that there will be rerun so maybe I will be able to cut my teeth some day, too.

On other news, I will fly down to Copenhagen next Wednesday. What a three days it will be. I’ll first take part in the first ever Building of Basecamp workshop in Europe. After that it’s time for Reboot7, a “meetup for the practical visionaries who are building tomorrow one little step at a time”. Besides making new contacts I will also meet some old OpenACS friends like Lars Pind and Guan Yang for whom I have a great respect. Can’t wait to get on the plane :-)

A few weeks ago I wrote about starting your lighttpd process automatically on server reboot. I promised then that I’d come back with a solution for keeping the process up even if it dies. Well, I’m about as late as it gets, but here I am anyway, with a hopefully helpful bit of advice.

Meet daedalus.rb

Daedalus is a pure-ruby script for monitoring processes and restarting them if needed. It’s part of FreeBSD sysutils but to be honest it’s really hard to find these days. Fortunately Jason Hoffman of TextDrive has graciously put up a copy on their Subversion server so your best bet is to just go and grab it from there. Then just unpack the archive in your home directory.

~# wget http://svn.textdrive.com/repos/scripts/daedalus-1.2.tar.bz2
~# tar jxvf daedalus-1.2.tar.bz2
~# cd daedalus

Now you’ll have a subdir called daedalus in your home directory. If you look around, you’ll find a few interesting files in there. daedalus.rb is the actual ruby script that does the monitoring. To save some hassle later on, open the script and change the locations of log and pid files to something residing under your home folder (around line 362):

$options['logfile'] = '/home/[YOU]/var/log/daedalus.log'
$options['pidfile'] = '/home/[YOU]/var/run/daedalus.pid'

Be sure that those folders exist, too ;-)

After this, you’re ready to start configuring your daedalus. In examples you have a file called daedalus.conf which you can use as a template for your own configuration. I created another subfolder config under daedalus and put my config files in there. Open up the config file in the editor of your choice:

~/daedalus# mv examples config
~/daedalus# emacs config/daedalus.conf

Inside the config file you find a few configuration examples separated by empty lines. You can comment them all out since we’re not going to need them. Instead copy this to the beginning of the file:

name: lighttpd
checkcommand: /bin/ps axww
checkregex: /lighttpd/
onfailcommand: /usr/local/sbin/lighttpd -f /usr/home/[YOU]/sites/rails/config/lighttpd.conf
checkinterval: 30
aftercommandwait: 15

Here’s what the above means:

• name is just a name for the configuration.
• checkcommand is the command daedalus executes in order to find out if your target is working or not. In local monitoring it’s most often “/bin/ps axww”. Be sure to include the two w’s in there, you’ll be sorry one day if you don’t¹.
• checkregex: this is the regular expression daedalus will try to find from the output of checkcommand. So /lighttpd/ means that we’re looking for…er… lighttpd. Brilliant! If you’re not yet familiar with regular expressions, here’s a resource with a tutorial that might help you.
• onfailcommand: if the given expression is not found, daedalus will execute this command. In our case this will be the command you’re starting lighttpd with in the first place.
• checkinterval: this is the polling interval in seconds. So 300 would mean that daedalus checks for this particular process every five minutes.
• aftercommandwait: how long (in seconds) daedalus will wait after a failed check and relaunch. If you know that your process takes a while to launch, you might want to shoot this up a bit.

Modify the config file to your needs and that’s it! You are ready to launch daedalus for the first time:

~# /usr/local/bin/ruby /home/[YOU]/daedalus/daedalus.rb -c /home/[YOU]/daedalus/config/daedalus.conf

You can now take a peek at home/[YOU]/log/daedalus.log with tail -f and see what it’s doing (if anything). Try to kill the process you’re monitoring and check that daedalus is able to restart it. After that make sure that your daedalus daemon is started on every bootup, otherwise it won’t help you very long (part I of this tutorial will tell you how to do that).

But what if it just freezes?

There can (and will, sooner or later) be situations when your web server just hangs. It seems to be running so daedalus won’t notice anything but it doesn’t serve the pages it should either. Now we’ll get real pragmatic and see how daedalus can help us to solve even this problem.

Create an uptime test action

(This advice is for Rails but you can easily emulate it in any environment.)

Create an action in a suitable controller that will somehow test that the db connection works and then return “success”. I e.g. have an action uptime in my PageController:

def uptime
      render_text "success" if @pages = Page.find_all
end

After you have something along those lines working in your web service, add another configuration to your daedalus.conf file:

name: lighttpd-external
checkcommand: /usr/local/bin/curl http://www.yourdomain.com/page/uptime
checkregex: /^success$/
onfailcommand: killall lighttpd; /usr/local/sbin/lighttpd -f /path/to/your/rails-app/config/lighttpd.conf
checkinterval: 300
aftercommandwait: 30

Configure it according to your needs and restart the daedalus daemon. After that you should be fairly safe with lighttpd.

Getting really paranoid

All Black Sabbath fans can take this approach one step further. Start another daedalus daemon (with another config file) that monitors the first one and starts it up again if needed. Then add another configuration to the first instance and voilá, you have a redundant array of daedalus processes monitoring not only your web services but each other, too. Implementing this will be left as an exercise for the reader.

That’s about it. If you have followed through part I and this second take, your lighttpd should survive quite a lot. If you have recommendations for making this approach better, please leave a comment.

Thanks to Jason Hoffman and Dave Goodlad for help in setting daedalus.rb up initially.

¹ ps ax will only print certain amount of letters to every line. If your command is very long (like lighttpd command often is), ps ax might cut the line which can cause the following regexp check to fail. I spent quite a bit of time tracking this down. The two w’s will cause ps to output even very long lines so using them is a good precaution for later headache.

There’s risen a helluva discussion about Google’s new service, Google Web Accelerator. The main point of the discussion is that GWA preloads all the links on a web page (even a password-protected one), possibly deleting items in a web app or otherwise wreaking havoc.

What is Google Web Accelerator?

GWA is a service that preloads all the links on a web page a user requests (note that this is not what their Webmaster Help indicates or what many people claim). This will theoretically (and also in practice, in many cases) make surfing a lot leaner for the user because a page behind a link is already preloaded and she will get it in front of her face in an instant when she clicks the link.

Clouds in paradise

The problem with GWA is that it follows the user everywhere, even to password-protected admin sections of web apps that have traditionally been safe harbour from e.g. web crawlers clicking links they shouldn’t. It seems that only links GWA will not prefetch are SSL-encoded (i.e. https://) pages. Unfortunately most of today’s web applications can’t use SSL because of added certificate and hardware costs.

When the user then loads a CMS admin page with a delete link for each article, GWA happily follows each link leaving you nothing but an empty system. Not exactly what you expected a “web accelerator” to accelerate.

So who’s to blame

It has been a known (but often forgotten/ignored) fact that GET method (i.e. normal links) shouldn’t be used to change a state of a web app. However, as Simon Willison quotes in a blog entry, “[SHOULD NOT] mean[s] that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful”. The fact that a link has been on a password-protected admin page has widely been considered as one of those “particular circumstances”.

It’d be easy to run around bashing the stupid developers but I doubt many of us is really pure enough to throw the first stone (even if they think so). Even if it’s been stupid and relentless to use links to perform critical actions, it’s just the fact of life. No matter how much malicious pleasure the current situation might give certain people, it just doesn’t help anyone very much to put all the blame on the application developers. It’s a total pipedream that all the developers in the world would wake up overnight and go fix their “broken” apps. And the one who suffers the most is always the end user who might lose some invaluable content in the process.

Lessons learned

This has been a good reminder for us that we should take the security considerations seriously even if we don’t store mission-critical data in our apps. Simon’s post reminds that even if using POST forms for actions like deleting items might save us from Web Accelerator it doesn’t protect us from cross-site request forgery attacks. They are really not a part of this post but it’s a good example of how we can learn more from these issues than what the first impression might suggest.

But as web developers have a lot to learn from this, so does Google. It doesn’t operate in a vacuum. It doesn’t operate in a perfect world. It doesn’t operate in a web where everyone obeys the rules to the letter. They should’ve known better. But no hard feelings, if they only take this to the heart: you need not only cooperate with the standards. You also need to cooperate with the world surrounding you.

P.S. There’s also another issue with GWA, Google storing possibly extremely private data on their servers (as Thomas Baekdal notes in the comments of the SvN post), but that’s a topic for another post…

P.P.S. There’s already a trick for protecting your Rails actions from GWA.

So I stumbled upon a problem with my Rails app. Or with MySQL, to be precise. I needed to use date() functions in a query to compare timestamps. I know, I could’ve done it some other way but I like to keep things simple. The only problem was that my laptop was running MySQL 4.0 which (imagine that!) didn’t support date().

I hesitated a bit if I wanted to go through the update process just for this but because I from now treat problems, not symptoms, I decided to dive straight into it.

I installed the latest MySQL with DarwinPorts and moved my old databases to the new folder (this wasn’t what I initially tried to do but it was how I got things working in the end. Besides, fighting with MySQL isn’t, despite the headline, the meat of this story). Then I changed the StartupItem in /Library to point to the new mysql.server script and the server was up.

The problems started (oh well, continued) when I tried to use my Rails app. I got the same errors as when I tried to use the old mysql client with the new server. I finally figured out that the reason for this was that the MySQL/Ruby bindings were compiled for the old server. I banged my head against a wall for a while before I found out from Jason’s OS X tutorials that gem install needs two extra dashes before giving the configure options to it. Finally I got it all working so here’s the command you need to get MySQL/Ruby bindings for a DarwinPorts-installed MySQL server:

sudo gem install mysql -- \\
--with-mysql-include=/opt/local/include/mysql/ \\
--with-mysql-lib=/opt/local/lib/mysql/

I’ve long suspected that David Heinemeier Hansson is some kind of mindreader. This morning, when Rails 0.12 came out, I’m even more convinced about that.

Eager loading of associations.

This is a very cool addition to the arsenal of ActiveRecord. You can now select which associated objects get loaded automatically when you use find to get a bunch of things from the database.

Example:

for model in Model.find(:all, 
                  :include => :products) do
  puts "Title:            " + 
     model.title
  puts "Number of models: " + 
     model.comments.size
end

In the olden days, this kind of code would have meant N+1 sql queries where N is the number of models. And now? One. Uno. Ett. The magical include clause in the find call causes Rails to automatically craft a join query that fetches not only the model objects but also all the product objects associated to them.

Even better AJAX support

This is what got me thinking about DHH’s supernaturality. Last week I was playing around with the new AJAX stuff and noticed that there was no way to specify the “normal” href attribute in the AJAX-powered links and forms, for those people who don’t have a browser supporting ajax (mainly people with disabilities). Well, I didn’t even have time to write a ticket about it when it’s already fixed.

Not convinced yet? Ok. How about this. I’m writing a simple intranet for our club and thought that it would be cool to have a simple bulletin board on the front page where people could just leave quick, non-threaded messages with AJAX. The problem was that even when the writer can see her note in no time, others can’t. So periodically_call_remote to help, updating the given part of the page periodically.

OK, but stop listening to me. Go read the full post about the enhancements yourself.

Reboot 7 in Copenhagen has now been officially announced and what a happening it will be! The speaker list consists of such guys as Doug Bowman, Cory Doctorow and Jimmy Wales, not to mention Rails and Basecamp heroes David Heinemeier Hansson and Jason Fried.

I also got an unofficial confirmation from David that they will be giving the Building of Basecamp workshop the day before Reboot. Geez, I have been waiting for that announcement like Christmas. Can’t wait for the day the registration begins. If the success is anywhere near the previous workshops, there’s not many days to spend before the gig is sold out.

Unfortunately I have a wedding on Saturday that I really have (and want) to attend, so I will miss the second day talks and the Saturday night ceremonies. Even then the whole show sounds like something I won’t forget very soon.

Wednesday really seemed to be too good a day for me as the following days have been pretty painful. It seems that I’ve been carrying a dermoid cyst (a kind of benign tumor) under my coccyx whole my life but this week of all time it decided to inflame, probably after it got irritated by some heavy (non-xml) sit-ups during a work-out. I tried to bite it and run last night a training with the help of some painkillers. During the night I had problems to get sleep because of the pain and after I caught quite high fever this afternoon I had to accept the fact that I won’t be participating any competitions this weekend. With the help of Maria’s father I got a time in the local emergency room and they literally cut a hole in my butt.

After some serious yelling and pain the tumor was emptied of about a litre of pus (or so it seemed. I wasn’t able to see the operation, for obvious reasons) and they plugged the hole with a roll of gauze tissue so that all the matter can exit the tumor in the next two days. Needless to say, I don’t really feel like sitting in an office chair right now so I guess I’ll be catching on my backlog of reading and movie watching this weekend.

Take care.