A nasty bug, with only a brief workaround available.

After installing mongrel_cluster, my dev. instance for an older project failed to startup with mongrel, although it would with webrick.

After some time chasing down the problem, I located this report of the bug, last month.

The short of it is, uninstall ruport, all versions. Mongrel is requiring the ruport-0.8.11 gem, which does not have an init.rb file. So things fail. I'm not sure of the long-term fix.

Apache 2.2.4 and mod_proxy_balancer

At a first glance, the idea of setting an Apache load balancer before a mongrel_cluster did not seem too unreasonable. I'd used Apache frequently and, while not a hardened sys-admin, have configured my fair share of web servers. Moreover there were a number of blogs which discussed the topic - codahale, davidwinter, and many others - whom had done something similar ... Alas, it took a little more to configure on the AMD 64 vps that Slicehost (generously, I like slicehost) offer us.

This configuration will give you an Apache fronted Mongrel cluster. Apache balances the load between the mongrel servers. It's that simple:

<Proxy balancer://mongrel_cluster>
  BalancerMember http://127.0.0.1:8000
  BalancerMember http://127.0.0.1:8001
  BalancerMember http://127.0.0.1:8002

Again, most of this is lifted from codahale (thanks).

Before you proceed and invest time into this configuration, please note this caveat. I'm experimenting with this configuration in a production environment - I obviously believed in it enough to chase it. I'll be willing to discuss it with my fellow roroers on IRC, but won't otherwise be offering advice or support on it. If you've compiled Apache from source before, and have configured it, and have an idea about general deployment to a cluster and load balancing scenario, you should be able to work it out. If you can suggest an improvement to the process, please let me know.

The solution below is really just me reading through the blogs mentioned above and working things out. What specifically did I have to work out for this configuration?

• apt-get for Ubuntu only pulls back Apache 2.2.0. Unfortunately, for mod_proxy_balancer you will need Apache 2.2.4, at least. While there are some Ubuntu package repo.s which seem to allow 2.2.4 to be installed via apt-get, nothing exists for the AMD 64 architecture which Slicehost uses. You're compiling.
• Configuring the Apache modules to enable mod_proxy_balance was painful. The solution ended up being to ensure that make was configured with the --enable-mods-shared=all flag. There are bugs when compiling Apache 2.2.6 with the same prerequisite. I painfully encountered them, before working out that I had to backstep. The good news is that they were acknowledged as critical (so sue me, I wasn't keeping track of every bug I had to step around, but 2.2.4 is solid, if you follow the steps below).

As all of the blogs mentioned above will tell you, you have to remove existing Apache installations from Ubuntu:

sudo dpkg --purge apache apache2

Then ensure you have all of the regular C compiler tools ready to go:

sudo apt-get install build-essential

Then, you must download zlib 1.2.3 and compile it. Another caveat, which might apply to the amd 64 arch. (don't really know), you must set a CFLAGS arg on the zlib compilation to cooperate with Apache 2.2.4. Thus

wget http://www.zlib.net/zlib-1.2.3.tar.gz

Compile zlib 1.2.3 with the CFLAGS arg. below. Otherwise compilation of Apache 2.2.4 will fail.

rails@scene:~$ cd zlib-1.2.3/
rails@scene:~/zlib-1.2.3$ export CFLAGS="-O3 -fPIC"
rails@scene:~/zlib-1.2.3$ sudo ./configure
Checking for gcc...
Building static library libz.a version 1.2.3 with gcc.
Checking for unistd.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf()
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for errno.h... Yes.
Checking for mmap support... Yes.
rails@scene:~/zlib-1.2.3$ sudo make
... to sudo make install

Pull down Apache 2.2.4 (it's available from apache.org, you just have to look at older downloads - http://archive.apache.org/dist/httpd/ . I downloaded the highest versioned tarball of 2.2.4 .

Make sure you reset the CFLAGS to a blank string before compilation -

export CFLAGS=''

Then configure with all-mods-enabled, as well as specifying the mod_proxy_balancer. This is probably overkill, but it works. I am running Apache from /opt/apache2:

sudo ./configure --prefix=/opt/apache2 --with-included-apr --with-mpm=worker --enable-modules=all --enable-mods-shared=all --enable-ssl=shared --enable-proxy=shared --enable-proxy-ht        tp=shared --enable-proxy-balancer=shared

Again, make and make install. If you hit an error when running make, asking you to recompile with -fPIC flag, go back and check your compilation of zlib 1.2.3 again, noting the CFLAGS requirement.

This is all that is required to install an Apache 2.2.4 deployment which will work with mod_proxy_balance.

I took most of my Apache load balancing configuration from codahale, although, because I had to compile Apache from scratch, the Apache configuration lacked a conf.d, and I worked most of it into my VirtualHost for httpd.conf .

Using Capistrano and mongrel_cluster

Again, most of my configuration came from reading the codahale blog (ibid). The only caveat I found is that the mongrel_cluster config. is particularly stupid about writing it's pid files to non-existent directories (i.e. it won't create them).

My mongrel_cluster thus looks like:

cwd: /home/rails/apps/super_secret_app/current
log_file: log/mongrel.log
port: "8000"
environment: production
address: 127.0.0.1
pid_file: /home/rails/apps/super_secret_app/shared/pids/mongrel.pid
servers: 3

I had to ensure I created the shared/pids directory , otherwise the three node servers would not start up via capistrano2.

That's it. I haven't repeated the steps explained in the referenced blogs. The main difficulty was compiling it on the Slice machine. But it was worth it, I only to have type 'cap production deploy' from my lappy and I have a three node and load balanced cluster solution deployed on another continent in a couple of minutes.

All I can say is this was a nasty, irritating, soul-destroying bug to find.

As I understand it (and I haven't really tried to), there are certain conditions when using virtual attributes (using the term quite lightly, basically an attribute which is transient, or not reflected into the db table) on a model when a method missing implementation will try to read the value of the attribute. So, the solution is to insert a different method_missing implementation earlier in the invocation chain.

Please note, this is not my own work. I found the the solution at this blog. And, apparently, that blog found it in the Rails Recipe book.

For the sake of posterity, in case the 'Made of Stone' blog disappears, I'm going to post the fix here also.

 def method_missing(symbol, *params)
   if (symbol.to_s =~ /^(.*)_before_type_cast$/)
     send $1
   else
     super
   end
 end

Problem: Running a system call from an object in a Rails thread - if the process outputs to stderr or stdout, and you have not set up IO buffers to hold the bytes, the process will stall until it times out. e.g.

def publish
...
  @gallery = fetch_gallery()
  system("script/generate gallery_creator #{gallery.id}") 
...
end

If the generator outputs to stderr or stdout, the process will stall. The solution is Open3 (thanks lachie).

 
def publish
...
  @gallery = fetch_gallery()
    Open3.popen3("script/generate gallery_creator #{@gallery.id}") do |stdin, stdout, stderr|
      logger.info("stdout - #{stdout.readlines}")
      logger.info("stderr - #{stderr.readlines}")
    end

...
end