If you’ve recently set up a rails app, you’re probably seeing a build-up of FastCGI processes. (If you’re not sure, run ‘top’ from the shell to view them.) Here at DreamHost you should see from about 1 to maybe 5 or so for a well-behaved app.

I’ve seen that on the mail list from DHH that the key to eliminating the rogue processes is to eliminate bugs from your code (see the last line of the reference above). But it took me going through the experience to really believe it.

So scour your production log for any possible errors or exceptions. I’ve had issues even with little things – for example, my 500.html page had a stylesheet with a relative link, something like “stylesheets/text.css”, and when a 500 error occurred from a url other than a top level, e.g. “admin/registrations” that stylesheet couldn’t be located. Changing the link to something like “/stylesheets/text.css” fixed it. I was getting a mysterious rails ‘routing’ error referencing the stylesheet. Any error seems to cause a fastcgi process to go south, but not go away.

Once I got rid of all the little (and big) errors the processes really settled down.

For some reason I’ve had to look this up twice recently and found it hard to find, so I’ll add it here. If you want to get a reasonable, human-readable version of a timestamp, here’s probably what you want:

obj.created_at.to_formatted_s(:db)

ActiveSupport::Date::Conversions

If you’re using fastcgi, either here at DreamHost or elsewhere, you may need to restart the processes from time to time.

You may have an app in Production mode and need to flush the fastcgi processes so a change will show up, or if you have something in development mode and have a bunch of dead and dying processes.

Generally if I see more than a handful of processes or if I see any ‘zombie’ processes, I’ll restart the fastcgis to reset things.

killall -9 dispatch.fcgi

This will get the whole bunch of them at once.

I found this most interesting in light of the recent spat over responsible deployment. It looks like the Java folks (and I count myself among them) are hankering for a bite of the dynamic deployment pie.

Read through the use case. It is very compelling — a user gets an error in a popup with a Mail button, sends it off, developers make a fix and check it in to subversion, the user refreshes the page and the error is gone. From a Java developer’s point of view that reads like science fiction.

This sounds strangely like the process used by 37signals , “release manager,” which will be released soon.

Hey, if it works, we can all get on board. Deployment in Java could certainly use some agility. I’ll be curious to see how it goes. Unfortunately this is just a proposal for an ‘incubator’ project at apache.org, which means it’ll likely be years before it will see light of day, if it does at all.

I wonder how long till we see the Rails release manager?

I don’t know if it’s rolled out to all servers yet, but the 2 I use are both at rails 0.13.1 now. I’ve got apps running on both that are doing great. This one has an unaltered typo, as well as an updated (new dispatcher, etc.) version of clubrs.

The fastcgi seems to be settling down very nicely. You can see below that the processes seem to be managed more sanely. To be fair, the processes exiting in the top entries are probably me killing them off to roll in the new changes to the production environment.

[Sat Jul 16 17:04:44 PDT 2005] Dispatcher failed to catch: exit (SystemExit)
  /usr/lib/ruby/1.8/fcgi.rb:10:in 'exit'
  /usr/lib/ruby/1.8/fcgi.rb:10
  /usr/lib/ruby/1.8/fcgi.rb:10:in 'call'
  /usr/lib/ruby/1.8/fcgi.rb:598:in 'each_cgi'
  /usr/lib/ruby/1.8/fcgi.rb:597:in 'each'
  /usr/lib/ruby/1.8/fcgi.rb:597:in 'each_cgi'
  dispatch.fcgi:18
FCGI process 14381 killed by this error

[Sat Jul 16 17:23:11 PDT 2005] Dispatcher failed to catch: SIGUSR1 (SignalException)
  /usr/lib/ruby/1.8/fcgi.rb:597:in 'each'
  /usr/lib/ruby/1.8/fcgi.rb:597:in 'each_cgi'
  dispatch.fcgi:18
FCGI process 10349 killed by this error

[16/Jul/2005:19:39:31 :: 29193] starting
[16/Jul/2005:19:39:32 :: 8639] starting
[16/Jul/2005:19:40:48 :: 21515] starting
[16/Jul/2005:19:45:10 :: 21515] terminated by explicit exit
[16/Jul/2005:19:47:29 :: 11480] starting
[16/Jul/2005:19:47:31 :: 29954] starting
[16/Jul/2005:19:53:29 :: 29954] terminated by explicit exit
[16/Jul/2005:20:10:43 :: 2931] starting
[16/Jul/2005:23:49:17 :: 2931] terminated by explicit exit
[16/Jul/2005:23:49:17 :: 2931] told to terminate NOW
[17/Jul/2005:05:29:47 :: 28109] starting
[17/Jul/2005:05:29:47 :: 12870] starting
[17/Jul/2005:05:37:09 :: 12870] terminated by explicit exit

A Test By Any Other Name

I really like this post on testing in rails. If you follow the pattern you end up with a nice set of tests and a bit of documentation to boot.

It looks to me like this will also help keep your objects focused. The naming convention would seem to help remind you of what the particular object ’should’ do.

And getting the free bit of documentation with the rake task is very cool!

This may be obvious, but I just spent some time on it so I’ll post it in case it can save anybody else a few minutes.

I was getting the Unknown Action errors from spiders requesting a robots.txt file, as noted earlier. Since it caused an Unknown Action error, which is thrown by Routes, I was thinking I needed to add an entry to my routes.rb file in config, but I was wrong. Just upload the file to your /public directory and you’ll be all set.

So this is interesting. After getting development stabilized and setting the app into production mode, this was the one of only 3 crashes that I saw yesterday:

production.log:

ActionController::RoutingError (No route for path: "robots.txt"):
    /usr/lib/ruby/gems/1.8/gems/actionpack-1.8.1/lib/action_controller/routing.rb:291:in `recognize!'
    /usr/lib/ruby/gems/1.8/gems/rails-0.12.1/lib/dispatcher.rb:32:in `dispatch'
    dispatch.fcgi:20
    dispatch.fcgi:18:in `each_cgi'
    /usr/lib/ruby/1.8/fcgi.rb:597:in `each'
    /usr/lib/ruby/1.8/fcgi.rb:597:in `each_cgi'
    dispatch.fcgi:18

fastcgi.crash.log:

[Fri Jul 01 23:08:54 PDT 2005] Dispatcher failed to catch: exit (SystemExit)
  /usr/lib/ruby/1.8/fcgi.rb:10:in `exit'
  /usr/lib/ruby/1.8/fcgi.rb:10
  /usr/lib/ruby/1.8/fcgi.rb:10:in `call'
  /usr/lib/ruby/1.8/fcgi.rb:597:in `each'
  /usr/lib/ruby/1.8/fcgi.rb:597:in `each_cgi'
  dispatch.fcgi:18
FCGI process 12490 killed by this error

At least that ought to be easy to fix. But you can see how any little item can cause fastcgi to flail. As you get all this stuff ironed out, it seems to run pretty nicely though.

As clubrs.org has just reached it’s first release, I’ve noticed that I have a lot fewer fastcgi related issues. I think fastcgi is very sensitive and will tend to crash where you wouldn’t expect it.

For example, I had an issue with concatenating a String and a Fixnum. This barely showed up in the log under webrick but killed fastcgi. As I got things more stable and switched to production mode, the app is running much more smoothly.

http://clubrs.org

Since we have ssh access at DreamHost, I just watch top for a while to get a feeling for how things are going. If I only see a couple of dispatch.fcgi processes things are good. More than that and I can figure it isn’t running too well.

I don’t know if this is the best way to do it, but it works (you’ll need shell access). If you make changes to an object in a production environment, you can flush them through by killing off the fastcgi processes. When the new ones are spawned they’ll pick up the changed objects.

kill PID -9

To see them, I use top, and you can actually kill them off from in there by typing ‘k’. It will prompt you for the PID and signal to send.