Return of the mem leak

Having written only in general terms about how to fix the memory leak in Psych yesterday, I’ll go into a little bit more detail today.

See in Rails it really isn’t all that simple to fix it. The most obvious way to start is to add gem 'psych' to the Gemfile. But Bundler itself requires Psych and YAML. So when the Rails boot cycle gets to executing the Gemfile, it will already be too late.

Instead it’s necessary to insert gem 'psych' into one of the files that gets required before Bundler is required. It could be, config/application.rb, config/environment.rb or config/boot.rb - either one of them will do.

That’s easy enough to figure out.

The real catch

If you’re using Phusion Passenger for your Rails app, the solution above will unfortunately make your app blow up when trying to dump YAML.

The reason is that psych somehow gets required before the gem 'psych' gets run. Psych will because of this be in an inconsistent state. When starting Passenger and requesting the first page, you’ll get the following two lines in your log:

/Users/rasmus/.rvm/gems/ruby-1.9.2-p180/gems/psych-1.2.0/lib/psych.rb:93: warning: already initialized constant VERSION
/Users/rasmus/.rvm/gems/ruby-1.9.2-p180/gems/psych-1.2.0/lib/psych.rb:96: warning: already initialized constant LIBYAML_VERSION

And when trying to dump some yaml, this is what you’ll see:

ArgumentError: wrong number of arguments (2 for 1)
    from /Users/rasmus/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1/psych/nodes/node.rb:33:in `to_yaml'
    from /Users/rasmus/.rvm/gems/ruby-1.9.2-p180/gems/psych-1.2.0/lib/psych.rb:190:in `dump'

Clearly not good!

And there really is no hint in the stack trace as to where the require 'psych' happens.

To find out where Psych was required, I opened up the psych.rb that’s bundled with Ruby and raised as the first thing in that file - et voila!

The culprit was passenger/lib/phusion_passenger/utils.rb: 325 (I’ve added a couple of extra lines):

    elsif File.exist?('Gemfile')
        # In case of Rails 3, config/boot.rb already calls Bundler.setup.
        # However older versions of Rails may not so loading boot.rb might
        # not be the correct thing to do. To be on the safe side we
        # call Bundler.setup ourselves; calling Bundler.setup twice is
        # harmless. If this isn't the correct thing to do after all then
        # there's always the load_path_setup_file option and
        # setup_load_paths.rb.
        require 'rubygems'
        require 'bundler'

Now how do I get around this? Don’t bother looking in the documentation for Passenger - it doesn’t say anything about this.

But fortunately there is plenty of comments in the stills.rb file. Just scroll a bit further up.

There are 3 solutions: .bundle/environment.rb, config/setup_load_paths.rb or a file pointed to in an option called load_path_setup_file. However how this option is passed to the prepare_app_process method isn’t documented and I haven’t digged around in the Passenger code long enough to know this.

The solution I like the best is anyways using the config/setup_load_paths.rb file - this can be added to version control instead of using some Bundler dependent file and some configuration option.

If you have added gem 'psych' to one of the files loaded in the Rails boot cycle, you don’t have to put anything in the config/setup_load_paths.rb file. It just has to be present to stop Passenger from requiring Bundler.

Ps.: Phusion, why is config/setup_load_paths.rb et al an undocumented feature?! I can’t be the only one who’s needed to be able to control what is loaded before Passenger starts loading Rails app stuff?

Update: I’ve also written about what to do when running tests.