Tuesday, July 15, 2014

Thoughts from the desk of a SysAdmin: Migrating from Apache + PhusionPassenger to nginx + Unicorn

Today's topic: Migrating from Apache + PhusionPassenger to nginx + Unicorn

I recently was presented with the challenge of building a new web application server. The previous server was outdated and so far behind on updates that replacing it was the best course of action. Since the previous server (we'll call it Boe, henceforth) was functional and not in need of immediate replacement, I had time to design and build a really awesome system. First, I needed to decide how I was going to resolve the following items:
  1. Replace PhusionPassenger
  2. What to use instead of PhusionPassenger
  3. Replace Apache
  4. What to use instead of Apache
Why replace PhusionPassenger? Regardless of Apache or nginx, it made the upgrade path too dirty and hairy to maintain. PhusionPassenger also wasn't very flexible or scalable enough. With nginx, you would have to re-compile the nginx binary and re-build the RPM each time. With Apache, you only need to re-compile mod_passenger each time. That is, so long as that version of Apache is compatible. There also didn't appear to be any functionality in PhusionPassenger to support multiple versions of ruby. These are the primary reasons as to why I wanted to replace PhusionPassenger.

What to use instead of PhusionPassenger? After a bit of research and reading, I felt that Unicorn was the best solution as the application server. It allowed me to keep the applications secure so that nginx talked to it over a unix socket. Unicorn is very configurable and I like how it utilizes a master/child worker architecture and integrated easily with the web application.

Why replace Apache? For application servers, I prefer to use nginx because of its small memory footprint, caching and performance. Apache is great, in my opinion, for hosting websites or simple applications like wordpress, cacti or nagios. This was my justification for replacing Apache.

What to use instead of Apache? Well, nginx immediately came to mind. Since the web applications are heavy and need a fast front-end, nginx was the best solution. I also like how nginx buffers the response from the back-end so that slow clients won't kill the response time of the application and result in degraded performance.

After additional research and preliminary testing, I built the new system and tested out my new configurations. I ran into a couple snags though with migrating away from PhusionPassenger. The first "gotcha" being that just because a web application lives at "/myapp" on your site, doesn't mean that's how it'll see requests coming across. The URI is actually re-written by PhusionPassenger so your web application thinks it is living at "/" instead of "/myapp". So, if you have an application route for "/webapp/programs/login", PhusionPassenger would actually re-write it as "/programs/login" then pass the request to the application. Part of the problem was resolved by setting the following option in the web application's "config/environments/production.rb" file:

config.action_controller.relative_url_root = '/myapp'

Since there were a handful of web applications coming from Boe, each of which were previously served via PhusionPassenger, it made nginx configuration a bit tricky. This is because PhusionPassenger was serving up static content in addition to the web application. There were also some issues with the web application generating links or redirecting to "/myapp/myapp/route/path". I resolved this with a nested location match, case-insensitive regular expression and URI re-write. This also increased performance by having nginx serve up static content while everything else was passed to the Unicorn. The location match and nested match I setup for "myapp" is as follows:

        location ^~ /myapp {
            location ~* /(images|javascripts|stylesheets|documents) {
                root /opt/myapp/public;
                rewrite ^/myapp/(.*)$ /$1? break;
            }
            root /opt/myapp/public;
            proxy_cache myapp;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_read_timeout 1000;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_pass http://myapp_app_server;
        }


Voila! The web application was fully functional and nginx served up page styling. After these changes, I was able to successfully migrate the web applications from the aging Boe to its replacement. There were a few more hiccups but they were mostly issues with application routes that the developers had to resolve. I will eventually be able to remove the URI re-write once the developers have finished updating the web applications.

Hopefully these notes will help someone else with a similar challenge. For posterity and your reference, here are links to the respective resources I utilized:

ruby: https://www.ruby-lang.org/en/
Unicorn: http://unicorn.bogomips.org/
PhusionPassenger: https://www.phusionpassenger.com/
PhusionPassenger (setup guides): Apache guide, nginx guide
nginx: http://nginx.org/
Apache: https://httpd.apache.org/

Until next time ...