Tuesday, July 15, 2014

Thoughts from the desk of a SysAdmin: Migrating from Apache + PhusionPassenger to nginx + Unicorn

Today's topic: Migrating from Apache + PhusionPassenger to nginx + Unicorn

I recently was presented with the challenge of building a new web application server. The previous server was outdated and so far behind on updates that replacing it was the best course of action. Since the previous server (we'll call it Boe, henceforth) was functional and not in need of immediate replacement, I had time to design and build a really awesome system. First, I needed to decide how I was going to resolve the following items:
  1. Replace PhusionPassenger
  2. What to use instead of PhusionPassenger
  3. Replace Apache
  4. What to use instead of Apache
Why replace PhusionPassenger? Regardless of Apache or nginx, it made the upgrade path too dirty and hairy to maintain. PhusionPassenger also wasn't very flexible or scalable enough. With nginx, you would have to re-compile the nginx binary and re-build the RPM each time. With Apache, you only need to re-compile mod_passenger each time. That is, so long as that version of Apache is compatible. There also didn't appear to be any functionality in PhusionPassenger to support multiple versions of ruby. These are the primary reasons as to why I wanted to replace PhusionPassenger.

What to use instead of PhusionPassenger? After a bit of research and reading, I felt that Unicorn was the best solution as the application server. It allowed me to keep the applications secure so that nginx talked to it over a unix socket. Unicorn is very configurable and I like how it utilizes a master/child worker architecture and integrated easily with the web application.

Why replace Apache? For application servers, I prefer to use nginx because of its small memory footprint, caching and performance. Apache is great, in my opinion, for hosting websites or simple applications like wordpress, cacti or nagios. This was my justification for replacing Apache.

What to use instead of Apache? Well, nginx immediately came to mind. Since the web applications are heavy and need a fast front-end, nginx was the best solution. I also like how nginx buffers the response from the back-end so that slow clients won't kill the response time of the application and result in degraded performance.

After additional research and preliminary testing, I built the new system and tested out my new configurations. I ran into a couple snags though with migrating away from PhusionPassenger. The first "gotcha" being that just because a web application lives at "/myapp" on your site, doesn't mean that's how it'll see requests coming across. The URI is actually re-written by PhusionPassenger so your web application thinks it is living at "/" instead of "/myapp". So, if you have an application route for "/webapp/programs/login", PhusionPassenger would actually re-write it as "/programs/login" then pass the request to the application. Part of the problem was resolved by setting the following option in the web application's "config/environments/production.rb" file:

config.action_controller.relative_url_root = '/myapp'

Since there were a handful of web applications coming from Boe, each of which were previously served via PhusionPassenger, it made nginx configuration a bit tricky. This is because PhusionPassenger was serving up static content in addition to the web application. There were also some issues with the web application generating links or redirecting to "/myapp/myapp/route/path". I resolved this with a nested location match, case-insensitive regular expression and URI re-write. This also increased performance by having nginx serve up static content while everything else was passed to the Unicorn. The location match and nested match I setup for "myapp" is as follows:

        location ^~ /myapp {
            location ~* /(images|javascripts|stylesheets|documents) {
                root /opt/myapp/public;
                rewrite ^/myapp/(.*)$ /$1? break;
            }
            root /opt/myapp/public;
            proxy_cache myapp;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_read_timeout 1000;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_pass http://myapp_app_server;
        }


Voila! The web application was fully functional and nginx served up page styling. After these changes, I was able to successfully migrate the web applications from the aging Boe to its replacement. There were a few more hiccups but they were mostly issues with application routes that the developers had to resolve. I will eventually be able to remove the URI re-write once the developers have finished updating the web applications.

Hopefully these notes will help someone else with a similar challenge. For posterity and your reference, here are links to the respective resources I utilized:

ruby: https://www.ruby-lang.org/en/
Unicorn: http://unicorn.bogomips.org/
PhusionPassenger: https://www.phusionpassenger.com/
PhusionPassenger (setup guides): Apache guide, nginx guide
nginx: http://nginx.org/
Apache: https://httpd.apache.org/

Until next time ...

Thursday, January 23, 2014

How-to: Connect to an SMB share via a NAT'ed IPsec tunnel

Yes, I know, you'd think this would be simple. Configure libreswan, establish the tunnel and either mount the smb share or "get" your file using smbclient. Unfortunately, it isn't that simple if you're using a NAT'ed ipsec tunnel as required by your remote endpoint. Since my peer address is on eth0 and my (assigned) NAT'ed address is on tunl0, that creates a slight problem when using smbclient. I attempted to use the interfaces="tunl0=eth0" option in ipsec.conf, but it did not behave as expected. Maybe the "interfaces" option isn't supposed to function like I thought it would, nevertheless, with the tunnel running my problem was now that I couldn't use smbclient.

Why can't you use smbclient, you ask? Well, I needed to connect to the remote party's server and smb share. However, you cannot tell smbclient which interface to use like you can with ping. I was able to confirm connectivity to the remote server via the tunnel by issuing the following in my shell:

$ ping -I tunl0 123.45.67.89

After a few hours of crawling through Google results using different search queries, I decided to develop my own solution. I knew about socat (netcat++) but had never used it; this seemed to be the perfect solution. After some reading and tinkering, I came up with the following command:

$ socat -d -d \
TCP4-LISTEN:139,bind=localhost,reuseaddr \
TCP4:123.45.67.89:139,bind=172.16.100.1

Let's break this down a bit ...

socat -d -d

This tells socat to run and print messages that are fatal, error, warning, or notice.

TCP4-LISTEN:139,bind=localhost,reuseaddr

This portion of the command tells socat to start listening for TCP/IPv4 connections on localhost at port 139. The "reuseaddr" option tells socat to let other sockets bind to the address even if only part of it is being used (i.e., the port we're listening on).

TCP4:123.45.67.89:139,bind=172.16.100.1

The last portion of the command tells socat to proxy TCP/IPv4 connections to port 139 on the remote system. The "bind" option tells socat to use the NAT'ed address that we've assigned to the device tunl0.

With the first half of the problem resolved, I was able to begin using smbclient to connect to the remote Windows server. Since I was going to ultimately script the whole process, I developed the following command using smbclient:

$ smbclient //PDC01/private \
-I 127.0.0.1 \
-U PDC\\MyUser%pass123 \
-c "get thisfile.txt /tmp/thisfile.txt"

Again, let's break this down a bit ...

smbclient //PDC01/private

This starts smbclient and tells it to connect to the smb share "private" on the Windows server named "PDC01".

-I 127.0.0.1

This tells smbclient to connect to 127.0.0.1 (localhost) where we're running our socat proxy.

-U PDC\\MyUser%pass123

This identifies our credentials; since Windows accounts are in the format "NT-Domain\UserName" we have to escape the forward slash, hence the "\\". To include a password for scripting purposes, we separate the account and password with a percent sign.

-c "get thisfile.txt /tmp/thisfile.txt"

This passes the above command to smbclient. Simply, we tell it what remote file to download and the location of where we want to save it.

After the file transfer completes, socat will detect that the TCP session has ended and thus shutdown nicely. With that, we have our solution; I hope this helps anyone that may encounter a similar situation.

If you would like to read more about socat, libreswan or samba, here are their respective websites:

socat: http://www.dest-unreach.org/socat/
libreswan: https://www.libreswan.org/
samba: http://www.samba.org/

Until next time ...