ffmpeg-convert.sh

i recently came across the need to quickly rip audio track from video files. my first thought was to use ffmpeg and wrap it in a script. sometimes i may have one or 20 MP4/MKV/webm files that need to be converted to an MP3. the files may be scattered across a few directories, but each serves its own purpose. thus, i wrote a script to automate a lot of repetitive work.

the script, called ffmpeg-convert.sh, makes a lot of assumptions and works fine as an underprivileged user. it expects the files to be actual video files and will fail gracefully if the file doesn't exist or ffmpeg unexpectedly exits. it will continue where it left off if you CTRL+C for some reason. it also has a neat spinner so that you don't have to wonder if it's working or not. some large files can take a while to process.

i endeavored to write as much of the functionality as possible in BASH. the sub-string search and replace is kind of disgusting, but it does a fairly decent job of sanitizing the filename.

the script works like this: when invoked, it scans the current working directory for files with "known" video file extensions. it prompts you with how many files were found and asks for permission to continue. it then processes each file via ffmpeg and bakes you a fresh MP3 file. it's pretty simple, and i might consider adding more functionality to the script. i've already received a couple of requests!

grab the script from here.

 

 

 

git log query, or: glq

Sometimes you've got to take the time to build something so you can be more productive. Also, it never hurts to make life easier on yourself. I've found git log can be a source of zen when combined with disciplined git commit messages.

To give you some background, I have a wrapper around the git command that templates commit messages. It helps to ensure consistent, detailed, quality commit messages for posterity. 

The template uses the following format:

KEYWORD: path/to/file/relative/to/repo/file.ext; Commit message here. (Ticket serial number, if applicable)

The KEYWORD can be one of: NEW, CHANGE, REMOVED, OOB. Some files are stored in a directory named after the host they're installed on, and the ticket serial number can vary depending on how the work request was received. 

Here is an example of a real commit message:

CHANGE: files/ssl-certificates/my.host.com/my.host.com.pfx; Updated SSL Certificate and rotated private key. (TKT002048)

This allows me to search the git log for things like "NEW" files, the last "CHANGE" made to an SSL Certificate or firewall config, or when files were "REMOVED". This also helps me provide an easily searchable git audit log, should I ever need to, during a Compliance Audit or Incident Investigation.

However, using git log --grep "CHANGE.*my.host.com.pfx" --extended-regexp -n 1 got to be a bit cumbersome. I wanted a short command that could also take an option to increment "-n" as needed. A script felt like overkill for this situation, so I opted for a BASH function that's loaded into the shell session via .bashrc (on most GNU/Linux) or .bash_profile (on macOS). 

With this function, I've created a simple command: glq

Here is an example of how glq can quickly return the latest commit where the firewall configuration was changed for the host my.host.com:

$ glq "CHANGE.*my.host.com/firewall.yml"

Since git commits are "first come, first serve", a commit authored in Oct 1 could be merged on Nov 1 and would thus be considered the latest commit (or change). So, when thinking about the commit log in terms of, "descending, chronological order," we must think of it as the reverse (starting from now), chronological order in which commits were added/merged into your working branch.

By default, glq will only return the latest, first matching commit. If you want to expand the limit to 5, you can specify an integer after the double-quote encapsulated search query. 

For example:

$ glq "CHANGE.*my.host.com/firewall.yml" 5

The above example (building off the previous example) has "5" specified after the search query. This (essentially) tells glq/git to limit matches to the five most recent commits, instead of only the latest one. 

For even more fun, if I want to find the latest commit where a customer requested a change to a system, I could use the following search query:

$ glq "CHANGE.*customer.host.com/firewall.yml.*TKT"

Which would match the following commit message:

CHANGE: host_vars/customer.host.com/firewall.yml; Customer requested opening port 8443. (TKT004096)

As you can see, this has the potential to be a very powerful tool. It really only works, though, if you are consistent with your commit message formatting either through policy or template, or both.

Have fun!

The calendar broke (again). I fixed it (again).

Such is the life of anyone in IT - you fix something, it works for a while, then it inevitably breaks again.

This time it's different, and I don't plan on it breaking again, unless there are massive changes across the board to the back end.

This update is minor and I've retrofitted all previous downloadable files with the fix.

In short, I didn't read the man page completely and thus the %b option for date printed the abbreviated month. Well, May is only three letters long, so its full name is the same as its abbreviated name. Since Jun does not equal June, the regex match for the month fails. If we change the date command's formatting argument to %B, then we get the full month and the regex match completes.

I've updated the patch file (if you're still using v1) and v2 of the script.

Link: Script v2

Link: Patch file (if you're still on v1 and want to upgrade to v2) 

Have fun!

A pretty, pretty cal(endar): Rev1

In my last blog post, I provided a BASH function that can operate as a lightweight replacement for cal. When I say "replacement", I mean, it just gives cal a bit of a facelift by making it pretty.

I'm here today to post a patch to that function. The regex got wonky when matching 20 or 21, so the header with the month and the year got dirty.

I also got tired of seeing the double-spaces at the end of the current week. That has only been an issue with the Apple version of cal. However, with some regex revisions, we arrive at a once again pretty, pretty cal(endar).

Somehow, my blog got marked as a malware-potential-phishing-don't-go-there-it's-Knockturn-Alley (please don't sue me, J.K.!) website because of my last post. To avoid this from happening again, I'm instead going to post links to the scripts and the patch (so you can see the diff).

 I have tested this out quite thoroughly on macOS and Linux, but do keep in mind that YMMV.

Again, standard attribution CC license applies to this code.

Link #1: v1 of the function

Link #2: v2 of the function

Link #2: The patch file

 

Until next time ...

A pretty, pretty cal(endar)

I live on the command-line and that means I need tools to keep me efficient. One tool that I use frequently is "cal". However, the default output is quite plain and (at times) is hard to read. So, I slapped together a shell function that colorizes the output and makes using it at-a-glance much easier.

The default output looks like this:

With colorized output, it now looks like this:

I always have a terminal session open with an active SSH connection to my Linux box, and another terminal session open for conducting business locally on my Mac. So, I needed this to work for both Mac and Linux. I learned the hard way that "echo -e" is really only useful on a Linux system. Thus, I have started using "printf" when I want formatted output beyond a simple "echo". I should probably revise this function to use "tput" and such, but for now I'm using ANSI escape codes for coloring text.

For macOS, you'll want to place this function in your ~/.bash_profile. On your Linux box, storing it in ~/.bashrc should be fine.

It probably goes without saying, this is provided to you without any warranty or guarantee, etc., etc., feel free to use or redistribute. Attribution would be nice, but not really necessary unless it's being included in something proprietary or a for-profit solution. Nonetheless, enjoy! 

 

Link #1: A pretty, pretty cal(endar) v1

Until next time!

Ref: https://misc.flogisoft.com/bash/tip_colors_and_formatting

Thoughts from the desk of a SysAdmin: fail2ban, sshd, and named

Today's topic: fail2ban, sshd, and named

I recently rebuilt one of my cloud VMs because it was aging and I wasn't happy with it anymore. After rebuilding it, I decided to use it to also run my own caching recursive DNS service that includes domain blacklisting (blocking known malware, phishing, and ad domains). Since this would have to be public-facing, I knew (at some point) someone would get curious and try to attack it. I have previous experience with just sshd being public, watching as countless bots scan and potential attackers probe for weaknesses.

I've always run SELinux and fail2ban, but I recently decided to take it a step further with hardened configuration files and custom fail2ban filters. fail2ban has some great builtin filters and they'll help protect a system against basic attacks with minimal configuration. However, what makes fail2ban really awesome are custom filters and actions. These enhance the functionality of fail2ban to give you the most protection possible. Nowadays, cyber criminals are after everyone, not just businesses.

Disclaimer: The file locations in this post are focused towards RHEL-based systems (i.e. CentOS, Fedora, etc.). I also have no experience (yet!) with firewalld; my current expertise is limited to iptables. However, the concepts and configurations should easily translate to any modern Linux system.

What's fail2ban?

fail2ban is an awesome IDS/IPS (Intrusion Detection System/Intrusion Prevention System) that uses regular expressions to scan logs then subsequently perform certain operations. Namely, it interfaces very well with iptables. Out of the box, it has support for a wide range of services and builtin regular expressions to help defend against the most common types of attacks. It also has support for custom filters and actions, making it highly extensible. fail2ban, leveraged with sound security practices, can provide assistance with protecting your system and services.

sshd (or, "Secure Shell Daemon")

sshd is the best service to use for performing remote system management. In some cases, it's the only way you can get into a box. Thus, taking advantage of the protection that fail2ban can bring to your system should be paramount. There are a few steps I would recommend making in addition to activating the sshd IDS/IPS protection in fail2ban.

First, set your default iptables INPUT policy to DROP and save the rules. The default policy provided by the vendor is ACCEPT, which is the equivalent of "accept all, deny none" unless you have a rule specifically blocking a port and/or protocol. Setting your default INPUT policy to DROP basically says, "deny all, accept none". This allows you to setup your iptables rules to only allow specific traffic. This protection is a basic rule that everyone should employ - least privilege methodology. Your system can still be compromised even if you have nothing listening; if you do and you forget to configure it to only listen on localhost, it immediately becomes an attack vector. This method also allows you to setup a new service and configure it before activating a rule, thereby allowing traffic to hit it.

Before setting your default policy to DROP, make sure you also have a rule that allows for STATEFUL,ESTABLISHED connections. This rule is typically included by default on RHEL-based systems. If you're not running on a RHEL-based system, make sure you have this rule at the top of your INPUT table.

Remember: Firewall rules are processed based on their order in the table.

Next, make sure you have a rule allowing SSH traffic on port 22. Once you have the aforementioned rules added, you can then feel safe to set your default INPUT policy to DROP. Don't lock yourself out of your own box! If you already have fail2ban installed, their rules should be first while the service is running. Simply stop the fail2ban service, add your rules (don't forget to save!), then start it again.

Second, secure your sshd service by hardening your sshd_config file. This file is typically found at /etc/ssh/sshd_config. Start by denying root login via SSH. Set PermitRootLogin to "no":

PermitRootLogin no

You should never remotely login to your box as root. Only allow logins by an underprivileged account that has permission (if necessary) to escalate to root or access the sudo command. Next, set some strong Key Exchange Algorithms, data encryption Ciphers, and Message Authentication Ciphers. This will protect you from attackers and the Big Bad (cough, cough - NSA). It's actually surprising how many attackers don't have a modern SSH client that supports newer ciphers. It's likely due to their targets being systems that aren't using modern cryptography. As clients are updated with modern algorithms, deprecated ones are removed. Nonetheless, any modern macOS and Linux operating system should support these ciphers. I haven't tested with Windows and PuTTY, though. If you'd like to ensure your session is encrypted using strong stuff, add the following text below the "Protocol 2" line:

# Specify Algos and Ciphers (so we're not vulnerable to default and weak cryptos)
KexAlgorithms diffie-hellman-group-exchange-sha256
Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512,hmac-sha2-256

Increase sshd's logging level. Set the LogLevel to VERBOSE:

LogLevel VERBOSE

This allows you to more closely audit the activity of sshd and key authentication. This is also required for a custom fail2ban filter that I will discuss later.

Disable password authentication and only utilize key-based authentication. It's simple:

PasswordAuthentication no

Make sure you generate a key pair (I recommend 2048-bit RSA or greater); store the public key in ~/.ssh/authorized_keys and chmod it to 0400. Only leave password authentication enabled if you're using a form of two-factor (key + account password).

Whether using key-based or password-based authentication, a valid user shouldn't have any reason to take longer than a minute to login. Set the LoginGraceTime to a conservative value; I have mine at one minute:

LoginGraceTime 1m

Let's also protect against resource consumption DoS attacks and reduce the number of chances an attacker has per connection. This is as simple as reducing MaxAuthTries:

MaxAuthTries 3

The default is set to six. As long as you have fail2ban blocking an IP after three authentication failures, you should be safe.

Finally, with regards to sshd_config, I set the following to reduce the attack surface on the server and client:

GSSAPIAuthentication no
X11Forwarding no

I don't use GSS and I don't want it available to a potential attacker. I don't use X11 on my remote server, so I close that open channel (which can be an attack vector on the client).

Third, custom sshd filter in fail2ban. Even though disabling password authentication and setting strong algorithms should seem like enough, it's not. While attackers may not reach a preauth or auth stage, they still consume resources by establishing a connection. To further prevent a resource consumption DoS, I setup the following file in /etc/fail2ban/filter.d/sshd-badmac.conf:

# This rule augments the default fail2ban sshd rules by adding an additional filter to block potential attackers.
# There really aren't any legitimate reasons why an authorized person would connect using unsupported macs, kexs, or algos.

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = sshd

# these are the log entries we're looking for
#Mar 28 14:30:21 myserver sshd[30255]: Connection from 123.45.67.89 port 98765
#Mar 28 14:30:22 myserver sshd[30256]: fatal: no matching mac found: client hmac-md5,hmac-sha1 server hmac-sha2-512,hmac-sha2-256

failregex = ^(?P<__prefix>%(__prefix_line)s)Connection from port \d+(?: on \S+ port \d+)?%(__prefix_line)sfatal: no matching mac found: .+$

ignoreregex =

[Init]
maxlines = 2

journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd

And activate it in /etc/fail2ban/jail.d/jail.local:

[sshd-badmac]
enabled = true
filter = sshd-badmac
port = ssh
logpath = /var/log/secure
bantime = 86400
findtime = 3600
maxretry = 1
action = iptables[name=SSHBADMAC, port=ssh, protocol=tcp]

Ba'da bing, ba'da boom. Hoohaw! Three steps to secure your sshd service and protect your system from attackers.

bind (or, "named")

In my opinion, bind is the best DNS server available. It's highly configurable, fast, and performs well under a heavy load. However, there are some security considerations to take in if you're going to host it on a public system. To make this work, you're going to have to allow queries from subnets outside of your host's public subnet. By default, bind allows recursive queries as long as the client is within the same subnet as the host. If you wish to take it a step further, you can block non-domestic IPs based on the ranges available from ARIN (American Registry of Internet Numbers). Even if you employ this technique, domestic attackers may still attempt to DoS your DNS server, compromise it, or even rack up your bandwidth charge because they're bored. There are a few steps you can take to protect your public-facing DNS service whether you're running a resolver or recursive name server.

First, we need to harden our bind configuration to rate-limit queries. I have averaged usage by a few users that I share my DNS service with and a reasonable user shouldn't be making more than 10 queries per second. Start by adding this to your bind config (in the options section), typically found at /etc/named.conf:

rate-limit {
        responses-per-second 10;
        log-only yes;
};

Make sure you also enable DNSSEC:

dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;

Also be sure to deny transfers:

allow-transfer { none; };

Make sure you aren't replying authoritatively for domains that you do not resolve for:

auth-nxdomain no;

Finally, block your version number from being exposed to attackers, thereby making their job easier. Blocking the version number prevents an attacker from looking up existing vulnerabilities and attacking your service. This makes their job harder and they will likely pass you up. To do this, set the following option:

version "unknown";

Second, setup your bind logging to log everything you'd ever need to know about bind's activity. This will also come in handy with setting the rate limiting I mentioned earlier. The fail2ban custom filter utilizes the query log to block potential malicious activity. Here's what my logging stanzas look like:

logging {
        channel default {
                file "/var/log/named-auth.log";
                severity info;
                print-time yes;
                print-category yes;
                print-severity yes;
        };
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
        category lame-servers { null; };
        category default { default; };
        channel "querylog" { file "/var/log/named-query.log"; print-time yes; };
        category queries { querylog; };
};

Third, setup a custom fail2ban filter to block users that are rate limited. I setup the following filter in /etc/fail2ban/filter.d/named-throttling.conf:

# fail2ban filter for hosts attempting to cause a DoS
# this works in conjunction with the bind query throttling. messages are printed when
# hosts start abusing the system.

# example of what we're looking for to block
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: query: nccih.nih.gov IN ANY +E (98.76.54.32)
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: would slip response to 12.34.56.0/24 for nccih.nih.gov IN ANY  (00111ec3)
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: query: nccih.nih.gov IN ANY +E (98.76.54.32)
# 28-Mar-2017 22:53:22.624 client 12.34.56.78#33283: would drop response to 12.34.56.0/24 for nccih.nih.gov IN ANY  (00111ec3)

[Definition]

# Daemon name
_daemon=named

# Shortcuts for easier comprehension of the failregex

__pid_re=(?:\[\d+\])
__daemon_re=\(?%(_daemon)s(?:\(\S+\))?\)?:?
__daemon_combs_re=(?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re)s?:)

#       hostname       daemon_id         spaces
# this can be optional (for instance if we match named native log files)
__line_prefix=(?:\s\S+ %(__daemon_combs_re)s\s+)?

failregex = ^%(__line_prefix)s?\s*client #\S+( \([\S.]+\))?: would (slip|drop) response .+$

ignoreregex =

[Init]
maxlines = 1

I then activated the filter in my /etc/fail2ban/jail.d/jail.local file with the following:

[named-throttling]
enabled = true
filter = named-throttling
port = domain
logpath = /var/log/named-query.log
bantime = 120
findtime = 60
maxretry = 1
action = iptables[name=NAMEDTHROTTLE, port=domain, protocol=udp]

This ensures that anyone who triggers bind's rate limiting is blocked for two minutes. As I stated earlier, any reasonable user shouldn't have more than 10 queries per second. So, if you're not abusing the service, you shouldn't have to worry. Keep in mind, I set this up for my friends and family to use, as well as myself. So, based on your usage, you may need to tweak the bind rate limiting.

There you have it, folks. My thoughts on how to secure your system with hardened configurations and custom filters. I have some advanced tips I'll likely cover in future articles on iptables, sshd, bind, and fail2ban; they just didn't fit here.

Versions

CentOS: release 6.8
bind/named: bind-9.8.2-0.47.rc1.el6_8.4.x86_64
sshd: openssh-5.3p1-118.1.el6_8.x86_64
fail2ban: fail2ban-0.9.6-1.el6.1.noarch

Links

CentOS: https://www.centos.org/
RHEL: https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux
EPEL (where you can grab fail2ban): https://fedoraproject.org/wiki/EPEL
bind/named: https://www.isc.org/downloads/bind/
sshd: https://www.openssh.com/
fail2ban: https://www.fail2ban.org/wiki/index.php/Main_Page


Until next time ...

Thoughts from the desk of a SysAdmin: Migrating from Apache + PhusionPassenger to nginx + Unicorn

Today's topic: Migrating from Apache + PhusionPassenger to nginx + Unicorn

I recently was presented with the challenge of building a new web application server. The previous server was outdated and so far behind on updates that replacing it was the best course of action. Since the previous server (we'll call it Boe, henceforth) was functional and not in need of immediate replacement, I had time to design and build a really awesome system. First, I needed to decide how I was going to resolve the following items:

1. Replace PhusionPassenger
2. What to use instead of PhusionPassenger
3. Replace Apache
4. What to use instead of Apache

Why replace PhusionPassenger? Regardless of Apache or nginx, it made the upgrade path too dirty and hairy to maintain. PhusionPassenger also wasn't very flexible or scalable enough. With nginx, you would have to re-compile the nginx binary and re-build the RPM each time. With Apache, you only need to re-compile mod_passenger each time. That is, so long as that version of Apache is compatible. There also didn't appear to be any functionality in PhusionPassenger to support multiple versions of ruby. These are the primary reasons as to why I wanted to replace PhusionPassenger.

What to use instead of PhusionPassenger? After a bit of research and reading, I felt that Unicorn was the best solution as the application server. It allowed me to keep the applications secure so that nginx talked to it over a unix socket. Unicorn is very configurable and I like how it utilizes a master/child worker architecture and integrated easily with the web application.

Why replace Apache? For application servers, I prefer to use nginx because of its small memory footprint, caching and performance. Apache is great, in my opinion, for hosting websites or simple applications like wordpress, cacti or nagios. This was my justification for replacing Apache.

What to use instead of Apache? Well, nginx immediately came to mind. Since the web applications are heavy and need a fast front-end, nginx was the best solution. I also like how nginx buffers the response from the back-end so that slow clients won't kill the response time of the application and result in degraded performance.

After additional research and preliminary testing, I built the new system and tested out my new configurations. I ran into a couple snags though with migrating away from PhusionPassenger. The first "gotcha" being that just because a web application lives at "/myapp" on your site, doesn't mean that's how it'll see requests coming across. The URI is actually re-written by PhusionPassenger so your web application thinks it is living at "/" instead of "/myapp". So, if you have an application route for "/webapp/programs/login", PhusionPassenger would actually re-write it as "/programs/login" then pass the request to the application. Part of the problem was resolved by setting the following option in the web application's "config/environments/production.rb" file:

config.action_controller.relative_url_root = '/myapp'

Since there were a handful of web applications coming from Boe, each of which were previously served via PhusionPassenger, it made nginx configuration a bit tricky. This is because PhusionPassenger was serving up static content in addition to the web application. There were also some issues with the web application generating links or redirecting to "/myapp/myapp/route/path". I resolved this with a nested location match, case-insensitive regular expression and URI re-write. This also increased performance by having nginx serve up static content while everything else was passed to the Unicorn. The location match and nested match I setup for "myapp" is as follows:

        location ^~ /myapp {
            location ~* /(images|javascripts|stylesheets|documents) {
                root /opt/myapp/public;
                rewrite ^/myapp/(.*)$ /$1? break;
            }
            root /opt/myapp/public;
            proxy_cache myapp;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_read_timeout 1000;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_pass http://myapp_app_server;
        }

Voila! The web application was fully functional and nginx served up page styling. After these changes, I was able to successfully migrate the web applications from the aging Boe to its replacement. There were a few more hiccups but they were mostly issues with application routes that the developers had to resolve. I will eventually be able to remove the URI re-write once the developers have finished updating the web applications.

Hopefully these notes will help someone else with a similar challenge. For posterity and your reference, here are links to the respective resources I utilized:

ruby: https://www.ruby-lang.org/en/
Unicorn: http://unicorn.bogomips.org/
PhusionPassenger: https://www.phusionpassenger.com/
PhusionPassenger (setup guides): Apache guide, nginx guide
nginx: http://nginx.org/
Apache: https://httpd.apache.org/

Until next time ...