Michael Trojanek (relativkreativ) — Bootstrapper and creator of things

This article was published on September 13th 2020 and takes about 5 minutes to read.

Use it with caution — it is probably still valid, but it has not been updated for over a year.

Why SELinux may block systemd services calling rbenv

An rbenv installation usually works unobtrusively. On a production server running SELinux however, things can get a little messy.

rbenv is a joy to use. Switching between multiple Rubies for development — its main intended use — works reliably across all platforms (its core mechanics are not very complicated after all).

If you are a fan like me, you will sooner or later use it on production machines to make upgrading Rubies easy or to install multiple Ruby versions for different applications. Even the promising Fullstaq Ruby project uses it as its version manager of choice.

For serving Rubies in production, we need different rbenv commands than for switching Ruby versions in development but for this article, we will focus on the rbenv exec command which runs a single script under a specific Ruby version (ignoring how rbenv decides which Ruby version to use, as this is of no relevance here).

The problem

The rbenv exec command is often used in systemd unit files to start a Ruby binary as systemd service. Think of the Puma application server or Sidekiq — we'll want them to start when our production server boots and they are easily managed through the systemctl command this way.

To start and stop Puma from within a systemd unit file, we can use commands like these:

ExecStart=/home/app/.rbenv/bin/rbenv exec bundle exec puma -C /var/www/myapp/shared/config/puma.rb
ExecStop=/home/app/.rbenv/bin/rbenv exec bundle exec pumactl -S /var/www/myapp/shared/pids/puma.state stop

They may look complicated but they are actually quite simple: rbenv exec makes sure that rbenv selects the proper Ruby version while bundle exec makes sure that the proper version of Puma according to our application's Gemfile is used.

But starting Puma with systemctl start puma.service does not work and we will find the following lines in our server's /var/log/messages file:

May 22 16:25:30 production systemd[6056]: puma-myapp.service: Failed to execute command: Permission denied
May 22 16:25:30 production systemd[6056]: puma-myapp.service: Failed at step EXEC spawning /home/app/.rbenv/bin/rbenv: Permission denied

Of course we are not talking about Permission denied because of a wrong owner or mode. That would be too easy.

What's going on here?

Turning SELinux off for debugging reasons reveals that this is an SELinux issue.

As always, searching the web for solutions turns up lots of tutorials explaining how to turn off SELinux. Before you follow them blindly, ask yourself if you would turn off your firewall in order to open port 80 — SELinux issues always have a clear cause and it pays to find out about it (SELinux logs denied actions in the file /var/log/audit/audit.log).

In this particular case, systemd is not allowed to execute the rbenv script because it has the SELinux type user_home_t (the default type for files in home directories).

If we look at other binaries which are started and stopped by systemd, it becomes clear that almost all of them are kept in /usr/bin or /usr/sbin. Examining their SELinux type with ls -Z, it turns out that all of them have the SELinux type bin_t.

The solution

We need to change the SELinux type of the rbenv command to bin_t (and while we're at it, we are also going to adjust the type of other rbenv files).

While we could use the chcon command to do so, this is rarely a good idea, since context changes made with chcon do not survive reboots.

We need to use the semanage command to permanently create a rule:

semanage fcontext -a -t lib_t "^/(home/[^/]+|root)/\.rbenv(/.+)?"
semanage fcontext -a -t bin_t "^/(home/[^/]+|root)/\.rbenv/bin(/.+)?"

The first line sets all rbenv files to SELinux type lib_t, the second one then sets bin_t to the bin directory and all its children (-a adds a rule and -t denotes the type). The regular expression is constructed so that the rule works for normal users and for the rare cases where you want to install rbenv for root. The order is important — do not swap these two lines as they are processed sequentially.

Labeling all other rbenv files as lib_t is not entirely correct but since most of the files under the .rbenv directory would be placed somewhere under /usr/lib, this is a good start.

If you are making changes like this through a configuration management tool (which you absolutely should do), here is the Ansible task which does the same:

- name: Set SELinux type of rbenv files
  sefcontext:
    target: "{{ item.regex }}"
    setype: "{{ item.type }}"
    state: present
  loop:
    - { regex: "^/(home/[^/]+|root)/\\.rbenv(/.+)?", type: "lib_t" }
    - { regex: "^/(home/[^/]+|root)/\\.rbenv/bin(/.+)?", type: "bin_t" }

Note the double backslash to escape the literal dot in .rbenv.

A word of warning

Rules added through the semanage command get written to /etc/selinux/targeted/contexts/files/file_contexts.local (if you are using the targeted policy, which is the default), so everytime you run the semanage fcontext … command, a new line is added to this file.

So when you are finished, make sure to edit the file and remove all lines which were a result of trial-and-error creating the proper rule.

You may have the idea to prepare the file_contexts.local file upfront and upload it to your server. This is dangerous if you are not 100% sure which rules need to be added by your whole playbook. The recommended way is using the semanage command.

Get in the loop

Join my email list to get new articles delivered straight to your inbox and discounts on my products.

No spam — guaranteed.

You can unsubscribe at any time.

Got it, thanks a lot!

Please check your emails for the confirmation request I just sent you. Once you clicked the link therein, you will no longer see these signup forms.