Yelp’s engineering team loves Docker. We’re already using it for a growing number of projects internally but there are applications that Docker isn’t a great fit for, such as providing independent (VM like) containers on shared hardware for people to interactively ssh into. You can, of course, run sshd inside a Docker container but you can only run one sshd per port. If you wanted multiple users to be able to ssh into the same server, without having a custom port allocated per user, you’re out of luck.

To solve this problem we built dockersh, a user shell for isolated, containerized environments in Docker. It’s available as a container on dockerhub with a 1 step install. This is a fully functional and usable implementation that you can play with. I’m already using it on some of my home servers to separate people’s screen/tmux sessions for irc into separate containers.

Originally I solved this by using the ssh ForceCommand setting so that users would log into the host and then immediately be forced to ssh to a Docker container. This was not ideal as key management and mapping user’s ForceCommand setting was complex.

After discussing this problem with a couple of my colleagues, and some inspiration from a recent blog post about sshd in containers, we had a rough idea of what we wanted to experiment with: making a shell which located a user inside a container using nsenter!

In brief, we planned to write a utility which would:

• Be invoked by the login system as a user shell (when a user sshs into the host)
• Start up a Docker container for the user (if it wasn’t already running), with the user’s home directory mounted
• Lastly, exec nsenter to give the user a shell inside this container

Theoretically, this could give us isolated environments, as each user would have their own network stack, process and memory namespaces, etc. If subsequent ssh sessions just enter the pre-existing container, it would look (to the user) like they had their own dedicated machine.

We thought we could hack together a prototype of this pretty quickly. Turns out we were right; with a couple of patches, and a custom nsenter version we were in business, at least the ‘terrible but kinda functional prototype’ business, perfect for our upcoming hackathon.

We decided to rewrite the prototype version in Go, which was a first for me. At the time, this was mostly an excuse to play with Go but it quickly proved to be a very sound decision. We were able to make use of some excellent libraries, like libcontainer, which did a lot of the heavy lifting for us.

dockersh can be explicitly invoked for testing, but more usually will be setup as an ssh ForceCommand, or added to /etc/shells and used as the shell for users in /etc/passwd. It reads a global config file and per user configuration files, and then sets up a container and invokes a real shell process for the user. This user shell process inside their container should be more secure than access to the host, as the container’s kernel namespaces and cgroups are applied to the new shell, in addition to dropping additional ‘capabilities’ for the process including SUID and SGID bits.

The utility we have now is just a start - I’ve already got a list of future improvements and further work (in the issue tracker), and I encourage you to have a play with the project and let us know what you think.

Back to blog