As an Operations engineer, my first priority is to keep the site up. A close second is enabling developers to quickly go from an idea to running code in production.

Once an organization grows, the only sane way to ship code with any reasonable frequency is to split it up into microservices, also known as building a Service Oriented Architecture (SOA). We’ve previously talked about the philosophy behind services and why we build them. This blog post explains the tool we use to make those services available to developers: PaaSTA!

What is PaaSTA?

PaaSTA is Yelp’s platform-as-a-service. It allows developers to declare, in config files, exactly how they want the code in their git repo to be built, deployed, routed, and monitored. PaaSTA powers a significant number of production services at Yelp, and has been running in production for more than 1.5 years. It even powers this very blog!

Before PaaSTA, Yelp’s internal “platform” was so manual, I would just call it “Mechanical Turk Operations.” In other words, getting a service into production was composed of manual steps, blocked on the operations team, taking days or months of setup time. Sound familiar? Almost every team has some sort of procedure like this, and like most procedures, it starts off as a manual process.

Eventually, Operations decided that our developers would be more powerful and, as an organization, we could ship things faster if we modernized our process and built a coherent set of tooling around shipping services. So PaaSTA was created!

How Does PaaSTA Work?

PaaSTA works by integrating existing open-source components that do their jobs very well:

• Docker for code delivery and containment
• Apache Mesos for code execution and scheduling (runs Docker containers)
• Mesosphere’s Marathon for managing long-running services
• Chronos for running things on a timer (nightly batches)
• SmartStack for service registration and discovery
• Sensu for monitoring/alerting
• Jenkins (optionally) for continuous deployment

Why are there so many pieces? Why isn’t PaaSTA a single Go binary? At Yelp we believe in not reinventing the wheel where possible. We want “seams” so we can take a technology and swap it out as we grow and scale. PaaSTA provides a place for these seams.

That’s Just a Laundry List of Tools. How Does it Actually Work?

It Starts with a Git Repo

A developer starts by setting up their service to run in a container by building a Docker image. The Operations team might help another team do this a little bit, but for the most part it is up to the service author to explicitly declare their runtime dependencies. Containers are great! They allow the developers to have the flexibility to setup the runtime environment as they see fit, and it gives the infrastructure a common “binary” to run, regardless of the underlying operating system.

Once a Docker image is built, the PaaSTA command line tool gives developers or Jenkins a way to upload their image and mark it as ready to be deployed.

Ok I’ve Got A Docker Image. Now What?

Once a Docker image is ready, the developer needs to declare how and where they want their service to be run. PaaSTA provides a DSL to describe this. Here is an example of a Service definition:

---
demo:
  cpus: 1
  mem: 500
  instances: 10
  monitoring:
    team: operations

This is what it looks like to build a declarative, implementation-agnostic platform-as-a-service for the long haul. There are no implementation-specific details in here. PaaSTA takes in these generic definitions and deals with the implementation-specific details behind the scenes.

To a developer, the interface is clean and straight-forward. All options are documented.

The Plate of PaaSTA is Set. Time to Dig In!

We have an image and a declarative language describing how to run it. Now PaaSTA can take over and put that code into Production. This is actually way easier than it sounds. Thanks to the existing engineering work put into Mesos, Marathon, and Chronos, PaaSTA can simply communicate with those API’s to make the developer’s wishes come true. PaaSTA constantly syncs the service definitions with the actual APIs:

But it is more than just POSTing to an API. With PaaSTA you are declaring your intent. Things don’t show up in prod because someone ran a deploy script. Things are deployed because they were declared that way in a config file, the cruise control was set, and PaaSTA stepped on the gas! If something catastrophic happens, no developer has to “redeploy” their service. PaaSTA will just try to get things back to the way that they were originally declared in the config files.

This is such a big idea that it justifies some more examples. Let’s say that Yelp engineers are trying to build out a whole new region. What does that look like? Well, in the PaaSTA world it looks like a big diff with new config files that declare that a bunch of services should be running in the new environment. Right after the git commit, PaaSTA will make it so. If we need to nuke that environment for some reason, no developer interaction is needed. We already told PaaSTA what we want, it will try to make it happen again.

My Service Is Running. How Do I Find It?

With any platform, service discovery is a BIG deal. If a service is deployed in an automatic way, then it needs to be discovered in an automatic way as well.

At Yelp we use SmartStack for service discovery. We love SmartStack at Yelp because it is platform and protocol agnostic. It takes a config file and configures loadbalancers on localhost. PaaSTA provides the glue, telling SmartStack which services to advertise and taking in hints from Docker, Mesos, or whatever technologies we adopt in the future.

A flexible service discovery mechanism that uses “boring” components is essential to the healthy long-term evolution of your infrastructure. We can tightly control SmartStack’s behavior because it is un-opinionated, and in the end reads whatever config file we generate. SmartStack doesn’t know anything about Mesos, or Docker, or Marathon, or whatever. This is important to us at Yelp, because we don’t want to be locked into a particular tool that tightly integrates with one particular piece of technology.

Want to know more about SmartStack? Check out this Video /Slides on how we use SmartStack at Yelp.

But It’s Not Done Till It’s Shipped Monitored!

Monitoring services in a dynamic world like this is a difficult challenge. Sensu is the only open-source monitoring tool I’ve seen that is up to this challenge.

But Sensu also doesn’t know anything about Mesos, Docker, Marathon, SmartStack, or PaaSTA. Its job is just to route alerts. PaaSTA is the glue that knows what services are out there and which team is responsible for them. Once you tie those pieces together, your monitoring system becomes, well, awesome!

Sensu is super flexible, and allows us to route our alerts in a team-centric way. Then, we let PaaSTA tell Sensu which services own which teams. No “Ops” team has to update the monitoring system when new services are created, Sensu can handle new events on the fly!

But What About Terraform, ECS, Nomad, BeanStalk, Flynn, Deis, Heroku, OpenStack…

There is certainly no shortage of platforms and tools out there, but with PaaSTA we can grow and use different tools as we need them, and developers shouldn’t have to change anything. After all, they have already told the system what code to run and how to run it. PaaSTA can figure out the rest.

In the infrastructure world, there is pretty much no such thing as a drop-in piece of software. Each component requires integration with the others. You can’t just plop down something like OpenStack in front of developers and say: “Here you go! Have at it!”

For Yelp infrastructure, we want to take raw, un-opinionated tools, and glue them together in a cohesive, opinionated, and sustainable way. PaaSTA is that opinionated glue! If you would like to know more technical details about how PaaSTA compares to other tools, check our our comparison doc.

Interested?

Has PaaSTA given you any cool ideas with what you want in your own PaaS? Checkout PaaSTA on GitHub to see how we glue the pieces together. Also check out this video to watch PaaSTA in action. If you want to talk to us in real time in #paasta on Freenode.

Check out our meetup on December 17, 2015 where we’ll talk more about the technology that powers PaaSTA and why this PaaS has worked out so well for us.

Also, to learn more about Yelp’s experience with Mesos, Docker and building a next-generation development environment, check out this case study.

Back to blog