In 2017, Yelp had over 40 Jira administrators to allow different teams across the organization to perform administrative tasks. With lots of admins came lots of changes, which lead to our Jira environment accumulating hundreds of orphaned workflows, screens, and schemes. To solve this problem, we built a scalable solution that empowers our engineers to create Jira projects themselves using code and source control – ensuring 100% standardization across all engineering projects and making Jira easier to manage, simpler to use, and better performing. This blog post will cover Yelp's use of Ansible to manage Jira server project creation, updates, and archive functionality.

Behind the Scenes

We first needed to decide which technologies to utilize to solve this problem. We settled on an open-source tool called Ansible as the main framework because of its procedural language and client-only architecture. While Puppet is used across Yelp, not having to deal with a complex infrastructure to run Ansible was key for us. In our implementation, Ansible is responsible for interacting with Jira's REST API to perform every step of the project lifecycle. We identified three critical project lifecycle elements, each with its own playbook:

• Project creation
• Project archival
• Project updates

We use Git to host our configuration while maintaining a full revision history. These configurations are maintained across three different repositories:

• Ansible_configs: Hosts Ansible configuration, playbooks, and roles.
• Ansible_precommithooks: Hosts configuration for the pre-commit hooks that were used in Ansible_configs repository.
• Ansible_jira_projects: Hosts YAML files defining each Jira project and its configuration.

Each project is represented by a single YAML file which describes the configuration in an easy-to-read way:

PROJECTBETA:
- key: beta
- leader: darwin
- board_type: agile
- description: the new revolutionary project
- security_schema: engineering

We then use the powerful templating feature from Ansible to transform the YAML configuration files into a JSON object that can be interpreted by Jira's REST API. Using Jinja2, we created custom templates that transform the YAML files into a JSON payload of the different Jira Server API requests. Here's an example of one of our Jinja2 templates we used to create a project:

{
    "key": "",
    "name": "",
    "lead": "",
    "projectTemplateKey": "",
    "issueSecurityScheme": "",
    "permissionScheme": "",
    "notificationScheme": ""
}

To verify that our engineers only push valid project manifests, we use a framework for managing and maintaining multi-language pre-commit hooks called "pre-commit." Each time an engineer makes a commit, our defined pre-commit hooks are automatically run to verify the manifest and identify any issues, such as invalid Jira project keys, names, or illegal characters.

Pre-commit has a lot of pre-built hooks available out-of-the-box. For this solution, we used ones like check-yaml and sort-simple-yaml, but also wrote our own custom hooks to ensure that all secrets were properly encrypted with Ansible Vault, and all existing and new YAML files only contained allowed keys/values and followed Yelp's Jira project standards.

Here's an example of one of our custom hooks that we created to validate our YAML syntax:

#!/usr/bin/python

import sys
import argparse
import yaml
import re

VALID_BOARD_CONFIGS = ['kanban', 'agile']


def main(argv=None):
    retval = 0
    parser = argparse.ArgumentParser()
    parser.add_argument('filenames', nargs='*', help='Jira project files to check.')
    args = parser.parse_args(argv)
    argv = argv if argv is not None else sys.argv[1:]
    for filename in args.filenames:
        try:
            YAML_PROJ_CONF = yaml.safe_load(open(filename))
            PROJECT_KEY = YAML_PROJ_CONF.keys()[0]
        except yaml.YAMLError:
            print('Error parsing: {}'.format(filename))
            retval = 1
            continue

        if 'key' in YAML_PROJ_CONF[PROJECT_KEY]:
            if not re.match(r"^[A-Z]{2,10}$", YAML_PROJ_CONF[PROJECT_KEY]['key']):
                print("{} isn't a valid key. The project key MUST be 2-10 characters, only A-Z".format(YAML_PROJ_CONF[PROJECT_KEY]['key']))
                retval = 1
        else:
            print('{}: is missing project key'.format(filename))
            retval = 1

    if not re.match(r"^[a-z]{2,10}$", YAML_PROJ_CONF[PROJECT_KEY]['lead']) and 'svc-' not in YAML_PROJ_CONF[PROJECT_KEY]['lead']:
        print("{} isn't a valid lead. The lead MUST be 2-10 characters, only a-z".format(YAML_PROJ_CONF[PROJECT_KEY]['lead']))
        retval = 1

    if YAML_PROJ_CONF[PROJECT_KEY]['board_config'] not in VALID_BOARD_CONFIGS:
        print("{} isn't a valid board config. You can choose between the following options {}".format(YAML_PROJ_CONF[PROJECT_KEY]['board_config'], VALID_BOARD_CONFIGS))
        retval = 1

    return retval

if __name__ == '__main__':
    sys.exit(main())

Jenkins (our continuous integration tool) puts all the pieces together by merging the three Git repositories, verifying that all new and existing YAML files pass validation through pre-commit hooks, and then executing in Ansible. The payload that’s built is based on the information included in the new manifest files and is executed against Jira’s API to process the changes.

Our Jenkins pipeline consists of four different stages:

1. Review validation: In order to avoid engineers pushing code without a code review, we’ve customized our Gitolite permissions to ensure engineers only push integration branches. By doing this, we avoid rogue changes in our repositories that haven’t undergone proper review from other team members.
2. Manifest validation: Runs pre-commit hooks against all YAML and configuration files and validates that they pass.
3. Playbook execution: If the first two stages pass, Jenkins executes the proper playbook based on the changes that were made to the repository.
4. Notification: If the Ansible project creation/update/deletion is successful, it notifies the requestor that the changes are now live.

Implementation

Now that you’re familiar with the technologies involved, let’s talk about implementation. This chart illustrates how each of these components works together to deploy changes automatically:

When an engineer wants to create, update, or delete a Jira project in our main production instance, they simply clone the “ansible_jira_projects” repository, which contains all existing project YAML files. At this point, engineers have the option to manually update an existing project YAML file or to use one of our self-service scripts to automatically generate a new project manifest based on prompted details:

Pre-commit hooks are run when an engineer makes a Git commit to submit the changes for review.

At this point, the engineer submits a code review. Included in the review are the results of all tests run in pre-commit hooks.

Once the code review receives a “ship it!” from one of our team members, the engineer can merge the changes to a deploy branch using a simple script.

Jenkins will get a notification of new changes in our Git repositories.

Benefits

By using Ansible, Jenkins, pre-commit hooks, custom written Python scripts, and easily comprehensible YAML we were able to:

• Ensure 100% standardization across all engineering projects, making Jira easier to manage, simpler to use, and better performing.
• Establish a rich auditing trail by ensuring that every action taken during the Jira project lifecycle is captured, reviewed, and logged.
• Eliminate repetitive tasks for our Jira administrators.
• Reduce the turnaround time of Jira project creation/update/deletion while also freeing our Jira admins to focus on other important tasks.
• Reduce the number of Jira global administrators.

Back to blog