Keeping project dependencies up to date is an ever-growing concern. An increasing number of dependencies is used for even the most simple applications. It’s easy for teams to deprioritize maintaining them, resulting in numerous security vulnerabilities. As dependencies become increasingly out of date, the level of effort to get a project into a good state increases significantly. Teams may even get blocked by outdated dependencies when doing critical development work.

Being proactive about applying upgrades goes a long way. Tools like Dependabot can really help with this. But what if you’re trying to enforce these practices across hundreds of teams and thousands of projects? And what if you have complex requirements that need to be enforced? At Yelp, this is where the Yokyo Drift service comes in.

Yokyo Drift actively scans all repositories in use at Yelp. It submits pull requests that upgrade any outdated dependencies, and tracks and monitors the progress of these upgrades.

Requirements

Building a generic solution that works for the majority of projects is challenging. Projects should be relatively standard. This is encouraged by providing a variety of tooling and quality of life upgrades to repositories that adhere to the Yelp standard. The more a project deviates from the standard, the more difficult it becomes to keep it automatically up to date.

In addition, projects must have a robust testing pipeline and good test coverage. Thorough automated testing should run as part of the CI pipeline before any change is accepted. Upgrading dependencies is likely to introduce bugs, and inadequate testing means that teams may not feel confident merging upgrades, thus encouraging them to stick to outdated dependencies. #Tracking Project State Batch jobs regularly collect and index a variety of information about Yelp repositories. Yokyo Drift monitors the specific dependencies that are used throughout the organization. When a vulnerability is discovered in a dependency, we can immediately identify all affected repositories and dispatch a fix to rapidly eliminate the vulnerability. All indexed information is available in a simple UI.

Upgrading Dependencies

Scheduled Upgrades

We encourage teams to always keep their project dependencies up to date. Small, frequent updates are much easier to manage. Repository owners can configure how frequently they’d like to receive updates. Yokyo Drift performs both major and minor version upgrades, typically on a monthly or quarterly basis.

Yelp projects rely on curated package repositories, and we are only able to upgrade to these pre-vetted versions, thereby ensuring we don’t introduce any unwanted security issues.

Scheduled upgrades are randomly distributed throughout the month. This ensures a consistent use of resources with few spikes. More importantly, it allows our teams to provide support to repository owners and not overwhelm them with too many pull requests at the same time. Performing upgrades for all repositories in one day would result in an overwhelming number of questions in a short amount of time.

Targeted Upgrades

Targeted upgrades allow us to upgrade specific libraries to specific minimal versions across the entire organization. These can be invoked dynamically by other teams using the Yokyo Drift API, or manually using the UI shown below.

This functionality is frequently used by security teams. Once a vulnerability is discovered in a specific version of a library, we can immediately see the impact and deploy a mass upgrade across all of Yelp’s projects. We then actively monitor the progress and ensure the vulnerability is eliminated in all of Yelp’s systems.

Library owners are also frequent users of targeted upgrades. They can rapidly deploy bug fixes and other improvements to all relevant projects.

Pull Requests

All changes are submitted as pull requests in GitHub. Since changes go through the existing CI pipeline, a variety of security and automated tests are executed. We rely on the Ownership service to determine the relevant team responsible for each repository. Pull requests are assigned for review to one of the repository’s owners, who is responsible for manually fixing small changes that may be required by library upgrades. The change automatically gets merged once all checks pass and the repository owner approves the pull request.

Occasionally, teams will be unable to review these pull requests in a timely manner, so automated reminders are sent to the reviewer at a set interval. In addition, Yokyo Drift attempts to always keep the pull request up to date. Merge conflicts are avoided by regularly pulling the latest changes from the master branch and performing the upgrade again if needed.

Scaling

Updating dependencies on one repository can be time-consuming. It may involve building the project, performing dependency resolution, and even running some automated checks. This is manageable when upgrading a single repository, but quickly becomes unusable when upgrading hundreds or even thousands of repositories. To address this, we need to be able to automatically scale up and down as needed.

Creating a new upgrade job enqueues a payload for each repository that needs upgrading. Workers are then responsible for taking items off the queue, performing the necessary changes, and submitting the pull requests. Workers are configured to automatically scale up as queue size increases and scale back down when the queue clears. Because of this, thousands of complex upgrades can be executed quickly.

Progress Tracking

The Yokyo Drift UI tracks the progress of each task. A typical successful task will move through the following stages: pending, in_progress (the upgrade is in progress), open (pull request is open), and merged.

The job progress page keeps track of how these updates affect repositories. A status of “checks_failed” indicates that the repository is failing automated tests. This status is not uncommon, however, a large number of repositories failing tests may indicate a fundamental problem with the upgrade. Migration authors such as package owners can investigate this and determine if any changes should be made, the end goal being to reduce friction with teams and make these upgrades as easy as possible to integrate.

This progress screen can also be used to directly control job progress. Upgrades can be rerun on individual repositories, the entire change can be canceled or reverted, and teams can be nudged to review and approve the changes if necessary.

Conclusion

Dependencies can easily become outdated and cause significant problems to development teams. Updating them regularly makes the process more manageable and reduces the number of security vulnerabilities. Enabling teams to upgrade a single dependency across thousands of projects is valuable, both for security teams and dependency developers.

Acknowledgements

Thanks to Luis Perez, Kyle Deal, James Flinn, Jason Tran, Rebecca Fan, Mitali Parthasarathy, Hanna Farah, and many others that have contributed to Yokyo Drift over the years.

Back to blog