This blog focuses on how Yelp successfully implemented a multi-year, cross-organizational initiative to modernize its billing processes. The goal was to automate its revenue recognition system by enhancing integration capabilities with third-party financial systems, all while maintaining the accuracy and reliability our users expect.

Summary

When Yelp first developed its billing system a decade ago, the database design was based on the requirements known at that time. These initial choices laid the foundation for the billing system, upon which multiple Yelp systems and processes were built. However, as the company evolved, it became evident that these design choices were not ideal and led to various challenges.

Image credit: XKCD

The legacy design choices of Yelp’s billing system became a significant blocker when the company sought to integrate with a third-party revenue automation tool due to data format discrepancies. This integration was essential for scaling Yelp’s revenue system to match the company’s growth, with a target completion date of July 2024. However, Yelp’s unique handling of invoices led to misalignments, much like trying to fit mismatched pieces into a jigsaw puzzle.

Image credit: Created by authors on Canva

To address these challenges, Yelp decided to overhaul its billing system to align with industry standards. This initiative required making changes to business-critical systems, where any disruption could have severe consequences, such as the inability to bill or charge customers. The complexity of executing this initiative best relates to the analogy of changing the tire of a car while it’s still running. To ensure a smooth transition, Yelp developed an execution plan, coordinating efforts across multiple teams over several years.

Image credit: Created by AI

Summary

Yelp’s legacy billing system introduced the concept of “Invoice Obviation” around a decade ago, which caused a customer’s unpaid balance to be carried over from one invoice to the next. This concept became central to the billing behavior, with the most recent invoice reflecting the total balance of the account.

The below diagram shows how a payment needed to be only applied to the most recent invoice in Yelp’s legacy billing system, requiring a one to one relationship between payment and invoice.

Image credit: Created by authors on Canva

A few years later, we realized that standard billing systems do not roll over balances from one invoice to another, which caused Yelp’s solution to behave very differently from industry norms.

The diagram below illustrates how, in a standard billing system, a single payment can be collected once but applied to multiple invoices, allowing one-to-many relationships between a payment and invoices.

Image credit: Created by authors on Canva

This caused data discrepancies in Yelp’s system vs any other standard billing system as shown in the table below:

Image credit: Created by authors on Canva

In Yelp’s legacy system, payments intended for January, February, and March were incorrectly applied solely to March’s invoice, resulting in the revenue being recorded in the wrong revenue period and making it hard to determine invoice aging. This misapplication affected not only payments but also other concepts like credit and debit memos. These inconsistencies also hindered Yelp’s ability to integrate with third-party revenue automation and other financial tools.

To address this, Yelp decided to invest in a more robust billing model that aligns with current requirements and supports long-term growth. The new model aims to:

• Eliminate the concept of invoice obviation.
• Enable one-to-many relationships, allowing payments and credit/debit memos to be applied across multiple invoices.
• Correct data discrepancies by ensuring accurate revenue allocation within specific revenue periods, thereby improving financial accuracy and reporting.

This blog post does not delve into the solution itself but rather explains how Yelp implemented an execution plan and delivered this initiative at scale.

Execution Plan

Implementing such changes required over two years of collaborative effort from a team of more than 50 people because we had to fundamentally change the foundation and then rebuild the functionality on top of it. Therefore, developing a robust execution plan was the key to achieving the successful delivery of this initiative. This blog post focuses on the approach taken to execute this massive initiative and roll it out without impacting any of Yelp’s systems ensuring users were billed seamlessly and accurately.

Step 1: Requirement Gathering

It was crucial to ensure that the system being redesigned using significant engineering resources was not short sighted and truly matched the needs of the business. We maintained close collaboration with business stakeholders to ensure we gathered the right long-term requirements.

Cross-functional requirement gathering often mirrors the popular childhood game of Telephone. The game illustrates how a message could end up distorted as it is conveyed from person to person. The more people involved, the higher the likelihood of distortion. A common cause of the communication distortion is an individual’s choice of words and the difference in comprehension of those words between the involved parties. We settled on a ubiquitous language concept from Domain Driven Design principles early on; modeled after pre-existing accounting terminologies and relying on domain experts as the single source of truth. Armed with a common vocabulary, we were able to reach a high level consensus on the final state of the system.

Step 2: Target Architecture

To meet the specific set of requirements outlined by our stakeholders, we explored a variety of third-party solutions that are widely used in the industry. However, none of the off-the-shelf products could fully satisfy all the new requirements along with Yelp’s existing use case. After careful consideration, we decided to develop a custom billing architecture tailored specifically to our business model.

We had the option to patch the existing system, which might have seemed like a simpler solution. However, this approach carried the risk of introducing unintended side effects. Instead, we took inspiration from industry standards for billing architectures to define our target state. Although the target state represented a long roadmap, we committed to moving towards it gradually through multiple projects over the next quarters and years. This strategic decision was key to successfully delivering the project.

Step 3: Project Planning

Identify Dependencies and Order of Execution

After defining the target architecture, we mapped out all necessary processes and their dependencies. For instance, to support account-level payment collection, it was essential to enable the new payments functionality, implement account-level refunds, and ensure that the user interfaces accurately displayed both payments and refunds at the account level. We prioritized these processes based on stakeholder/business needs. All features required to support a specific process were identified as a single deliverable, helping us in managing interdependencies effectively.

Image credit: Created by authors on Canva

Cross Functional Teams

Tackling projects of this scale required changes across multiple systems, involving the participation of several teams. Instead of having each team take ownership of work within their respective domains, we created multiple cross-functional teams of 4-6 engineers, each assigned to an ongoing project. We called them the Tiger Teams—focused, invincible, and goal-driven! The majority of the teams lasted for the duration of a project, and at any time, 2-4 teams worked in parallel, adapting to evolving priorities.

Coordination Across Teams

An Engineering Manager was assigned to oversee these tiger teams, managing the timelines, staffing, and prioritization. Multiple new processes were introduced to help the EM manage these cross-functional teams:

• Sprint planning is typically done for individual teams at Yelp. However, for the tiger teams, we developed a new process to conduct sprint planning that facilitated cross-functional collaboration.
• A Gantt chart was maintained by the EM to track timelines, staffing, and blockers. This chart was also used to update leadership and stakeholders, building trust in the path forward and keeping them informed.
• Regular sync-up meetings were set up to track blockers, and if any arose, the teams worked proactively to resolve them.

The Engineering Manager’s role was crucial in ensuring that each tiger team could deliver their projects in realistic timelines.

Step 4: Incremental Delivery

We learned the true meaning of iterative development by delivering smaller changes in multiple iterations, always ensuring we provided a minimal valuable product. While we sometimes compromised on the number of features delivered, we never delivered unfinished ones. Initially, we limited rollouts to a small group of customers who didn’t require advanced functionality. This approach allowed us to roll out to ensure correctness in a controlled environment while continuing to support these customers as additional features were developed.

Naming our rollouts proved to be a successful strategy. We chose F1 racetrack names for the rollouts, adding an element of fun and improving communication with stakeholders. This made it easy to convey that feature X was part of the Baku rollout, while feature Y would be available soon in the Suzuka rollout.

A significant achievement was our successful collaboration with business stakeholders. The Finance teams, accustomed to a 100% crossover rollover strategy, initially lacked experience with iterative rollouts, which are more common in engineering. They were skeptical of its reliability due to the nature of their domain. However, through effective communication and demonstration of the approach’s benefits, they became confident and appreciative of the iterative method.

Image credit: Created by authors on Canva

Step 5: Single A/B Rollout

A/B testing is commonly used in the industry to measure the performance of new features by subjecting users to different experiences. By measuring differences in key metrics between the two groups, we can ensure that new features do not negatively impact our business.

Rather than A/B testing individual features introduced in this project, we opted to incrementally release new features to a singular group of users. This avoids the complications of managing multiple experiments and simplifies the measurement of key metrics.

Image credit: Created by authors on Canva

Step 6: User Acceptance Testing

The impact of making such a fundamental change was very widespread. We were unable to rely solely on automated tests, as the technical changes affected the business processes of many non-engineering teams, including Finance, Accounting, Customer Support, Analytics, and more. Therefore, we decided to be extremely thorough with the user acceptance testing to allow us to test the correctness of the system and the processes.

To ensure comprehensive coverage, we involved stakeholders from all the affected teams in the user acceptance testing process. We created detailed test plans and scenarios that covered every aspect of the new billing system. Each stakeholder was responsible for verifying that their systems and processes functioned correctly and as desired.

We documented our testing process in a Google Sheet and stakeholders were asked to execute the test cases relevant to their domain and mark them as pass or fail. This collaborative approach helped us identify any unintentional impact on internal processes and systems early, ensuring a smooth transition to the new system.

Image credit: Created by authors on Google Sheet

Step 7: System Observability

Ensuring system observability was crucial to roll out a foundational change quickly to all customers to meet the company’s revenue automation timeline of July 2024. We implemented a multi-layered approach to monitor and maintain the integrity of the system:

• Alerts/Logging and Monitoring: We set up comprehensive alerts, logging and monitoring dashboards to provide real-time visibility into system performance and anomalies.

• Integrity Checkers: We developed integrity checkers as our last line of defense. These checkers were designed to continuously validate data in product for consistency and report any anomalies that deviated from the expected behavior.

• Stakeholder Dashboards: We created dashboards for the stakeholders, providing them with relevant metrics and insights. This transparency helped build trust and allowed stakeholders to monitor the progress and stability of the system.

By combining these observability practices, we were able to ensure that any discrepancies were caught and addressed before they could impact the customers.

Successful Delivery

Even though the initiative required a massive effort from over 50 people working collaboratively for more than two years, it was successfully delivered by following the structured execution plan outlined above. Unlike traditional rollouts at Yelp for critical systems, which usually involve a gradual release, we adopted an accelerated approach to meet our tight timelines for integrating with third-party revenue automation systems. We initiated the update with a small group of customers in October 2023, and by steadily increasing the rollout pace, we achieved 100% customer adoption by July 2024.

The rollout speed was unusually high for our organization, as the billing system requires high accuracy and customers typically receive only one bill per month, providing limited opportunities to verify the updates’ correctness.

Image credit: Created by authors on Canva

This execution plan allowed us to meet our deadlines while ensuring a seamless transition without compromising system integrity or functionality, thanks to the dedication and cross-functional coordination of the team.

Acknowledgements

We would like to thank everyone across multiple organizations at Yelp for their continued support and tenacity in making this a reality. Their efforts have been crucial in helping Yelp automate 90% of its revenue by strengthening the data foundation.

What’s Next

Keep an eye out for our next blog posts on Yelp’s integration with the third party financial tool, where we’ll dive into how we automated revenue processes once the new billing system was in place.

Back to blog