Yelp relies heavily on streaming to synchronize enormous volumes of data in real time. This is facilitated by Yelp’s underlying data pipeline infrastructure, which manages the real-time flow of millions of messages originating from a plethora of services. This blog post covers how we leverage Yelp’s extensive streaming infrastructure to build robust data abstractions for our offline and streaming data consumers. We will use Yelp’s Business Properties ecosystem (explained in the upcoming sections) as an example. Key terminology Let’s start by covering certain key terms used throughout the post: Offline systems - data warehousing platforms such as AWS Redshift or...

Continue reading

While we once used Elasticsearch at Yelp, we have since built a replacement called Nrtsearch. The benefits and motivations of this switch can be found in our blog post: Nrtsearch: Yelp’s Fast, Scalable and Cost Effective Search Engine. However in this blog post, we will discuss the motivations behind building Nrtsearch Coordinator - a gateway for Nrtsearch clusters. We will also go over how Nrtsearch Coordinator adds sharding logic to Nrtsearch, handles scatter-gather queries, and adds support for dark/live launching cluster improvements. Motivations We traditionally used a gateway to call Elasticsearch, which provides metrics, isolation rate-limiting per client, and geo...

Continue reading

At Yelp, Apache Spark and JupyterHub are heavily used for batch processing and interactive use-cases, such as in building feature models, conducting ad-hoc data analysis, sharing templates, making on-boarding materials, creating visualizations, and producing sales reports. Our initial deployments of Jupyter at Yelp were iPython notebooks managed at an individual level. Later on when Jupyterlab was released (2018), our notebook ecosystem was extended to Jupyter Servers running on dev boxes, which was managed by individual engineering teams. Over time with growing use-cases and data-flow, this introduced unnecessary version variability, became error-prone due to the number of manual steps, caused config...

Continue reading

Merging code safely can be quite time consuming for busy repositories. A common method is to test and merge branches serially, and one at a time, in order to ensure the safety of the main branch. However, this method does not scale well when many developers want to merge code at the same time. In this blog post, you’ll see how we’ve sped up code merging at Yelp by creating a batched merge queue system! Why Merge Queues? In our blog post about Gondola, our frontend Platform as a Service (PaaS), we talked about the benefits of moving to a...

Continue reading

Keeping project dependencies up to date is an ever-growing concern. An increasing number of dependencies is used for even the most simple applications. It’s easy for teams to deprioritize maintaining them, resulting in numerous security vulnerabilities. As dependencies become increasingly out of date, the level of effort to get a project into a good state increases significantly. Teams may even get blocked by outdated dependencies when doing critical development work. Being proactive about applying upgrades goes a long way. Tools like Dependabot can really help with this. But what if you’re trying to enforce these practices across hundreds of teams...

Continue reading

In 2018, Yelp switched from using the MVP architecture to the MVI architecture for Android development. Since then, adoption of our new MVI architecture library has risen and we’ve seen some great performance and scalability wins. In this blog post, we’ll cover why we switched to MVI in the first place, how we managed to get performant screens by default, and our take on unit testing MVI. What is MVI? One of the main reasons to use an architecture is to make things easier to test by separating concerns. For Android, this means keeping the Android SDK out of our...

Continue reading

Yelp aims to offer easily accessible high-quality content. We need to tag, organize and rank online content to attain this goal. For this purpose, Yelp engineers have started using general embeddings on different data. It improves usability and efficiency for all kinds of model development. Having embeddings that encapsulate semantic information readily available for the massive amounts of data Yelp owns makes implementing new deep learning models easier, since it can serve as an excellent baseline for any model input. This blog post discusses how the Content and Contributor Intelligence team generates low-dimensional representations of review text, business information and...

Continue reading

The Yelp website serves millions of users and business owners each day, and engineers in our product teams are constantly adding and improving features across hundreds of pages. Webcore, Yelp’s frontend infrastructure team, is always looking to ensure that web developers can ship their changes quickly and safely, without the burden of maintaining complex team-specific infrastructure. To achieve this, we made some significant changes to our internal deployment model for React pages in late 2019. This blog post will explain why we made these changes, describe the new architecture we implemented, and share some of the lessons we learned along...

Continue reading