In our previous post, we discussed how we used PySpark to build a large-scale distributed machine learning model. In this post, we will describe our experience and some of the lessons learned while deploying PySpark code in a production environment. Yelp’s systems have robust testing in place. It’s a hallmark of our engineering. It allows us to push code confidently and forces engineers to design code that is testable and modular. Broadly speaking, we found the resources for working with PySpark in a large development environment and efficiently testing PySpark code to be a little sparse. By design, a lot...

Continue reading

Here at Yelp our core mission is to connect users with great local businesses. On the engineering side, that requires us to tackle big data and machine learning problems. One of the natural ways to apply machine learning to our core mission is to deliver relevant and personalized business recommendations. This helps us deliver great content to users. For example, we can let users know when a hot and new coffee shop we think they will love opens in their neighborhood through push notifications or features like collections. This is an area of active research and development at Yelp, and...

Continue reading

#perfmatters At Yelp, we’ve been working hard to improve the performance of our search results on mobile devices (iOS and Android). We know that performance matters from a numbers perspective, but another reason why we’ve decided to invest more in performance is due to a recent increase in user studies commissioned by Yelp. We noticed that most users “grunted or made noises when waiting for search to load”. We took this as a sign that something needed to be done! In our quest to make search faster on the Android mobile app client, we broke down performance into two different...

Continue reading

What is Salsa? Salsa is an open source library that renders iOS views and exports them into a Sketch file. We built Salsa to help bridge the gap between design and engineering in an effort to create a single source of truth for visual styling of UI. Why use Salsa A few years ago, we started putting together a library of common components that developers and designers could use to build features. Initially, we had to manually maintain consistency between the designs in Sketch and their implementations in code. When we only had a handful components, this wasn’t difficult to...

Continue reading

Many enterprise professionals use passwords that are weak and easily compromised. Equipped with this knowledge, as well as the exposure of more and more password leaks, dictionary attacks focused on compromised or popular passwords have become increasingly effective. As such, the National Institute of Standards and Technology recommends password blacklisting as a highly-effective means of preventing such attacks. Unfortunately, use of password blacklisting countermeasures has remained a relatively new innovation that has not yet achieved widespread corporate adoption. At Yelp, however, we strive to add the latest and greatest defense mechanisms to our arsenal, which is why we adopted such...

Continue reading

Since Yelp introduced its real-time streaming data infrastructure, “Data Pipeline”, it has grown in scope and matured vastly. It now supports some of Yelp’s most critical business requirements in its mission to connect people with great local businesses. Today, it has expanded into a diverse ecosystem of connectors sourcing data from Kafka and MySQL, and sinking data into Cassandra, Elasticsearch, Kafka, MySQL, Redshift, and S3. To ensure that the whole ecosystem is functioning correctly, Yelp’s Data Pipeline infrastructure is continually growing its repertoire of reliability techniques such as write-ahead logging, two-phase commit, fuzz testing, monkey testing, and black-box auditing to...

Continue reading

With hundreds of engineers, developers and designers working on Yelp, ensuring visual consistency across Yelp is a challenging task. We’ve been migrating our web components from Yelp Cheetah to React to increase designer and developer productivity while ensuring visual consistency across our web app. Along the way, we built Lemon Reset - a package containing consistent, cross-browser React DOM tags, powered by CSS Modules. Since our design system components are the building blocks of our frontend, we had to port them to React as the first step before our developers could port their features. We made a lot of design...

Continue reading

Introduction The collection and processing of logs is essential to good security. One of the primary functions of a security team is to keep organizations safe by eliminating blind spots in infrastructure. Breach investigations without logs result in a lot of guesswork. Worse, the activities of an attacker can easily remain undiscovered without adequate logging. To ensure we have a robust log storage and visualization platform, we use Elasticsearch, Logstash and Kibana (ELK). These tools form part of the toolset that we use in our Security Incident and Event Monitoring (SIEM) solution. ElastAlert is the primary means by which alerts...

Continue reading