« July 2012 | Main | December 2012 »

August 2012


Firefly: Illuminate Your Website's Performance

One of Yelp's core values is "play well with others."  So it's no surprise that Yelp thrives with open source projects written by others, and gives back by sharing projects of our own.  That's why I'm excited to share this post by the manager of our Infrastructure team, Oliver N. (or as he's known around the office, "BigO"), which adds to our library of open source projects.  


How do you know if your website slows down as a result of a code push?  How do you keep tabs on the performance of your most important endpoints?  How do you know if your error rates spike, or what their baselines are?  If you're not actively using it, how do you even know your website is serving traffic?  For Yelp's Infrastructure team, the answer is an emphatic "GRAPHS".  Tasked with keeping the site up and running smoothly, we rely heavily on graphing the data from a variety of real-time metric systems.  We keep these graphs open on our work computers as well as splashed across large LCDs in our office, and they communicate to us the heartbeat of a system that serves approximately 78 million uniques per month.  Today we are releasing the home-grown tool we use to navigate, explore, annotate and graph these time series metrics on github.  Meet Firefly:


Firefly is not a metric collection system itself.  It is a pluggable front-end for exploring time series metrics stored anywhere, combining these metrics into graphs, combining those graphs into dashboards, and sharing those dashboards with colleagues.  It's designed to scale across datacenters and to bring together data from disparate sources.  It supports automatic graph titles and legends, as well as highly configurable graph annotations (vertical bars to record particular events). It has a stateful UI based on HTML5 pushState to allow easy sharing of dashboards.  Its server is built in Python on top of the Tornado framework, and its front-end is built in JavaScript atop the fantastic D3 graphing engine.

Firefly consists of a Data Server and a UI Server.  The Data Server can run in multiple datacenters and provides the primary data interface - responding to requests that essentially translate as "give me the time series data for these particular metrics over this particular time window."  It is the abstraction between a unified UI and a potential myriad of backends, be they Ganglia RRDs, HyperTable, MySQL, Redis or any other data source.  Each distinct data source is described as a (surprise) DataSource subclass, which exposes the standard interface methods of list_path(path), data(sources, start, end), legend(sources), and title(sources).  The UI server configures and serves up the HTML and JavaScript base interface, and handles the URL minification/expansion that underlies the stateful pushState features.

Firefly has a ton of features, and has been super useful to us.  We're going to keep expanding those features and are also really interested in seeing what uses the community can find for this tool.  We're only shipping it with a DataSource configured for Ganglia right now, but adding new sources is designed to be easy and over time we'll be looking to release some more parts of this system.  Fork Firefly on github, give it and shot and let us know what you think!


GPS vs WiFi: The Battle for Location Accuracy Using Yelp Check-Ins

Our next post comes from Mason G., superstar intern on our IOS team.  While not assembling a 20-monitor wide version of Mario that displays the entire level at once, Mason has been cranking out features for our shiny iPhone and iPad apps.  Yelp thrives on data, so it is no suprise that one of Mason's projects was to evaluate the technologies we use for pinpointing users' geolocation.  Read on to get the full scoop!

“Checking in” to a business using Yelp’s mobile applications is a fun way for Yelpers to engage with businesses and others in the Yelp community, but we don’t allow users to check in from just anywhere -- they have to earn those badges! In order to make sure everyone’s playing fair, we compare a user’s reported location with the location of the business they’re trying to check in to figure out how far away they are. We thought it would be interesting to figure out how Wi-Fi positioning and accuracy compared to GPS and, in order to do this, we had to take a look at Yelp check-ins from different mobile devices.

First, a little background. A phone can figure out its location in several ways: by using GPS, cell tower triangulation or Wi-Fi positioning. GPS connects to satellites orbiting the Earth, and figures out where it is compared to the satellites. Cell tower triangulation connects to nearby cell towers, and performs a similar calculation. Wi-Fi positioning depends on companies like Skyhook, who record the locations of Wi-Fi networks. Your phone can then look at nearby Wi-Fi networks and figure out where it is. 

Using Yelp’s open-source map reduce framework, mrjob, we analyzed all Yelp check-in data from June 2012, breaking it out by device type. Android and iPhone can use GPS, cell tower triangulation and nearby Wi-Fi networks to get a precise fix. On the other hand, iPods can only use Wi-Fi positioning. By comparing iPods to Android and iPhones, we were able to see how accurate Wi-Fi positioning is in comparison to a mix of all three techniques.

After gathering all the data from June, we graphed a probability distribution function of Yelp check-ins versus distance in order to see the distance between users and the businesses they’re checking in to. This graph shows the percentage of check-ins attempted (y-axis) at a given reported distance from a business (x-axis). As you can see, check-ins from iPhones tend to be closest to the business, followed by Android phones and iPods.

For example, 40% of Android check-ins are less than 0.1 miles away from the business.


Next we converted this graph into a cumulative distribution function, in order to see how many Yelp check-ins occurred within a specific distance from the business. iPhone fared better than Android, likely due to lower-quality GPS chips in some Android devices. For example, 7.5% of Yelp check-ins from Android, the phone thought that the user was actually more than two miles away from the business, while only 4.5% of iPhone check-ins were that far away.


For example, 79% of iPod check-ins were less than one mile from the business.


When checking in, we also record the reported location inaccuracy. This number represents the phone’s best guess at the number of miles it might be off by. We account for this inaccuracy by subtracting the number from the distance to the business. This way users don’t miss out on the chance to check in.


The dot shows your reported location, and the larger circle represents the inaccuracy.


Android devices gain the most from this comparison, although iPhones still have better location services overall. However, the iPod line barely moves – the reported accuracy is always close to 0.


For example, 90% of Android check-ins were less than one mile from the business after adjusting for the accuracy radius.


By analyzing millions of data points, we can easily see how, on average, different platforms perform. iPhones consistently have the most accurate positioning, with a fairly small accuracy radius. Android phones are often inaccurate, but reliably reported that inaccuracy. And finally, iPods using Wi-Fi positioning proved the least accurate and usually reported incorrect accuracy radii. We can use this analysis to make sure that users have a great experience checking in, no matter their device.