Menus aren’t just lists, they’re a window into a restaurant’s unique offerings, specialties, and personality, shaping where and what we choose to eat. But here’s the challenge: reading “Kung Pao Chicken - Stir-fried chicken with peanuts in spicy sauce” doesn’t tell you what the portions look like, whether other diners loved it, or if it matches your expectations.

At Yelp, we knew we had the solution sitting in our user-generated content: hundreds of millions of photos, reviews, and prices for dishes. The problem? Users had to manually search for each dish, an experience that doesn’t work well when you’re at a restaurant table making quick decisions.

Menu Vision solves this problem: point your camera at a menu, and it uses AI, augmented reality, and text recognition to instantly identify dishes and show you what other diners photographed and reviewed. Behind this seemingly simple interaction lies a sophisticated system combining on-device ML, real-time computer vision, fuzzy matching, and a distributed data pipeline processing millions of dishes across millions of restaurants.

How It Started: The Hackathon Spark

During a Yelp hackathon, a team of engineers explored how to make dish discovery more seamless for diners. Rather than forcing users to manually search for each menu item, they built a proof-of-concept that used your phone’s camera to bridge the gap.

The team developed an Android prototype using a third-party ML kit for real-time text recognition and CameraX for smooth camera integration. The scanner analyzed live camera frames, extracting visible text and matching it—after basic text normalization—against a curated list of menu items. When a recognizable dish was detected (like “Clam Chowder”), it appeared as a tappable pill displaying the number of available Yelp photos (e.g., “CLAM CHOWDER (969)”). Tapping the pill brought up a photo grid, filtered for that specific dish.

Despite being built in just two days, the prototype addressed a clear user pain point and won top honors at the hackathon, validating the concept and setting the foundation for production development.

Making It Production-Ready

Production development began in September 2025 with a clear focus: deliver the feature across both iOS and Android in six weeks while maintaining optimum backend latency and real-time dish recognition. To rapidly prototype solutions and iterate on implementation details, the team leveraged agentic coding tools, which helped to accelerate development.

Building a production-ready system at scale presented several key challenges: we needed to fetch dishes for millions of businesses with low latency, match and display relevant dishes in real-time, ensure accuracy across diverse menu formats and languages, and surface the feature at the right moment without disrupting the core browsing experience.

Here’s how we tackled these challenges:

Data Source

At Yelp, we have historically maintained menu data from third-party providers and also generated inferred menus from user review content. While these inferred menus have been used to power features like Popular Dishes by highlighting dishes mentioned by diners, their coverage has remained limited.

To increase coverage, we expanded the dataset by extracting dish signals not only from reviews, but also from user-submitted photo captions. Photo captions often contain direct references to dishes shown in images, making them a valuable source for connecting menu items with real diner photos.

For our feature, we only needed dishes with at least one associated photo, which meant we could tap into a much broader set of this curated data. By combining menu data from business owners and partners with this expanded curated dataset, we significantly increased the number of businesses we could support with Menu Vision, ensuring the feature could serve a much wider audience.

Our approach was straightforward:

• Step 1: Combine all the menu data sources (i.e., from business owners and third-party partners to in-house curated data).
• Step 2: Deduplicate the combined list to ensure each dish appeared only once.
• Step 3: Filter the menu data to include only dishes with at least one photo.
• Step 4: Store the final collection of dishes in a Cassandra database for fast and reliable retrieval.

This approach allowed us to significantly expand business coverage without building a new system from scratch—and, by pre-processing the data and storing it in a Cassandra database, we ensured data retrieval times consistently met our low latency targets.

These key decisions, reusing existing data sources and pre-processing for speed, allowed us to scale quickly without sacrificing performance.

On-Device Processing for Seamless Scanning

For the production version, we maintained the approach from our hackathon prototype by performing all text recognition, text cleanup, and dish matching directly on the client using each platform’s native ML frameworks. This on-device approach allowed us to avoid sending images to the server, minimizing latency and eliminating concerns about varying internet speeds.

To further reduce wait times, we decided to prefetch and locally store dish data as soon as a user opens a business page. This means that when users launch the scanner, all necessary information is already on their device, eliminating the need for additional client-server communications.

To improve the scanning experience and help users get more accurate results, we integrated Apple VisionKit’s built-in visual guidance system. This provides real-time feedback during scanning—such as prompts to “Slow down” or adjust their camera angle, helping users optimize their scanning technique for better text recognition.

For the user interface, we refined the pill design from the prototype. Popular dishes are now highlighted with badges, making them easy to spot. Users can tap these to navigate to Popular Dishes for those items, while other pills direct to the photo search page for non-popular dishes.

By prefetching data and keeping the matching process client-side, we consistently achieved our goal of displaying relevant dishes quickly.

Figure 1: Flow of the feature on the client side.

Feature Discovery

Getting users to discover a camera-based feature requires careful consideration—we needed to surface it at the right moment without disrupting the core browsing experience. We implemented a two-pronged discovery strategy: contextual prompts for users at restaurants, and permanent entry points for ongoing access.

Figure 2: At the Business Entry Point.

At the Business Entry Point

When users visit a restaurant’s business page while physically present at the location, a bottom sheet appears inviting them to “Scan the Menu.” This prompt is powered by Yelp’s existing educator framework, which handles impression tracking, frequency capping, and experimentation. Eligibility is determined server-side for consistency and to keep the client streamlined.

For each business page view, our backend evaluates:

• Location Verification: The system calculates the distance between the user’s GPS location and the restaurant. The feature is only shown to users who are near the restaurant, ensuring it appears when users are actually at the restaurant—not just browsing from elsewhere.
• Content Availability: We query our datastore to confirm the business meets our minimal dish photo threshold before displaying the prompt.
• Frequency Capping: To prevent overexposure, we use a robust capping system with limits on prompt frequency and cooldown periods after dismissal.

By keeping this logic server-side, we maintain a lightweight client and gain the flexibility to adjust targeting rules without needing to release new app updates.

Figure 3: Photo Search Entry Point (left) and Business page Entry Point (right).

Photo Search and Business Page Entry Points

We added two permanent entry points to make Menu Vision easily accessible. The first is a button on the photo search page toolbar, strategically placed near the search bar where users actively looking for dish photos would naturally find it. To help onboard new users, we included a tooltip next to the button for guidance. The second is a button within the menu section of the business page, enabling users who are already reviewing a restaurant’s offerings to quickly launch the scanner.

This dual approach addresses different user needs: the photo search entry point supports users in planning mode who are researching dishes ahead of their visit, while the business page entry point serves those actively browsing menu details and ready to scan when they arrive at the restaurant. To maintain a quality experience, we implemented adaptive behavior: if the system repeatedly fails to find dishes for a specific restaurant (indicating potential data gaps), the feature is temporarily suppressed for that business to avoid frustrating users.

Taking It Live

Menu Vision launched at the end of October 2025 on both Android and iOS. Despite a tight development timeline, our strong quality assurance practices ensured a smooth release, with no major bugs or crashes reported. The rollout was conducted gradually, stage by stage, while we closely monitored client-side logs, server-side load, and error rates throughout the process.

Initial Results

Menu Vision showed promising engagement from the start. Users who discovered the feature were actively scanning menus and clicking through to dish photos, validating our core hypothesis that visual discovery could enhance the decision-making experience at restaurants.

The impact extended beyond immediate engagement: Menu Vision increased retention for users who used the feature. This validated our broader hypothesis that investing in delightful, user-need-solving at-restaurant features can drive meaningful retention improvements. Early engagement metrics indicated strong user adoption and continued usage.

Challenges and Learnings

While the feature showed strong engagement, our logging data revealed several areas for improvement:

• Matching Accuracy: Our initial exact-match algorithm was too strict. Users would scan “Garlic Noodles” on a menu, but our system required an exact character-for-character match with our database entry. Menu variations like “Garlic Noodle” (singular) or “Garlic Noodles w/ Pork” would fail to match, even though we had dish data and photos available. This resulted in missed opportunities to surface relevant content.
• Limited Dish Coverage: The average scan returned fewer dishes than expected. We discovered this stemmed from two issues: our matching algorithm (mentioned above) and gaps in our dish inventory. Some restaurants had dishes with photos in our system, but those dishes weren’t included in our pre-processed dataset due to how we filtered the data.
• Menu Data Quality: Even when matches occurred, we found inconsistencies in dish naming. The same dish might appear as “Pad Thai” in our menu highlights dataset but “Thai-Style Stir Fried Noodles” in food highlights. Without accounting for alternative names and regional variations, we were under-serving our users.

These insights directly informed our roadmap for future enhancements to the feature. The learnings proved that while the core concept resonated with users, the technical execution needed refinement to deliver on the full promise of the feature. Over the next few months we iterated on the experience to deliver a significantly enhanced version of Menu Vision that was more visual, detail-rich, and accurate.

Enhancing Menu Vision: Better Data, Smarter Matching, and a Richer Visual Experience

After incorporating our learnings from the initial launch, we invested in three areas: improving our underlying data quality to expand dish coverage, building a smarter matching algorithm to improve accuracy, and transforming the visual experience from simple text pills to rich, photo-driven dish cards.

Menu Data Quality Improvements

We created a unified, standardized menu for each business by leveraging AI to clean, normalize, and enrich data from multiple sources through a three-step LLM-based pipeline:

• Standardizing Partner Menus: Partner-provided menus were processed through our LLM pipeline to standardize dish names, extract synonyms, and tag valuable attributes such as price, portion size, dietary labels, and calories.
• Processing Customer Language: We applied the same workflow to reviews and photo captions, capturing the authentic ways customers describe dishes. This helped us understand that “Tonkotsu Ramen with Chashu Pork” might also be called “pork ramen” or “tonkotsu ramen with pork belly” in real-world usage.
• Intelligent Combination: Both data sources were merged and deduplicated, with popularity indicators added to create a single, comprehensive menu reflecting the most accurate and complete set of dishes for each business.

Matching Algorithm Enhancements

Beyond improving our data, we enhanced the matching logic itself. Our initial exact-match algorithm was too rigid—”Garlic Noodles” wouldn’t match “Garlic Noodle” (singular) or “Garlic Noodles w/ Pork.” We implemented a three-phase matching process:

• Phase 1: Exact Match: The system first attempts exact character-by-character matching against both the primary dish name and all synonyms. This ensures precise matches when the menu text perfectly aligns with our data.
• Phase 2: Bidirectional Substring Matching: If no exact match is found, the system checks both if the recognized text contains the primary dish name and if the dish name contains the recognized text. This handles variations like dietary indicators or modifiers (e.g., “Spicy Pad Thai (V)” matches “Pad Thai”).
• Phase 3: Similarity Matching: As a final fallback, we apply the Jaro-Winkler similarity algorithm to the primary dish name. This handles spelling variations, OCR errors, and transliterations, with the algorithm prioritizing matches at the beginning of strings—perfect for dish names where key identifiers typically come first.

From Text Pills to Rich Visual Dish Cards

With better data and matching in place, we turned our attention to transforming the visual presentation to be more informative and engaging for consumers. In our initial launch, recognized dishes appeared as simple tappable text pills displaying the dish name. While functional, this required users to tap through to discover what the dish actually looked like.

In our April 2026 update, we completely reimagined how dishes are presented to users. Now, when you scan a menu, dish images appear directly over the menu text in your live camera view, along with the dish name, the number of available photos and reviews, “Popular” badge for standout items, and price information. This transforms the experience from reading a list of matched names to instantly seeing what each dish looks like. Tapping any dish card opens a detailed view with all available photos and relevant reviews.

Figure 5: Dish Cards.

The cards are presented in a smooth, swipeable carousel that handles large menus gracefully through intuitive pagination.

Inventory Fallback and QR Code Detection

We also added smart fallback behavior: if the scanner doesn’t detect any dishes, it automatically displays the restaurant’s full dish inventory—popular dishes first—ensuring users still benefit from the feature even when automatic recognition struggles. Additionally, we integrated QR code detection on iOS, so when users point their camera at a menu QR code, they’re prompted to view the restaurant’s digital menu directly.

Together, these enhancements transformed Menu Vision from a simple text-matching tool into a rich, visual discovery experience that helps diners make confident ordering decisions.

Figure 6: Inventory Fallback (left) and QR Code Detection (right).

Conclusion

Menu Vision represents our commitment to making Yelp indispensable for in-restaurant decision-making. From a hackathon prototype to a production feature driving significant growth in weekly usage, the journey demonstrated that thoughtful technical execution combined with user-centric design can create meaningful impact. The team continues to evolve the feature with additional enhancements and improvements.

Acknowledgements

This was a project that involved dozens of engineers, product managers, marketers and designers.

We would like to acknowledge the efforts of User Engagement, Content and Contributor Intelligence and Applied Machine Learning team for their exceptional contributions, especially Jason Liu, Richa Srivastava, Tobia Crivellari, Rachael Schachtler, Sheng Pu, Prerna Manaktala, Shalini Pusapati, and Shilpa Gopi.

We would also like to thank all the Yelpers for testing, supporting and motivating us to launch this feature.

Thank you!

Back to blog