We’ve extended Nrtsearch with the Inference Plugin, which embeds ML-based ranking directly in the search layer — eliminating the need for a standalone scoring service.

We use Nrtsearch (read more information on the blog post), a Lucene-based open-source search engine built by Yelp, to power a variety of applications such as business search, reviews search, ad delivery and photo search.

In this blog post, we give a high-level overview of the Machine Learning (ML) based scoring workflow in Nrtsearch. We’ll show how ML models are configured and loaded, and how different applications use custom business logic to develop, test, and deploy their own ML ranking.

Motivation

Lucene’s BM25 scoring is primarily term-based. While fast and effective, it relies on fixed hyperparameters and doesn’t learn weights or model nonlinear feature interactions. It also doesn’t natively use richer document or user signals. ML models can help bridge this gap to improve relevance. We introduced a two-stage inference flow: Nrtsearch retrieves candidates and a standalone inference service scores and re-ranks the results.

The picture above illustrates the inference flow with a standalone inference service. When the web server receives a request from the client services:

1. Query Nrtsearch to get eligible candidate ids to return to the client.
2. Fetch features of the candidate ids from feature stores and relevant services.
3. Send the candidates’ features to the inference service to score and rank all the candidates.

While this flow worked for a lot of use cases, there were some scalability challenges with this inference flow:

1. The number of eligible candidates could be large.
2. The size of candidate features could be large.

Both scenarios led to performance bottlenecks due to network transfer overhead and large model feature payloads. To solve this, we designed a new inference flow using an Nrtsearch Inference Plugin. By co-locating feature storage and inference, we were able to reduce latency and serialization overhead. Leveraging Nrtsearch’s custom plugin capability, we developed a solution that met the flow’s requirements, specifically mandating that all features be extracted from the index document or request parameters.

Solution

Leveraging Nrtsearch’s plugin system, we developed the Inference Plugin to embed ML models straight into the ranking pipeline. Now, Nrtsearch can perform model inference on each query within the search engine, delivering faster, more relevant results without external calls. This plugin enhances Nrtsearch by integrating ML models like XGBoost and neural networks for ranking documents. This capability is used in critical Yelp applications such as Search, Ads, and Home Feed to provide users with the most relevant results.

The Nrtsearch Inference Plugin is able to extract features from the index documents and apply the ML model to score the document. In this flow, the client service can directly send the Nrtsearch request to the Nrtsearch cluster. The cluster is able to:

1. Search the index to get eligible document ids
2. Call the Inference Plugin to extract features
3. Score the document with the ML model with the extracted features

In addition to the scalability improvement, the plugin also:

1. Has straightforward access to the real-time features within the search engine context and model context.
2. Enables streamlined iteration and deployment process which simplifies the development and deployment of the inference flow.

The Nrtsearch Inference Plugin is callable in both the Nrtsearch Function Score query and the rescoring phase which provide full flexibility to do complicated inference.

Design Goals

These were the design goals we set for our new system:

1. Compatible with Yelp’s machine learning platform - this would let us easily integrate existing ML models with our Nrtsearch Inference Plugin.
2. Flexible and easy to create custom modular scoring - the Inference Plugin is intended for use across multiple use cases. Each team should be able to integrate the plugin with their own cluster and models with custom java modular scoring code. Each team should only need to extend the interfaces to define their own features extracted from the data and apply the models.
3. Fast & reliable model deployment - this would allow our ML engineers to iterate over the ML models with confidence

High Level Overview

Our ML Scorer acts as the document-level orchestrator that, for every document:

• Chooses which model(s) to apply
• Extracts and transforms features using model configs
• Runs MLeap-based inference for each model
• Aggregates and outputs a final score

This plugin enables pluggable, flexible ML scoring, supporting multi-model execution, batch inference, safe field access, and robust error handling. It serves as a comprehensive solution for search-powered ML relevance architectures.

The following sequence diagram illustrates the complete workflow for deploying and using trained models in Nrtsearch.

We use MLeap and MLflow at Yelp as the framework for our ML applications. ML engineers can store their MLeap-based XGBoost, neural network, and other models in MLflow. Then, they can use our internal config manager to specify which models should be loaded for each cluster.

Nrtsearch clusters employ a built-in TensorFlow-based model server that resides in the same Java Virtual Machine (JVM) on their replica nodes to perform ML-driven ranking for search queries. Unlike the primary node, which manages indexing, these replica nodes are responsible for processing search requests and thus host the model server engines.

Once a replica node that’s configured with ML-based scoring bootstraps, it loads all the model bundles specified in the config file from MLflow. After the replica node is fully loaded, it is ready to serve requests that ask for ML-based ranking using models loaded into this node. The node processes these requests by narrowing down required documents based on the preliminary filters provided in the query. It then extracts features from each document and passes them, along with the model name, to our in-house model server engine that runs inside Nrtsearch to get a score for that document. As an optional step, it combines scores for multi-model requests and attaches the final rescore in the response object that is ultimately presented to users.

All of this happens in a matter of milliseconds to serve live responses.

Creating Custom Scorers

The Nrtsearch Inference Plugin’s modular design allows customization of document recall and scoring stages. We customized the Scorer section by implementing the plugin’s Java interface.:

• Teams can implement the above interface through a custom module that will be loaded into the Nrtsearch Inference Plugin when a cluster is created.
  • Suitable for teams with complex needs to fine-tune every step of scoring.
• Alternatively, teams can simply use the built-in generic ML scorer that comes out of the box with the plugin.
  • Suitable for teams that don’t have complex requirements but require fast onboarding time and quicker developer velocity.
  • In the generic ML scorer, teams can simply define the features using Lucene expressions in the query without writing module code.

Testing

Before custom scorers are deployed, it is necessary to test them to ensure the reliability of our services. We do this by ensuring scorers have high coverage of unit tests. We also have integration tests that launch a full cluster with the Inference Plugin and custom scorers, and run a series of test queries to ensure the system is healthy.

The integration test environment is capable of benchmarking and profiling to discover potential bottlenecks caused by changes in different components. This allows us to troubleshoot issues, especially those related to latencies, without using production nodes.

Deploying

Once all automated tests pass, we deploy the models or scorer changes into our dev environment and monitor latencies and errors. The way we deploy models differs from scorer deployment.

When there’s no scorer change and all we need to do is update a running model, we use Jenkins to update the model for a cluster. The Jenkins job does this by updating the model in one node first using an endpoint in Nrtsearch. If it succeeds and has no errors, it will proceed with the rest of the replica nodes. Using this method, we don’t need to restart the Nrtsearch cluster. The following diagram illustrates the deployment flow for models into Nrtsearch:

When a custom scorer is ready to be deployed to production, we first deploy it to a Canary node in Nrtsearch using our internal config manager. We monitor the latencies and errors using our various monitoring tools and dashboards. Once everything is confirmed to be good, we deploy it to Nrtsearch, which causes a rolling restart of the cluster. As a rule of thumb, we try to start a smaller number of nodes at a time to decrease the potential impact of any bugs that weren’t discovered in earlier stages.

Monitoring

We have a variety of automated tools, alerts, and dashboards to monitor Nrtsearch’s performance and health. These tools are great for ensuring Nrtsearch works smoothly all the time, but how do we know if a model is still efficient? We answered that question by exposing Prometheus metrics—a time-series monitoring system that scrapes latencies, error rates, and other performance data—to track the performance and behavior of ML models on Nrtsearch. Using Prometheus metrics we can create alerts to notify engineers when a model’s performance has degraded over time for various reasons. This monitoring allows us to quickly identify the root cause of issues and maintain consistently high model performance.

Future Work

While our ML-based ranking has been serving us well over the past few years, we are still working to add new features to meet more complex requirements for future enhancements. We will support GPU-based ranking to speed up the process of ranking documents using larger and more complex neural networks. We will be potentially working on supporting multiple inference engines on top of the current MLeap one.

Acknowledgements

Much of this work would not have been possible without help from various people and organizations at Yelp. Special thanks to Andrew Prudhomme for creating the architecture in both Nrtsearch and the Inference Plugin, and to the many other folks who contributed to the Nrtsearch project. We also thank Yunhui Zhang and his team for providing guidance on integrating Yelp’s internal model server with Nrtsearch.

Back to blog