Building Search Interface Using Apache Solr in .NET

Building Search Interface Using Apache Solr in .NET

A quick, granular and accurate search interface is of prime importance for most web applications today. Many have traditional search interfaces where the scope of search is restricted to specific fields; thus limiting the ability to get relevant search results. However, most commercial web sites will require an advanced search interface that will index the content of documents, build complex queries based on multiple criteria and fetch maximum results. In this paper, we introduce the Apache Solr Search Engine that will, most importantly, provide content search, explain how to construct queries involving multiple search criteria using Solr and integrate with the application to build a quicker, accurate and more refined search interface.

Apache Solr is an open source enterprise search server, based on the Lucene Java search library, with XML/HTTP and JSON APIs. It runs in a Java servlet container, such as Tomcat. Input to Solr is the document with optional metadata. We put documents in it (called as indexing) via XML, JSON, CSV or binary over HTTP. We query it via HTTP Get and receive XML, JSON, CSV or binary result. It is architected to deliver very fast search operations across wide variety of data.

Features

  • Advanced full-text search – Users can search for one or more words or phrases in the content of documents, specific fields, or combinations of one or more fields, thus providing results that match user’s interests.
  • Faceted Search – User can narrow down the search results further by applying filters on the fields (numeric, date fields, unique fields) if the user wishes to drill down. Thus providing categorized search.
  • Sort – User can prioritize the search results based on field count.
  • Pagination – User can display the search results in pages of fixed size.
  • Hit-Term Highlighting – Provides highlighting of the search keyword in the document.
  • It is optimized for high-volume web traffic.
  • It supports rich Document Parsing and Indexing (PDF, Word, HTML, etc.)
  • Admin UI – It has a very simple and user-friendly interface for designing and executing queries over the data.
  • Caching – It caches the results of filter queries, thus delivering faster search operations.

Architecture

Diagram

The above block diagram shows the sequence of actions for uploading documents to Solr and executing Search queries as per specific search criteria to get the relevant matches.

Building Search Interface for Your Web Application:

We assume that Solr is configured and running. User will be required to know the Solr endpoint.

For more information on installing, configuring and running Solr, go here.

We propose a generic search interface that can be implemented to search any application specific entity that is indexed by Solr. Search method accepts the SearchParameters and returns the SearchResult of generic type .

public interface ISearch    {        SearchResult Search (SearchParameters parameters);    } 

Let us see how the SearchParameters look like and how they are constructed.

public class SearchParameters {        public const int DefaultPageSize = 4;         public SearchParameters () {SearchFor = new Dictionary();Exclude = new Dictionary();SortBy = new List();FilterBy = new List();PageSize = DefaultPageSize;PageIndex = 1;        }         public string FreeSearch { get; set; }        public int PageIndex { get; set; }        public int PageSize { get; set; }        public IDictionary SearchFor { get; set; }        public IDictionary Exclude { get; set; }        public IList SortBy { get; set; }        public IList FilterBy{ get; set; }        }  }
  • SearchFor : We add the advanced full-text parameters to this dictionary in the following pattern:
Key – Name of field Value – Word/phrase
Title Azure, “cloud computing”
tags Azure, “cloud computing”
  • Exclude: These are parameters to be excluded in the advanced full-text Search added to a dictionary.
Key – Name of field Value – Word/phrase
Title business
tags business
  • SortBy: This is a List of SortQuery items. FieldName will map to the field on which the sorting needs to be done. Order will indicate the SortOrder (Ascending/Descending)
public class SortQuery    {        public string FieldName { get; set; }        public SortOrder order { get; set; }    }public enum SortOrder        {            Ascending,            Descending,        }
FieldName order Description
Like _integer Descending Will sort the search results in descending order of number of Likes
  • FilterBy: This is a list of FilterQuery items.
public class FilterQuery    {        public string FieldName {get; set;}        public string LowerLimit { get; set; }        public string UpperLimit { get; set;}        public string Value { get; set; }        public string DataType { get; set; }    }
  • FieldName = The field on which filtering needs to be done
  • Value = Value of the filter field
  • DataType = If filtering values are restricted to a particular range, this will indicate datatype of the filter field
  • LowerLimit = If filtering values are restricted to a particular range, this will indicate lowerlimit of the filter field
  • UpperLimit = If filtering values are restricted to a particular range, this will indicate upperlimit of the filter field

Table1

  • PageSize: This is used for pagination to specify number of search results per page.

  • PageIndex: This is used for pagination to specify the offset for query’s results set. It will instruct Solr to display results from this offset.

Implementing search interface:

SolrNet is a free Open Source API that can be integrated with your .NET web application to build queries programmatically and execute them over Solr.

Creating Solr search result entity

We first create a class with properties that will map to the fields returned in the Solr search result. To identify the fields we will fire the search query in Solr Admin UI or contact the administrator.

For example:
To get search results for keyword “twitter” we will enter the keyword in the “Query String” textbox of Solr Admin UI and hit the Search button.

A query such as http://endpoint/solr/select?q=twitter&fq=&start=0&rows=10 will appear in the browser which is the search query.

The response will be XML with nodes for each search result. Users can identify the fields returned from this response and create the properties accordingly.

SolrNet.Attributes namespace contains attributes to map fields from Solr search result to entity. These attributes can be used to augment existing application entity or create a parallel entity.

Example:

public class Product {        [SolrUniqueKey("company_id_text")]        public string CompanyId { get; set; }         [SolrField ("product_count_integer")]        public int ProductCount { get; set; }          [SolrField("title_text")]        public string Title { get; set; }           [SolrField("created_on_datetime")]        public DateTime CreatedOn { get; set; }         [SolrField("downloadable_boolean")]        public bool Downloadable { get; set; } }

A SolrUniqueKey uniquely identifies the document. In database terms, it is the primary key. So users should choose to map a SolrUniqueKey in case the field is unique for each document, or SolrField can be used. The property name must exactly map to the field name in the Solr output XML.

Initialization:

using SolrNet; public class Search Product: ISearch{ static ISolrReadOnlyOperations< Product > solr;static SolrConnection connection;         static Search Product ()        {                                  connection = new SolrConnection("solrendpoint");                      Startup.Init< Product >(connection);           Solr = ServiceLocator.Current.GetInstance>();        }}

The above code snippet will initialize the SolrConnection. We also instantiate ISolrReadOnlyOperations variable that we will use to build the Solr query. Here Product refers to the type of search result to be fetched.

Building Queries:

public SearchResult Search(SearchParameters parameters)        {            int? start = null;            int? rows = null;            if (parameters.PageIndex > 0)            {                start = (parameters.PageIndex - 1) * parameters.PageSize;                rows = parameters.PageSize;            }                       var matchingProducts = solr.Query(BuildQuery(parameters), new QueryOptions            {                FilterQueries = BuildFilterQueries(parameters),                Rows = rows,                Start = start,                OrderBy = GetSelectedSort(parameters),            });             return new SearchResult< Product>(matchingProducts)            {                TotalResults = matchingProducts.NumFound,            }}

The Query() method has the following prototype:

SolrQueryResults Query (ISolrQuery query, QueryOptions options); 
  • Query = the advanced full-text search query
  • Options = Filter, Sort, Pagination options.
  • Returns SolrQueryResults of type T. In our case T = Product.
  • In the above code,FilterQueries = the filter query to be executed on the search results obtained after applying the full-text search query.
  • OrderBy = the sort query to be executed on the search results obtained after filtering.
  • Rows = for pagination specifies the number to search results to be returned.
  • Start = for pagination specifies the offset in the Solr response from where the results will be fetched.

Build advanced full-text search query:

public ISolrQuery BuildQuery(SearchParameters parameters)        {            if (!string.IsNullOrEmpty(parameters.FreeSearch))                return new SolrQuery(parameters.FreeSearch);             AbstractSolrQuery searchquery = null;             List solrQuery = new List();            List solrNotQuery = new List();            foreach (var searchType in parameters.SearchFor)            {                solrQuery.Add(new SolrQuery(string.Format("{0}:{1}", searchType.Key,   searchType.Value)));            }             if (solrQuery.Count > 0)                searchquery = new SolrMultipleCriteriaQuery(solrQuery, SolrMultipleCriteriaQuery.Operator.OR);             foreach (var excludeType in parameters.Exclude)            {                solrNotQuery.Add(new SolrQuery(string.Format("{0}:{1}", excludeType.Key,  excludeType.Value)));            }             if (solrNotQuery.Count > 0)            {                searchquery = (searchquery ?? SolrQuery.All) - new SolrMultipleCriteriaQuery(solrNotQuery, SolrMultipleCriteriaQuery.Operator.OR);            }             return searchquery ?? SolrQuery.All;        }
  • new SolrQuery(“fieldname : value”) = This class will create a SolrQuery for a full-text search of the given word/phrases in the fieldname
  • new SolrMultipleCriteriaQuery(solrQuery, Operator (AND/OR etc.)) = This class will apply the respective boolean operator on a list of solr queries.
  • SolrQuery.All = This will return all the documents in Solr without applying any search query.

Build Filter Queries:

public ICollection BuildFilterQueries(SearchParameters parameters)        {            List filter = new List();             foreach (var filterBy in parameters.FilterBy)            {                if (!String.IsNullOrEmpty(filterBy.DataType) &&  filterBy.DataType.Equals(Constants.DATE_DATATYPE))                {                     DateTime upperlim = Convert.ToDateTime(filterBy.UpperLimit);                    DateTime lowerlim = Convert.ToDateTime(filterBy.LowerLimit);                    if (upperlim.Equals(lowerlim))                    {                        upperlim = upperlim.AddDays(1);                    }                    filter. Add(new SolrQueryByRange(filterBy.FieldName, lowerlim,                        upperlim));                }                else                {                    string[] filterValues;                         if (filterBy.Value.Contains(";"))                        {                            filterValues = filterBy.Value.Split(';');                            List filterForProduct = new List();                            foreach (string filterVal in filterValues)                            {                               filterForProduct.Add(new SolrQueryByField(filterBy.FieldName, filterVal) { Quoted = false });                            }                            filter.Add(new SolrMultipleCriteriaQuery(filterForProduct, SolrMultipleCriteriaQuery.Operator.OR));                        }                        else                        {                            filter.Add(new SolrQueryByField(filterBy.FieldName, filterBy.Value));                        }                    }                }                       return filter;        }

There are 2 types of filters:

  • SolrQueryByField:
new SolrQueryByField(filterBy.FieldName, filterVal) { Quoted = false })

This accepts the fieldName and filter value. Quoted = false indicates that character escaping is disabled.

  • SolrQueryByRange:
new SolrQueryByRange(filterBy.FieldName, lowerlim, upperlim) 

We use SolrQueryByRange when our filter values fall between some ranges. This needs the datatype of the field. DateTime in our case. The fieldname for filter, its lower limit and upper limit.

Build Sort Queries:

private ICollection GetSelectedSort(SearchParameters parameters)        {            List sortQueries = new List();            foreach (var sortBy in parameters.SortBy)            {                if (sortBy.order.Equals(SortQuery.SortOrder.Ascending))                    sortQueries.Add(new SortOrder(sortBy.FieldName, Order.ASC));                else                    sortQueries.Add(new SortOrder(sortBy.FieldName, Order.DESC));            }            return sortQueries;        }

SortOrder class will accept the fieldName to be sorted on along with the Sort Order (Ascending/Descending).

Relevance Score

Solr will by default order the search results based on the relevancy score that is calculated to determine how relevant a given Document is to a user’s query. The more times a query term appears in a document relative to the number of times the term appears in all the documents in the collection, the more relevant that document is to the query. Thus the priority of the search results will always be on track unless one explicitly gives a sort parameter in the search query.

Conclusion

This paper has provided a detailed description of the search options available with Apache Solr and will hopefully serve as a thorough guide in deciding on the search parameters, constructing queries and for building an Advanced Search interface.

References

About the Author

Amruta Morajkar has more than four years of IT experience with .NET, Windows Azure, C#, WCF, Entity Framework, XML, SQL Server, and Asp.Net MVC, etc. She has also worked on niche technologies such as SOA and Cloud Computing and is currently working with Infosys Limited, Pune, India.

devx-admin

devx-admin

Share the Post:
Apple Tech

Apple’s Search Engine Disruptor Brewing?

As the fourth quarter of 2023 kicks off, the technology sphere is abuzz with assorted news and advancements. Global stocks exhibit mixed results, whereas cryptocurrency

Revolutionary Job Market

AI is Reshaping the Tech Job Market

The tech industry is facing significant layoffs in 2023, with over 224,503 workers in the U.S losing their jobs. However, experts maintain that job security

Foreign Relations

US-China Trade War: Who’s Winning?

The August 2023 visit of Gina Raimondo, the U.S. Secretary of Commerce, to China demonstrated the progress being made in dialogue between the two nations.

Pandemic Recovery

Conquering Pandemic Supply Chain Struggles

The worldwide coronavirus pandemic has underscored supply chain challenges that resulted in billions of dollars in losses for automakers in 2021. Consequently, several firms are

Game Changer

How ChatGPT is Changing the Game

The AI-powered tool ChatGPT has taken the computing world by storm, receiving high praise from experts like Brex design lead, Pietro Schirano. Developed by OpenAI,

Apple Tech

Apple’s Search Engine Disruptor Brewing?

As the fourth quarter of 2023 kicks off, the technology sphere is abuzz with assorted news and advancements. Global stocks exhibit mixed results, whereas cryptocurrency tokens have seen a substantial

GlobalFoundries Titan

GlobalFoundries: Semiconductor Industry Titan

GlobalFoundries, a company that might not be a household name but has managed to make enormous strides in its relatively short 14-year history. As the third-largest semiconductor foundry in the

Revolutionary Job Market

AI is Reshaping the Tech Job Market

The tech industry is facing significant layoffs in 2023, with over 224,503 workers in the U.S losing their jobs. However, experts maintain that job security in the sector remains strong.

Foreign Relations

US-China Trade War: Who’s Winning?

The August 2023 visit of Gina Raimondo, the U.S. Secretary of Commerce, to China demonstrated the progress being made in dialogue between the two nations. However, the United States’ stance

Pandemic Recovery

Conquering Pandemic Supply Chain Struggles

The worldwide coronavirus pandemic has underscored supply chain challenges that resulted in billions of dollars in losses for automakers in 2021. Consequently, several firms are now contemplating constructing domestic manufacturing

Game Changer

How ChatGPT is Changing the Game

The AI-powered tool ChatGPT has taken the computing world by storm, receiving high praise from experts like Brex design lead, Pietro Schirano. Developed by OpenAI, ChatGPT is known for its

Future of Cybersecurity

Cybersecurity Battles: Lapsus$ Era Unfolds

In 2023, the cybersecurity field faces significant challenges due to the continuous transformation of threats and the increasing abilities of hackers. A prime example of this is the group of

Apple's AI Future

Inside Apple’s AI Expansion Plans

Rather than following the widespread pattern of job cuts in the tech sector, Apple’s CEO Tim Cook disclosed plans to increase the company’s UK workforce. The main area of focus

AI Finance

AI Stocks to Watch

As investor interest in artificial intelligence (AI) grows, many companies are highlighting their AI product plans. However, discovering AI stocks that already generate revenue from generative AI, such as OpenAI,

Web App Security

Web Application Supply Chain Security

Today’s web applications depend on a wide array of third-party components and open-source tools to function effectively. This reliance on external resources poses significant security risks, as malicious actors can

Thrilling Battle

Thrilling Battle: Germany Versus Huawei

The German interior ministry has put forward suggestions that would oblige telecommunications operators to decrease their reliance on equipment manufactured by Chinese firms Huawei and ZTE. This development comes after

iPhone 15 Unveiling

The iPhone 15’s Secrets and Surprises

As we dive into the most frequently asked questions and intriguing features, let us reiterate that the iPhone 15 brings substantial advancements in technology and design compared to its predecessors.

Chip Overcoming

iPhone 15 Pro Max: Overcoming Chip Setbacks

Apple recently faced a significant challenge in the development of a key component for its latest iPhone series, the iPhone 15 Pro Max, which was unveiled just a week ago.

Performance Camera

iPhone 15: Performance, Camera, Battery

Apple’s highly anticipated iPhone 15 has finally hit the market, sending ripples of excitement across the tech industry. For those considering upgrading to this new model, three essential features come

Battery Breakthrough

Electric Vehicle Battery Breakthrough

The prices of lithium-ion batteries have seen a considerable reduction, with the cost per kilowatt-hour dipping under $100 for the first occasion in two years, as reported by energy analytics

Economy Act Soars

Virginia’s Clean Economy Act Soars Ahead

Virginia has made significant strides towards achieving its short-term carbon-free objectives as outlined in the Clean Economy Act of 2020. Currently, about 44,000 megawatts (MW) of wind, solar, and energy

Renewable Storage Innovation

Innovative Energy Storage Solutions

The Department of Energy recently revealed a significant investment of $325 million in advanced battery technologies to store excess renewable energy produced by solar and wind sources. This funding will

Renesas Tech Revolution

Revolutionizing India’s Tech Sector with Renesas

Tushar Sharma, a semiconductor engineer at Renesas Electronics, met with Indian Prime Minister Narendra Modi to discuss the company’s support for India’s “Make in India” initiative. This initiative focuses on

Development Project

Thrilling East Windsor Mixed-Use Development

Real estate developer James Cormier, in collaboration with a partnership, has purchased 137 acres of land in Connecticut for $1.15 million with the intention of constructing residential and commercial buildings.

USA Companies

Top Software Development Companies in USA

Navigating the tech landscape to find the right partner is crucial yet challenging. This article offers a comparative glimpse into the top software development companies in the USA. Through a

Software Development

Top Software Development Companies

Looking for the best in software development? Our list of Top Software Development Companies is your gateway to finding the right tech partner. Dive in and explore the leaders in

India Web Development

Top Web Development Companies in India

In the digital race, the right web development partner is your winning edge. Dive into our curated list of top web development companies in India, and kickstart your journey to

USA Web Development

Top Web Development Companies in USA

Looking for the best web development companies in the USA? We’ve got you covered! Check out our top 10 picks to find the right partner for your online project. Your

Clean Energy Adoption

Inside Michigan’s Clean Energy Revolution

Democratic state legislators in Michigan continue to discuss and debate clean energy legislation in the hopes of establishing a comprehensive clean energy strategy for the state. A Senate committee meeting

Chips Act Revolution

European Chips Act: What is it?

In response to the intensifying worldwide technology competition, Europe has unveiled the long-awaited European Chips Act. This daring legislative proposal aims to fortify Europe’s semiconductor supply chain and enhance its