How NLP & NLU Work For Semantic Search

NER will always map an entity to a type, from as generic as “place” or “person,” to as specific as your own facets. While NLP is all about processing text and natural language, NLU is about understanding that text. Spell check can be used to craft a better query or provide feedback to the searcher, but it is often unnecessary and should never stand alone. This spell check software can use the context around a word to identify whether it is likely to be misspelled and its most likely correction. Just as with lemmatization and stemming, whether you normalize plurals is dependent on your goals. This is because stemming attempts to compare related words and break down words into their smallest possible parts, even if that part is not a word itself.

Other NLP And NLU tasks

Computers seem advanced because they can do a lot of actions in a short period of time. You could imagine using translation to search multi-language corpuses, but it rarely happens in practice, and is just as rarely needed. Intent detection maps a request to a specific, pre-defined intent. When ingesting documents, NER can use the text to tag those documents automatically. Increasingly, “typos” can also result from poor speech-to-text understanding.

How NLP & NLU Work For Semantic Search

Even trickier is that there are rules, and then there is how people actually write. For example, capitalizing the first words of sentences helps us quickly see where sentences begin. NLU, on the other hand, aims to “understand” what a block of natural language is communicating.

How LLMs Interpret Content: How To Structure Information For AI Search

This detail is relevant because if a search engine is only looking at the query for typos, it is missing half of the information. If you decide not to include lemmatization or stemming in your search engine, there is still one normalization technique that you should consider. For example, to require a user to type a query in exactly the same format as the matching words in a record is unfair and unproductive. With these two technologies, searchers can find what they want without having to type their query exactly as it’s found on a page or in a product. A user searching for “how to make returns” might trigger the “help” intent, while “red shoes” might trigger the “product” intent.

• Usually, normalizing plurals is the right choice, and you can remove normalization pairs from your dictionary when you find them causing problems.
• This isn’t so different from what you see when you search for the weather on Google.
• If you decide not to include lemmatization or stemming in your search engine, there is still one normalization technique that you should consider.
• On the other hand, if you want an output that will always be a recognizable word, you want lemmatization.

You can then filter out all tokens with a distance that is too high. One area, however, where you will almost always want to introduce increased recall is when handling typos. Usually, normalizing plurals is the right choice, and you can remove normalization pairs from your dictionary when you find them causing problems.

Summaries can be used to match documents to queries, or to provide a better display of the search results. For searches with few results, you can use the entities to include related products. This is especially true when the documents are made of user-generated content.

Plurals

Lemmatization will generally not break down words as much as stemming, nor will as many different word forms be considered the same after the operation. Conversely, a search engine could have 100% recall by only returning documents that it knows to be a perfect fit, but sit will likely miss some good results. Much like with the use of NER for document tagging, automatic summarization can enrich documents.

It isn’t a question of applying all normalization techniques but deciding which ones provide the best balance of precision and recall. As we go through different normalization steps, we’ll see that there is no approach that everyone follows. Each normalization step generally increases recall and decreases precision. These kinds of processing can include tasks like normalization, spelling correction, or stemming, each of which we’ll look at in more detail. In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses — and careers. Few searchers are going to an online clothing store and asking questions to a search bar.

The best typo tolerance should work across both query and document, which is why edit distance generally works best for retrieving and ranking results. The simplest way to handle these typos, misspellings, and variations, is to avoid trying to correct them at all. We have all encountered typo tolerance and spell check within search, but it’s useful to think about why it’s present.