Login | Register   
RSS Feed
Download our iPhone app
Browse DevX
Sign up for e-mail newsletters from DevX


Use Semantic Language Tools to Better Understand User Intentions : Page 2

Leverage the power of WordNet to create applications that can more meaningfully interpret English language input.




Full Text Search: The Key to Better Natural Language Queries for NoSQL in Node.js

Date: 1/31/2018 @ 2 p.m. ET

Searching for Synonyms
Enough background—let's see how you might put WordNet to use. Let's assume that our storefront has an inventory that includes "pants." If a user searches for synonymous words, such as "trousers," we want the "pants" item from inventory to show up in the results. This could be expressed in a unit test as follows:

// src/test/java/com/devx/storefront/StorefrontTest.java ... @Test public void testSearch_ExactNotFound_SynonymnFound() { store.addItem(new Item("pants")); Set<Item> matchingItems = store.search("trouser"); assertEquals(1, matchingItems.size()); assertTrue(matchingItems.contains(new Item("pants"))); } ...

The preceding test verifies that the storefront includes synonymous words in the results presented back to the user.

There are several Java APIs for WordNet that you could use to implement this unit test; this article will explore two: Java WordNet Library (JWNL) and Jawbone. Here's a JWNL example in which the storefront delegates the synonym search to a JWNL dictionary class:

// src/main/java/com/devx/storefront/Storefront.java ... public Set<Item> search(String name) { Set<Item> matchingItems = new HashSet<Item>(); ... // Then add other similiar words in inventory Set<String> candidateWords = new HashSet<String>(); ... candidateWords.addAll(jwnlDictionary.lookupSynonyms(name)); ... return matchWordsToInventory(candidateWords); } ...

The interesting logic regarding WordNet really occurs in the JwnlDictionary class:

src/main/java/com/devx/storefront/JwnlDictionary.java ... public Set<String> lookupSynonyms(String lexicalForm) { Set<String> synonyms = new HashSet<String>(); IndexWord indexWord = dictionary.getIndexWord( POS.NOUN, lexicalForm); if (indexWord == null) return synonyms; Synset[] synSets = indexWord.getSenses(); for (Synset synset : synSets) { Word[] words = synset.getWords(); for (Word word : words) { synonyms.add(word.getLemma()); } } return synonyms; } ...

Figure 3. Searching by synonyms: As shown in this figure, results for searches for "trousers" include "pants" as a synonym of "trousers."
The preceding code snippet extracts all possible lexical representations of all possible meanings of the word passed in as a method parameter. These various lexical forms are matched to items in inventory by the Storefront object. To step back momentarily, the user's search generates a set of related word forms that are compared to the store's inventory, and any matching results are returned. Thus a search for "trousers" returns a match for the inventory item "pants" as shown in Figure 3.

Searching for Hypernyms and Hyponyms
The inclusion of synonymous words in the search results is nice, but there are several other types of searches you could perform that would return other possibly relevant items in the search results. Two other possibilities are hypernyms and hyponyms. Hypernyms include more general terms while hyponyms include more specific terms. For example, "garment" is a more general term than "pants" and thus is a hypernym to "pants." On the other hand, "jeans" is a more specific term and thus is a hyponym. The two unit tests below express this behavior:

// src/test/java/com/devx/storefront/StorefrontTest.java ... @Test public void testSearch_ExactNotFound_HyponymFound() { store.addItem(new Item("pants")); Set<Item> matchingItems = store.search("jeans"); assertEquals(1, matchingItems.size()); assertTrue(matchingItems.contains(new Item("pants"))); } @Test public void testSearch_ExactNotFound_HyponymFound_Levis() { store.addItem(new Item("jeans")); Set<Item> matchingItems = store.search("levis"); assertEquals(1, matchingItems.size()); assertTrue(matchingItems.contains(new Item("jeans"))); } ...

Here are the corresponding JWNL implementations:

// src/main/java/com/devx/storefront/JwnlDictionary.java ... public Set<String> lookupHypernyms(String lexicalForm) { return lookupWordsFollowingPointer( lexicalForm, PointerType.HYPERNYM); } public Set<String> lookupHyponyms(String lexicalForm) { return lookupWordsFollowingPointer( lexicalForm, PointerType.HYPONYM); } private Set<String> lookupWordsFollowingPointer( String lexicalForm, PointerType pointerType) { Set<String> hypernyms = new HashSet<String>(); IndexWord indexWord = dictionary.getIndexWord( POS.NOUN, lexicalForm); if (indexWord == null) return hypernyms; Synset[] synSets = indexWord.getSenses(); for (Synset synset : synSets) { if (hasPointer(synset, pointerType)) { PointerTarget[] targets = synset.getTargets(pointerType); for (PointerTarget target : targets) { Word[] words = ((Synset) target).getWords(); for (Word word : words) { hypernyms.add(word.getLemma()); } } } } return hypernyms; } ...

This code is very similar to the code for synonyms, with an additional operation to navigate either a hypernyms or hyponyms pointer from inclusion in search results. Figure 4 shows an example of a hypernym search, while Figure 5 shows hyponym search results.

Figure 4. Searching By Hypernym: Because the word "garment" is a hypernym encompassing various types of clothing, this search returns all more specific forms of clothing contained in inventory.
Figure 5. Searching by Hyponym: Because "levis" is a hyponym, or a specific type, of "jeans," this search result includes "jeans" in the results list.

Comment and Contribute






(Maximum characters: 1200). You have 1200 characters left.



Thanks for your registration, follow us on our social networks to keep up-to-date