Browse DevX
Sign up for e-mail newsletters from DevX


Go Beyond Keywords! Perform a Visual Image Search  : Page 4

Searching graphics files via keywords can be tedious. Learn how to use image-matching technology to find images by matching shapes, patterns, colors, and textures.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

Step 3: Segmentation Map & User Selection
When the toolkit processes a new image, it automatically segments the image into regions, which correspond roughly to objects or parts of objects in the image. The set of regions is called a segmentation mask or map. The toolkit stores the segmentation mask in the MediaObject. Using the toolkit, you can create interfaces that let the user select specific objects in the image to help create a more focused search—a partial image search (see Figure 2).

Figure 2
As part of the analysis process, each MediaObject automatically includes a copy of the image that was analyzed and a segmentation mask for that image. The segmentation mask is a two-dimensional (2D) byte array the same size as the image that maps each pixel in the image into a particular region. Therefore, for example, if you analyzed an image at a maximum resolution of 128, the analysis routines will store an image in the MediaObject that is 128x96 pixels in size. In addition, the MediaObject will also contain the segmentation mask, which, in this example, would be a 2D byte array (128x96). For an image with 3 possible unique regions, each byte would contain the value 0, 1, or 2 corresponding to the particular region the image's pixel was mapped to (see Figure 3).

The getProperty method of the MediaObject can return both the image and the segmentation mask. For example, mediaObject.getProperty("image") returns a byte array, which is actually a raw jpg byte stream. Calling mediaObject.getProperty("segmentationMask") returns a 2D byte array—the segmentation mask.

To perform an object-specific search you:
  1. let users select search regions by clicking on them
  2. extract the Visual Signatures for the regions from the MediaObject
  3. construct a "dummy" MediaObject containing the Signatures extracted in step 2
  4. pass the "dummy" MediaObject to search as the source
Here's an example using a previously analyzed MediaObject. This will be your query image.

String analyzedImagePath = new String("/myMediaCollection/edf/myImage.edf"); MediaObject myMediaObject = (MediaObject) Eve.newMediaObject(); myMediaObject.loadFrom (analyzedImagePath);

Then create an empty "dummy" MediaObject that will be filled with just the object regions that the user selects.

MediaObject tempMediaObject = Eve.newMediaObject();

To create the interface so the user can select particular objects, retrieve the segmentation mask from MediaObject.myMediaObject. By default, this will return a new image with three object regions separated by the colors red, green and blue. Alternatively, you can call getSegmentationMaskImageIcon() to retrieve an ImageIcon for use in a Swing application.

Authors' Note: The number of object regions can be increased or decreased by editing the eVe.maxRegions of the eVe.properties file or changing the value passed to the analyze method when the engine analyzes the MediaObject.

Image segmentationImage = Eve.newImageManager().getSegmentationMaskImage (myMediaObject);

Now that you have an image divided into three separate regions you need to specify which to search. This can be accomplished in a simple GUI by capturing the (x,y) coordinates of a mouse click and bouncing the results against our segmentation mask image.

For simplicity in this example, simply set an int array to meet this purpose. To search for red only set regions[] = {0}, for blue set regions[] = {1} and for green set regions = {2}. To search for multiple regions i.e. red and blue set regions = {0,1} to search for blue and green set regions = {1,2}. Note: searching for all three regions = {0,1,2} would be the same as searching the entire MediaObject, thus defeating the purpose of this section. The example code below searches only the red and green regions within the images.

int [] regions ={0,2};

Create a vector and add all of the MediaObject's analyzed information to it. *Note: After each MediaObject is analyzed, data is stored in a Vector array of length 4, where Vector[0] holds information about the color, Vector[1] contains information about the texture, Vector[2] contains Shape attributes and Vector[3] holds information about the regions of the MediaObject. What you are doing is essentially building an analyzed MediaObject manually.

Vector[] myMediaObjectVector = {myMediaObject.getIndex(Eve.COLOR), myMediaObject.getIndex(Eve.TEXTURE), myMediaObject.getIndex(Eve.SHAPE), myMediaObject.getIndex(Eve.REGION)};

Now create an array to setup a loop that adds the information about each attribute to the empty "dummy" MediaObject.

int[] attributes = {Eve.COLOR, Eve.TEXTURE, Eve.SHAPE, Eve.REGION};

Iterate once for each attribute COLOR, TEXTURE, SHAPE, REGION.

for (int i = 0; i < attributes.length; i++) { // Create a new vector that will temporarily // hold the MediaObject attribute. Vector tempAttributeVector = new Vector(); // Each iteration of this loop adds the // current attribute data only // for the region(s) we specified 0, 1, or 2. for (int j = 0; j < myMediaObjectVector[i].size(); j++) { for (int n = 0; n < regions.length; n++) { if (regions[n] == j){ tempAttributeVector.addElement (myMediaObjectVector[i].elementAt(j)); } } } // Add the attribute of the regions to // the empty MediaObject. tempMediaObject.setIndex(attributes[i], tempAttributeVector); }

Now you have a MediaObject with only the selected region(s). At this point, this MediaObject can be used as the query MediaObject for a search in a MediaCollection.

myMediaObject = tempMediaObject;

Thanks for your registration, follow us on our social networks to keep up-to-date