Browse DevX
Sign up for e-mail newsletters from DevX


Ink Recognition and Ink Analysis : Page 3

Digital Ink is only a collection of lines rendered on the screen, but with Ink recognition and analysis, you can turn it into meaningful information such as text, drawings, gestures, commands, and even the relationship between two shapes. The Tablet PC SDK makes it surprisingly simple to detect all these types of information in Digital Ink.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

Recognizing Gestures
All the examples I've shown so far have used the default text recognizer to turn handwriting stored as digital Ink into string data. A somewhat different aspect of recognition is the ability to recognize gestures. Using the same Ink collector object you've seen in all the examples so far, you can enable gesture recognition for individual gestures you're interested in and then apply your own logic to react to those gestures.

Support for gestures such as "scratch out" can significantly improve the user experience.
For example, consider the "scratch out" gesture which allows the user to scratch out (delete) Ink that was previously written. As a first step in this scenario, you want to enable gesture recognition on the Ink collector and to specify which gestures you are interested in.

this.collector = new InkCollector(this.panel1); this.collector.CollectionMode = CollectionMode.InkAndGesture; this.collector.SetGestureStatus( ApplicationGesture.Scratchout, true); this.collector.Enabled = true;

Note that it is not enough to just enable gesture recognition on the collector. You also need to specify the individual gestures you are interested in, otherwise, the system would spend a lot of processing time finding gestures you do not care about and then alert you about an overwhelming number of gestures. For this example, you will only be interested in the scratch out gesture, but you could enable gesture recognition for as many different gestures as you desire.

Based on this configuration, the collector object fires events whenever it thinks it has discovered a gesture. You can hook that event in the following fashion:

this.collector.Gesture += new InkCollectorGestureEventHandler( this.collector_Gesture);

Whenever you receive a gesture event, you need to first decide whether you agree that a scratch out gesture occurred, and subsequently, decide how to react to that gesture.

To confirm that a scratch out gesture occurred, you need to verify that the ID of the recognized gesture is that of a scratch out gesture. In the current scenario no other gestures should occur, but in most scenarios, you will use more than one gesture. Furthermore, you'll only consider the scratch out gesture to really have occurred if the recognizer is reasonably certain about its findings. In other words: you'll ignore gestures with anything lower than intermediate recognition confidence.

But there is more. A scratch out is really only a scratch out if it scratched something out. So you need to check whether there is any Ink under the area that has been scratched out. To be somewhat tolerant, you need to find all previously existing strokes of Ink that have at least a partial overlap with the performed scratch out gesture. I generally find a 60% overlap to work very well for this sort of operation. The way to implement this check is to first get the bounding box of the scratch out gesture (the rectangular area within which the scratch out gesture occurred) and then use that information to query all the previously existing strokes that fall within that area (give or take 40%). This is relatively complex functionality, but luckily the Tablet PC SDK provides it all for you.

Once you're certain that a scratch out did in fact occur, you delete the affected existing strokes. Note that the scratch out gesture also created a set of Ink strokes. Because you've determined that they were used to perform a scratch out gesture, you do not want the system to use that Ink for anything else (such as handwriting recognition). You'll therefore set the Cancel property on the event arguments object to true and also remove all of the new strokes from the Ink object.

Voila! Your scratch out support is complete. Listing 2 shows the implementation of the Gesture event handler.

The Ink Divider
When the amount of Ink grows beyond a few words or lines, additional semantics, such as paragraphs and lines, become increasingly important.
Whenever you need to analyze small amounts of text, the techniques introduced above are usually sufficient. However, when the amount of text grows to multiple lines and paragraphs, you may need to apply further semantics to the Ink before you perform any recognition. For instance, you may need to first identify paragraphs and then perform recognition on an individual paragraph. To do so, you need to be able to find all the strokes that make up a paragraph.

The same semantic identification requirements may arise for needs other than handwriting recognition. For instance, the user may resize your Ink area, forcing you to either stretch or shrink the Ink (this possibility arises whenever only small amounts of Ink need to be resized) or even re-flow the Ink. In that case, you need to implement word wrap for Ink, which is completely dependent on your ability to find strokes that make up individual words within a large collection of strokes.

Microsoft created the Ink Divider object for exactly these purposes. It can take a collection of strokes and apply logical divisions. Once a division result has been achieved, the developer can ask to get specific sets of information based on various criteria. Common scenarios are paragraph identification, line identification, and word identification. "Words" are actually called "segments" because a clear separation of words is not always easy. Is "InkDivider" one word or two words? With regular text, it is one word if there isn't a space in it, but with Ink, boundaries are not as clear cut. For most purposes, you can equate "segments" to "words."

Figure 5. Dividing Ink: You can divide Ink by paragraphs (blue), lines (green), and words (yellow), using an Ink divider.
Listing 3 shows an example that accepts all strokes in an Ink object for spatial analysis and division. The sample next retrieves three division results to find all paragraphs, lines, and segments, and then draws bounding rectangles around all the strokes that make up these sub-sections of the Ink. You can see the result in Figure 5. Note that there is much more you can do with these division results than retrieving the bounding rectangle. The results provide quick access to the top-alternate recognition result for each section. The result also provides access to the collection of strokes that make up each segment, allowing the developer to perform all operations that can be performed on other stroke collections, such as recognition, manipulation, or transformation.

It also needs to be mentioned that Ink Dividers have two small but (for some) significant shortcomings: Ink Dividers cannot retrieve recognition alternates the same way regular ink recognition does. Secondly, Ink division results cannot be stored externally after they have been retrieved. Both of these shortcomings are addressed in technology I will talk about next: Ink Analysis.

Thanks for your registration, follow us on our social networks to keep up-to-date