Browse DevX
Sign up for e-mail newsletters from DevX


Ink Recognition and Ink Analysis : Page 4

Digital Ink is only a collection of lines rendered on the screen, but with Ink recognition and analysis, you can turn it into meaningful information such as text, drawings, gestures, commands, and even the relationship between two shapes. The Tablet PC SDK makes it surprisingly simple to detect all these types of information in Digital Ink.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

Ink Analysis
Everything you have seen in this article so far is available in the Tablet PC SDK version 1.7 and later. But it doesn't stop there. Microsoft is busy working on additional capabilities for developers who attempt to make programmatic sense of the mess of strokes and package points that is Digital Ink. The Community Technology Preview (CTP) for the Tablet PC SDK released in September 2005 features some very intriguing capabilities that all fall under the umbrella of Ink analysis. Because this is CTP technology, it is still in beta as I write this article. Nevertheless, it is functionality that I should introduce you to.

Ink analysis is a superset of the older recognizer context and Ink divider.
Ink analysis is a superset of everything I've discussed in this article so far, including various ways of Ink recognition and Ink division. However Ink analysis takes these concepts to the next level and provides a much smarter and more convenient approach to extracting meaning from Ink beyond strings. The Ink divider can find important logical units such as paragraphs, but with Ink analysis, you'll be able to distinguish more sophisticated concepts, such as annotations or bulleted lists. Ink analysis also provides new recognition options, such as recognition of graphical primitives (circles, lines, rectangles, triangles, and more).

To start out, let's look at the hierarchical representation of flow text. To retrieve a hierarchical representation of Ink, you first have to analyze a collection of strokes. To do so, you'll create an Analyzer object, assign it some strokes, and call the Analyze() method.

InkAnalyzer analyzer = new InkAnalyzer(this.collector.Ink,this); analyzer.AddStrokes(this.collector.Ink.Strokes); analyzer.Analyze();

Figure 6. Analyzing Ink: You can use Ink analysis to turn Ink into a hierarchy of writing regions, paragraphs, lines, words, and much more.
At this point, the analyzer contains detailed information about the Ink assigned to it. You can retrieve this information in a variety of ways. For instance, you can look for certain pieces of information you are interested in (such as all the paragraphs). Another alternative is to simply start out with the root of the Ink and drill into its drawing areas, paragraphs, lines, words, and much more. Listing 4 shows an implementation that drills through the entire tree and lists all the nodes and their types, as well as the text recognized for each node. Figure 6 shows the result.

There are a few differences between this approach and the Ink divider approach shown above. For one, Ink analysis provides a more consistent approach, with spatial analysis, context parsing, and handwriting recognition all provided by a single service. (It is also possible to pass cultural information to AddStrokes(), allowing for easy multi-lingual recognition.) Also, using Ink analysis is much easier than using the Ink divider, yet at the same time it is more powerful because it can detect more meaning in Ink. It can find things such as bulleted lists and it also does a much better job in finding words, rather than just segments.

Furthermore, Ink analysis provides services that aren't provided by the previous version of the SDK at all. Drawing analysis is one such example. The following code snippet finds all the drawings in a collection of strokes.

InkAnalyzer analyzer = new Microsoft.Ink.InkAnalyzer( this.collector.Ink,this);

Figure 7. You can recognize drawing primitives, such as circles and triangles.
analyzer.AddStrokes(this.collector.Ink.Strokes); analyzer.Analyze(); ContextNodeCollection drawings = analyzer.FindNodesOfType( ContextNodeType.InkDrawing); string drawingString = "You drew:\r\n\r\n"; foreach(InkDrawingNode drawing in drawings) { drawingString += drawing.GetShapeName() + "\r\n"; } MessageBox.Show(drawingString);
This finds drawings such as rectangles, circles, various types of triangles, trapezoids, and much more. Figure 7 shows an example.

Some lines are just drawings and others are used to put emphasis on words by underlining them. The difference between the two types of lines is significant.
Another great ink analysis feature is the ability to derive meaning from various strokes of Ink and their association with each other. For instance, a line under a word is generally interpreted by humans as an underlined word. The line communicates meaning by emphasizing a word. The line certainly should not be interpreted as a separate drawing.

Similarly, a circled word with an arrow pointing to additional text would be interpreted by a human reader as an annotation.
Figure 8. Detecting Annotations: In this example, a spatial relationship (annotation) is detected and color coded.
It also implies certain behavior when that Ink is manipulated. If the circled word moves to another line, for instance, the annotation should still link to the word and not to a blank space or to a different word that moved into the place of the old word. In other words, the annotation needs to be linked to the Ink (or other data) it annotates and move with it.

Listing 5 shows an example that iterates over the entire tree of nodes and looks at logical links to other nodes. These links are then color-coded to provide a visual indication. Figure 8 shows an annotation that is recognized as being linked to a word. Of course, finding annotations is only one use of this technology. Another idea combines spatial analysis with the ability to recognize shapes, allowing developers to build a diagram engine that can analyze and recognize things like flow charts.

Of course, these are but a few examples of what the new Ink analysis API can do. I encourage you to poke around in this exciting new SDK.

Being able to collect and store digital Ink on Tablet PCs is great, but the real power emerges when additional logic and smarts get applied to Ink information. Unrecognized Digital Ink is better than Ink on a piece of paper, but not much. The ability to turn Ink into text or other meaningful information is where the real advantage of Tablet PCs comes into play. And even if Ink is stored as Ink, recognition still is a tremendously important component, be it to allow you to search Ink, or for spatial analysis and the contextual information derived from that.

Ink recognition and analysis is surprisingly easy, and every developer should have a fundamental understanding of these concepts, especially as Ink moves into the core operating system and will be available on all versions of Windows Vista.

Markus Egger is president of EPS Software Corporation, located in Houston, Texas. He is also the founder of EPS Software Austria, located in Salzburg. He concentrates on consulting in COM-based, object-oriented development and Internet applications. He is an international author and speaker, and is co-publisher of Component Developer Magazine. He is also the author of "Advanced Object-Oriented Programming with Visual FoxPro," from Hentzenwerke Publishing. For the past several years, Markus has received the Microsoft MVP award. Several applications he has worked on (mostly as project manager) have received Microsoft Excellence Award nominations. He is the author of several tools, such as GenRepoX (public domain), the Fox Extension Classes, and Visual WebBuilder.
Thanks for your registration, follow us on our social networks to keep up-to-date