Browse DevX
Sign up for e-mail newsletters from DevX


Ink Recognition and Ink Analysis : Page 2

Digital Ink is only a collection of lines rendered on the screen, but with Ink recognition and analysis, you can turn it into meaningful information such as text, drawings, gestures, commands, and even the relationship between two shapes. The Tablet PC SDK makes it surprisingly simple to detect all these types of information in Digital Ink.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

Recognition Beyond Text
The recognition result contains information beyond text, such as the baseline. The baseline is crucial in all handwriting recognition because it tells the recognizer where letters start and end. Typically, when you write on a piece of paper, you write on lines. These lines are the equivalent of baselines. The baseline determines the bottom of characters without under-lengths. (Characters without under-lengths are characters such as "a" or "b" but not the character "g".)

Here is some code that finds the baseline for a single line of recognized text and draws it on the window.

Figure 4. Finding Baselines: You can find the baseline easily with a little code.

if (status == RecognitionStatus.NoError) { Line bl = res.TopAlternate.Baseline; Graphics g = this.panel1.CreateGraphics(); Renderer rend = this.collector.Renderer; Point p1 = bl.BeginPoint; Point p2 = bl.EndPoint; rend.InkSpaceToPixel(g,ref p1); rend.InkSpaceToPixel(g,ref p2); g.DrawLine(Pens.Red,p1,p2); }

The tricky part of this example is that all coordinates retrieved from Ink are stored in Ink space. This is the fancy way of saying that Ink is stored in higher resolution than display information. As a result, the code has to translate Ink coordinates to screen coordinates. This is pretty straightforward in this scenario because the Ink overlay used to collect the Ink happens to have a Renderer object, which provides translation functionality you can use. (Note: For detailed information on how to translate Ink coordinates to screen coordinates, see the "Real Time Stylus" article.) Figure 4 shows a detected baseline in a single line of Ink.

Author's Note: The preceding approach for detecting baselines works only if the Ink contains a single line of text. This is never guaranteed in Ink. If there is more than one line of Ink (or if the recognizer thinks there is more than one line of Ink), you need to use the LineAlternates collection to retrieve all the available baselines.

Having access to a recognizer context not only allows you to retrieve a significant amount of information about the recognition result, but it can also be used to improve recognition quality. The simplest way to do so is by providing a factoid. For instance, you can set the context's factoid to "EMAIL", indicating to the recognizer that you expect to find an e-mail address. This improves recognition results because the recognizer can apply rules such as "there shouldn't be any spaces in an e-mail address" or "the weird character is probably an @". This code sample specifies such a factoid.

RecognizerContext context = new RecognizerContext(); context.Factoid = "EMAIL"; context.Strokes = this.collector.Ink.Strokes; RecognitionStatus status = RecognitionStatus.NoError; RecognitionResult res = context.Recognize(out status); if (status == RecognitionStatus.NoError) { MessageBox.Show(res.TopAlternate.ToString()); } context.Dispose();

The Tablet PC supports such factoids that you can set and combine at will (see the SDK documentation for a complete list.) For example, you can provide a factoid indicating that the recognition result should be either an e-mail address or a URL.

context.Factoid = "EMAIL|WEB";

There's also a Factoid class featuring static members that you can use instead of having to remember the actual factoid strings assigned to factoids. In other words, you could also set the "EMAIL" factoid for the preceding example with this code:

context.Factoid = Factoid.Email;

Note that factoids are always strings and are not limited to those defined in the Factoid class, nor is there any guarantee that all the Factoid class settings are supported by each language or culture.

You can take a similar approach by providing a word list suggesting a list of potential recognition results. This is very useful whenever you want to use Ink recognition to recognize a set of terms. For instance, an insurance application can be used to annotate a photo of a car. Using a word list, you can optimize the recognizer to find key terms such as "hail damage." Here's how it works.

WordList lst = new WordList(); lst.Add("hail"); lst.Add("damage"); lst.Add("car"); lst.Add("crash"); context.WordList = lst; context.Factoid = Factoid.WordList;

If you are following along with these examples, you may have noticed that Ink recognition is pretty resource-intensive, and whenever you ask the recognizer to recognize large amounts of text, things can get pretty slow. Whenever you run into this particular problem, you may want to think about asynchronous Ink recognition.

In asynchronous scenarios, the recognizer context is created and stored away for later use. It is also assigned a stroke collection right away. Everything else is handled through an event model. Recognition happens on a secondary thread, but all events are funneled back to the UI thread ("your" thread) and therefore developers do not have to worry about threading issues. There are two events of importance: Whenever a new stroke is added to the Ink collector, this stroke also has to be added to the recognizer context and subsequently, background recognition has to be triggered, which will cause a Recognition event on the recognizer context object. Here is the code that defines the context and wires up all the required events.

private RecognizerContext myContext; private Strokes myStrokes; private void button7_Click(object sender, System.EventArgs e) { this.myContext = new RecognizerContext(); this.myStrokes = this.collector.Ink.Strokes; this.myContext.Strokes = this.myStrokes; this.collector.Stroke += new InkCollectorStrokeEventHandler( collector_Stroke); this.myContext.Recognition += new RecognizerContextRecognitionEventHandler( context_Recognition); }

Here is the code that reacts whenever a new stroke is received.

private void collector_Stroke(object sender, InkCollectorStrokeEventArgs e) { this.myStrokes.Add(e.Stroke); this.myContext.BackgroundRecognize(); }

This code subsequently triggers the following code (length of the delay depends on the amount of Ink, but will likely be very short), which receives the recognition result and displays it in the form's title bar.

private void context_Recognition(object sender, RecognizerContextRecognitionEventArgs e) { this.Text = e.Text; }

Note that this event passes along all the recognized text each time. This is important, because previously recognized text can change as more Ink becomes available. For instance, the recognizer may think the user wrote an "I," but after a few more strokes, that character may look more like an "H." This also applies for words and sentences. A recognition result of "1am" may later turn into "I am here" after the last word has been added, as that makes a lot more sense than "1am here."

Thanks for your registration, follow us on our social networks to keep up-to-date