ablet PC and Mobile PC development is very popular today, and will only get more popular and important in the future. For Mobile PC developers, there are significant changes coming in the near future, some in the Windows XP timeframe, others in Windows Vista. This article provides an overview over what’s on the drawing board or already available in technology preview builds.
Since they were first released, Tablet PCs have gained significantly in popularity; as time goes on, they will move more and more into the mainstream. Currently, Tablet PCs run on a specialized version of Microsoft Windows XP. In the future, all Windows operating systems will support Tablet PC fundamentals, such as digital Ink.
Mobile PC development has been important for a long time. In fact, in May of 2005, notebook sales surpassed desktop PC sales in the US for the first time ever (for a full month), and they have not given up their lead since then (nor are they predicted to). These devices are used in an ever-growing variety of scenarios, and expectations from features, such as extended battery life, are on the rise.
The Tablet PC and Mobile PC scenarios are of strategic importance to Microsoft, and as a result, many exciting new things are scheduled to release with Windows Vista and with new versions of the Tablet PC SDK, currently available as a CTP (Community Technology Preview) and which also runs on Windows XP.
Software developers are usually most interested in what software can do, which SDKs are available, and how they can use these components in their own applications. But in Mobile PC and Tablet PC scenarios, the hardware is equally important, as it is part of what drives new scenarios and opportunities.
One of the obvious next steps is for devices to get both bigger and smaller in physical dimensions to supplement the currently available devices. On one end of the scale, this means that devices will have much larger displays with higher resolutions. On the other end of the scale, devices are getting smaller, moving toward the domain that is currently solely occupied by Pocket PCs.
A development that may be less obvious and less expected is the move toward touch displays. Conventional Tablet PCs use electromagnetic digitizers to sense the tip of the pen on the display. This has some advantages; for example, it lets hardware manufacturers mount the digitizer (the device that senses the pen) behind the LC display, providing a better and brighter display quality. But there are also disadvantages, such as that you always need a special pen that the EM digitizer can sense. If lost, replacing the pen can be expensive, and even worse, it’s not possible to interact with the device without the special pen. On the other hand, it is possible to double-tap an icon with a finger(nail) on Pocket PCs (which typically use touch-sensitive digitizers).
Touch screen Tablet PCs provide an additional option. The reason we have not seen them before is that there are some technical hurdles that need to be overcome to provide touch-sensitive displays. The problem is that people rest their hands and arms on the large Tablet PC displays while writing. This is not a problem on smaller devices such as Pocket PCs. On larger displays, it is important to differentiate between a tap of a fingernail intended to select an icon (or interact in another meaningful way), and an accidental tap or palm pressure that occurs accidentally while writing with the pen. The software has to do a lot of intense processing to filter unwanted “noise” from meaningful interaction data.
Touch devices have the disadvantage that hovering fingers can not be detected the same way a hovering electromagnetic pen can be. Hovering (or any mouse movement that is not a click) is therefore not natively available on touch devices. Microsoft provides software that makes up for that shortcoming.
For developers, these new devices provide new opportunities, and they also create some new challenges. Higher screen resolutions in Windows Vista will rely on resolution-independent display technologies handled through vector graphics and exciting new techniques and APIs such as the Windows Presentation Foundation (WPF) formerly known as Avalon.
Smaller form factors also require some developer consideration. User interfaces that may have worked well on 1024×768 displays may not work well on smaller devices, so developers may have to account for that in their applications. Using techniques such as flow layout (a layout option available in WPF) are helpful in these scenarios. Using this approach, user interfaces can re-flow to accommodate smaller displays without the need for the dreaded horizontal scrollbar, which is known to not work well at all.
New Tablet PC SDK Features for Windows XP
The Tablet PC SDK?and its ability to make very complex tasks such as handwriting recognition ridiculously easy?has always been nothing short of amazing. Microsoft keeps pushing the limits. A recent Community Technology Preview (CTP), made available in September 2005 targets the Windows XP-based Tablet PC operating system and provides some amazing new features, particularly in the area of Ink recognition.
|Figure 1. Analyzing Ink: A sample application uses Ink Analysis to identify paragraphs, bulleted lists, and even graphical elements such as rectangles.
The most exciting new feature is Ink Analysis, which is Ink recognition on steroids. Although simple Ink recognition is based on a collection of strokes, Ink Analysis takes a more sophisticated approach, applying both Ink recognition and Ink parsing at the same time. These two aspects are worthy of further explanation.
Ink recognition (or handwriting recognition) refers to the task of recognizing handwriting stored in individual strokes of Ink, and returning it as strings of text. Handwriting recognition is usually tied to a specific spoken language and culture.
Ink parsing refers to two further distinct tasks: Ink classification and layout analysis. Ink classification refers to the division of Ink and strokes into meaningful entities, such as words, lines, and paragraphs. It is also possible to designate certain areas of a larger recognition area, such as a title area in a notepad-style Inking area.
Layout analysis determines the position of Ink and how it relates spatially and semantically. For instance, layout analysis can determine that certain parts of a document are circled and connected to a paragraph, making it an annotation. This allows for advanced features, such as keeping annotations with a certain part of the document, even if the document is re-formatted.
|Figure 2. Analyzing Ink: The ink content after analysis and recognition of text, shapes, and bullet points.
Ink analysis also allows for shape recognition, a feature that was previously very difficult to implement. This feature can be used to recognize simple primitives, such as rectangles, circles, different types of triangles, trapezoids, and more (about a dozen, all in all). This ability is very useful as the basis for many non-text recognition models, such as the recognition of flow diagrams and the like. The provided default set of recognized shapes should handle many needs. Figure 1 and Figure 2 show examples of Ink interpreted through Ink analysis and exported to MS Word. And there is an opportunity for third parties wanting to extend that default set.
For some examples of what’s possible with Ink analysis, see the “Ink Recognition and Ink Analysis” article in this issue.
Another important new feature is the COM-based Real Time Stylus support. In previous releases of the SDK, the Real Time Stylus API was only available for managed developers. C++ developers, and other COM-based developers such as Visual Basic 6 developers, could only use interop to use the Real Time Stylus. This, of course, defeated one of the fundamental goals of the Real Time Stylus-performance-and has now changed with a native (COM-based) implementation of the Real Time Stylus API. (For more information on the Real Time Stylus, see the article “Get Control and Performance with the Real Time Stylus API.”)
New in Windows Vista
In Windows Vista, many of the components that were specific to the Tablet PC version of Windows XP have been migrated into the main operating system. Fundamental features, such as the ability to recognize or render Ink, are available on all versions of Windows Vista; in other words, you can treat any version of Windows Vista as a Tablet PC operating system. There are still a few components that are specific to actual Tablet PCs. Those fall into the category of accessory applications, such as the Windows Tablet PC Journal, or the TIP (Tablet PC Input Panel), which simply do not make sense unless you have a real Tablet PC.
Windows Vista also sports an area of major change when it comes to handwriting: Vista supports personalization. This means that you can train the handwriting recognizer to recognize your handwriting. In Windows XP, all handwriting was recognized the same way, no matter who used the device. Some people think that recognition in Windows XP gets more accurate over time, but this really is not the case; people tend to adjust their handwriting subconsciously. People get trained by the device. Vista supports the opposite, allowing users to run a wizard that can be used to train the recognizer to various degrees, ranging from simple training for characters or words the recognizer gets wrong regularly (see Figure 3), to in-depth training by providing vast handwriting samples (see Figure 4). Furthermore, users can send feedback to Microsoft about incorrect recognition results, similar to the way you can submit application error information to Microsoft today.
Another area that has been greatly improved in Vista is the TIP (Tablet PC Input Panel), which gains intelligence about where and at what size it appears on the screen. You can also resize it?an ability that was sorely lacking in Windows XP. The TIP can now appear either as a little icon as in Windows XP or as a docked panel that can be customized by the user. Figure 5 shows the new TIP in action.
|Figure 5. New TIP Panel: The new TIP (Tablet PC Input Panel) in Windows Vista fully supports the “glass” style.
Writing on the TIP has been improved as well. It now supports more sophisticated scratch-out gestures as well as end-of-pen erasing. The overall input area has been optimized for space, which is especially useful in languages that tend to have longer words and sentences.
The new TIP now also supports autocomplete in applications such as Internet Explorer (see Figure 6 and Figure 7). In XP versions of the TIP, when you write a word, such as “Microsoft,” it doesn’t get communicated to the control with the focus (in particular, to the address bar in Internet Explorer) until you hit the insert button. Internet Explorer’s autocomplete does not appear before the insert button is tapped. In Vista, all recognized text is sent to IE’s autocompletion engine immediately, showing the list of sites IE associates with the word right away, and making the whole user experience much more satisfying. This works not only in Internet Explorer, but also in other applications, such as in WinForms 2.0 autocomplete. There is an API developers can use for autocompletion, improving text entry in controls where Ink auto-complete is not supported natively.
The overall experience for pen users has also improved in Windows Vista through the addition of “flicks.” Flicks are simple gestures you can perform by quickly drawing a line in a certain direction. Draw a quick line to the right (flick the pen to the right) to perform a gesture that is interpreted as “navigate back”. There are different degrees of flick support users can choose, from relatively simple flicks in two or four directions, to more advanced flicks in eight directions. The action associated with each flick is user-configurable (see Figure 8).
|Figure 8. Gestures: “Flicks” are a quick and user-configurable way to use the pen for navigation and other features.
Programming Vista Tablets
There are changes for Tablet PC developers in Windows Vista. Some are small, such as the addition of APIs or new features in existing APIs. Others are larger and go along with the new UI development paradigm inherent to WinFX and WPF (Windows Presentation Foundation) development.
Let’s start out with API changes. The new TIP API affords much greater control over the TIP (Tablet PC Input Panel) and its features, compared to components such as PIP on Windows XP. There is a special Text Input API that can be used to control how text is recognized and inserted by the TIP. This API can be used to suggest alternate recognition results right inside the TIP. A related API is used to control the TIP’s interaction with text controls that support autocomplete (see above).
One feature has been retired in Windows Vista; in previous versions of the Tablet PC SDK, developers were able to “content tag” controls on existing applications, providing hints to the recognizer as to what kind of content is to be expected. For instance, textboxes used to enter e-mail addresses could be tagged with the “EMAIL” factoid, which drastically improves recognition results for e-mail addresses. This tagging information was stored in external manifests. It turned out that this approach was somewhat fragile and not entirely productive. Whenever developers reused the same control in various user controls or classes, they had to tag the same control in each instance where it was used. It would have been much more productive to programmatically indicate the content type of the control once. Also, if the external manifest were lost, corrupt, or inaccessible for some reason, none of the tagging information would be communicated to the recognizer. Windows Vista no longer supports this content-tagging approach; instead, this is now handled programmatically through the InputScope property on the UIElement base class. Context tagging is gone, but you can set the recognizer hints in XAML.
The Windows Presentation Foundation
With WinFX, Microsoft introduces a successor to the Win32 API. The public API surface of WinFX is completely managed (written in .NET) and includes technologies such as the Windows Presentation Foundation (WPF, formerly known as Avalon), Windows Communication Foundation (WCF, formerly known as Indigo), and the Windows Workflow Foundation (WWF), among others. Tablet PC-related functionality focuses almost exclusively on user interfaces. For this reason, WPF is most important for our purposes. WPF was originally conceived as a display technology specific to Windows Vista. However, based on user feedback, Microsoft has made the decision to make WinFX (including WPF) available on Windows XP as well. This includes all the Ink features discussed in this article, such as the InkCanvas control.
In the past, all versions of Windows (including Windows XP), created almost all graphical output using a technology known as GDI (Graphics Device Interface). Although this technology has served us well for a long time, it has its problems. GDI is completely pixel-based. Everything drawn on the screen is created by pixel-based drawing operations that take place on the PC’s main CPU. Unfortunately, pixel-based drawings do not scale well, which means that pixel-based output can not easily adjust to various display resolutions. Also, modern displays have a lot of pixels, which puts a significant burden on the CPU, while advanced graphics accelerator cards that are present in almost all computers these days often lie dormant.
Some of the downsides of GDI become apparent in applications that require high-performance graphics output, such as games or multi-media applications (videos). For this reason, these kinds of applications do not use GDI; they use DirectX, a high-performance interface capable of accessing graphics hardware directly, including the utilization of graphics accelerators known as GPUs (Graphics Processing Units), which are secondary processors optimized towards processing graphics.
The problem with using both approaches is that they do not mix very well. The issue is most easily demonstrated with videos on Web pages. Whenever a Web page with a video is viewed in a browser and the browser window is moved, the video lags behind, although the rest of the window moves very smoothly. This is a result of the combination of two rather different technologies. For Tablet PC developers, the effects also become obvious in the support for Inking. It is possible to collect and render Ink in GDI and it is also possible to collect and render Ink in DirectX. However, it is very difficult to Ink annotate a video and then write beyond the edge of the video and continue the annotation on a GDI-rendered area of the screen.
WPF takes a completely different approach. From the ground up, it was built to be a unified, high-performance graphics API. All output in WPF is based on vector graphics instead of pixels. This has the advantage of resolution independence. All graphical output can be resized or transformed in other ways at a very high level of quality. WPF also utilizes graphics hardware if it is present, making all the CPU power available for other processing.
WPF doesn’t draw screen output in the same way GDI or even DirectX did; instead, it “composes” output. Typical WPF user interfaces are composed of windows, buttons, textboxes, lines, and other WPF controls, such as videos. These controls are themselves composed of primitives defined as vectors (such as lines). The composition engine allows for great flexibility. For instance, a button control can itself be composed of a text label and a video. The same video can be used to compose the contents of a drop-down list, a menu, or any other WPF control.
What is particularly important for Tablet PC developers is that Ink is a native component of WPF. Therefore, every WPF application can support Ink natively, without the need for any additional components. Of course, because Ink is based on vectors, it is a perfect match for WPF’s composition engine and can be drawn very quickly at a high level of quality.
Before you look at your first WPF Ink example, I should discuss one more technology introduced by WinFX: XAML.
XAML is a declarative language that supplements the conventional programmatic languages. In programmatic languages, you instantiate objects, set properties, and call methods in a language such as C# or Visual Basic .NET. In a declarative environment, you use some sort of markup language (XML in this case) to specify the desired result. For instance, the following could be a programmatic approach for creating WinForms UI (as it is found in the designer-generated code region):
Button btn1 = new Button(); btn1.Top = 100; Form.Controls.Add(btn1);
In XAML, the same idea could be expressed as follows:
In many cases, the declarative approach is simpler and more efficient, because you only worry about the “what” and not the “how.” In the programmatic example, the button object had to be instantiated and the developer had to know whether or not the object received constructor parameters. The developer also had to know to add the button to the form’s Controls collection for the form to become functional. In the declarative approach, all of this is left up to the XAML compiler and the developer is concerned only with the desired outcome.
XAML is a great addition to the developer’s toolbox, with advantages so compelling that practically all WPF examples are based on it. However, it is important to point out that XAML is entirely optional in WinFX, and developers are free to use the more conventional programmatic approach if they so desire.
As a simple XAML example, a window containing a few controls can be defined in the following fashion:
The result of this is shown in Figure 9. This code may appear a bit confusing for people unfamiliar with XAML or WPF, but it is actually pretty straightforward. The root tag defines a window as well as some namespaces used by XAML. The window contains a canvas element/control, which means that you want to create a drawing area in your window that is 500×500 pixels?in “logical” pixels (1/96th of an inch)?that you can place other controls on (or in). The nature of this control is such that you can position other controls (rectangles in this case, but it could be anything) within it, providing absolute positions and dimensions.
At this point, you may be wondering why you needed a Canvas, rather than just positioning other controls on the window. The reason is that WPF supports many different ways of positioning controls. The Canvas that contains controls with absolute positions and dimensions is only one example (this is also known as grid layout). Another example is flow layout. Within a flow layout control, all other elements are positioned left-to-right and top-to-bottom, similar to Web pages.
What does it take to add Ink functionality to this example? Not a lot! All you need to do is add an InkCanvas object to your composition (or replace the regular Canvas with an InkCanvas). The following example shows a modified version that supports Ink:
? Figure 10. Modified XAML: A modified version of the simple XAML window.
The result can be seen in Figure 10.
It is important to note that although it is possible to write XAML by hand, most people will be using visual designers, such as the recently announced Sparkle designer that is part of Microsoft’s Expression series of products. Because Ink controls are part of the native WPF control set, interactive designers like Sparkle support Ink controls (such as the InkCanvas) natively.
Microsoft is pushing very hard on Tablet PC and Mobile PC technologies as the market for these technologies gains importance. For users, this means new and better devices and software, and new ways to interact with them. For developers, the push provides lots of opportunities. Because the new versions of the Tablet PC SDKs are more sophisticated, in many ways they are easier to use, and, in the case of WinFX and Windows Vista, they are more integrated in all versions of the platform.
To sum it all up: the future for Tablet PC and Mobile PC developers and users looks very bright.