he primary pointing, selecting, dragging, and input device for Tablet PCs is the pen. All Tablet PC computers have a digitizer beneath the screen that accepts pen input. Ink is a new data type designed for use on the Tablet PC that provides real-time visual feedback for pen-based input. This data type represents both pen strokes and holds metadata about the strokes. An ink-enabled application has the ability to recognize both ink and text. The Tablet PC Pen, Ink, and Reco (recognizer) APIs permit pen entry that is as responsive and natural as writing with traditional ink on paper. Because the existing architecture for Windows mouse messages does not permit this level of responsiveness in the event mechanism, pen movements and actions must be sent directly to applications in real time to create viable Tablet PC ink applications.
You can build applications for Tablet PC that support various levels of functionality for pen, ink, and ink stroke recognition, ranging from recognizing simple text input to creating and editing complex ink. Even if your application itself does not accept ink input directly, it can still receive ink input through the Microsoft Tablet PC Input Panel. See the sidebar Development Environment Required for Tablet PC Development for the hardware and software requirements you’ll need.
Available Ink Controls and Objects
Microsoft provides the ink controls and programmatic objects shown in Table 1 for use in applications, documents, or Web services.
| Control/Object | Usage |
| InkEdit | Capture, recognize, and display ink. A visible control |
| InkPicture | Intended for scenarios in which ink is not recognized as text. A visible control |
| Ink Collector | Collects points from an input device and puts them into an ink object. A transparent object |
| InkOverlay | Primary intended to display ink as ink. A transparent object. |
Table 1: Ink controls and programmatic objects available for use in applications, documents, or Web services.
The Tablet PC operating system includes InkEdit, and InkPicture controls as well as the InkCollector and InkOverlay objects. If you need more complex functionality, you can create additional controls inherited from the Managed Library or from the objects in the Microsoft Tablet PC Automation Object Model.
You can quickly generate a simple pen-based user interface with the basic Tablet PC ink controls. By positioning these controls on a form you define a fixed region for pen and ink input. You can embed ink controls in a Microsoft Visual Basic form, a Web page or even a Word document. The controls have properties and methods to retrieve and manipulate the ink and its properties. You can control their appearance programatically. The controls generate events for which you can write custom handlers.
| Author Note: Because the .NET versions of the InkPicture and InkEdit controls have not been marked safe for scripting, you should not use them in ASP.NET or HTML pages. If your HTML page requires an ink control, use the ActiveX version by embedding it as an |
| Automation Control | ClassID | ProdID |
| InkEdit | E5CA59F5-57C4-4DD8-9BD6-1DEEEDD27AF4 | InkEd.InkEdit.1 |
| InkPicture | 04A1E553-FE36-4FDE-865E-344194E69424 | msinkaut.InkPicture.1 |
Table 2: ActiveX ClassID and ProgID values required to embed Ink controls in HTML pages.
The InkCollector and InkOverlay objects are non-visual programmatic objects designed to capture ink input from the tablet device. The InkCollector object collects input from only one specific window. InkCollectors are inherently assigned to the ink controls however, you can attach an InkCollector to any window.
Suppose you have a standard GroupBox control (System.Windows.Forms.GroupBox) located on a windows form (see Figure 1). You can attach an InkCollector to it by passing its handle to the InkCollector’s New method.
'Create the InkCollector and attach it ' to the GroupBox myInkCollector = New InkCollector(gbInkArea.Handle) myInkCollector.Enabled = True
After creating, attaching, and enabling the InkCollector, you can employ any Tablet PC device (including the mouse) to lay down ink in real time on the associated window. The InkCollector stores collected ink strokes in an associated Ink object.
 | |
| Figure 1: The frmInkSave form contains a GroupBox associated with an InkCollector as well as an InkEdit Control. |
The InkOverlay object is a superset of the InkCollector that provides editing support. It’s a COM or .NET object that displays transparently, and is useful in situations where users are concerned with the size, shape, color, and position of the ink and not with recognition of the ink. It’s well suited for annotation and basic scribbling. The InkCollector stores collected data in an Ink object. Ink objects contain collections of other objects such as the Strokes collection, which in turn, contains one or more Stroke objects. A stroke is a set of data captured in a single pen-down, pen-move, and pen-up sequence. You can manipulate Stroke objects with the Ink object’s methods and properties. This is useful if you want to apply drawing properties, implement measurement algorithms, selectively delete (such as undo functionality) or extract specific strokes to copy into a new Ink Object.
Exploring Ink Data File Formats
You can represent ink in a number of different formats, including Ink Serialized Format (ISF), HTML, RTF, and XML. Of these, the most important and basic format is Ink Serialized Format (ISF), which is a highly compressed binary representation of the ink data from a lone ink object. The ISF format contains stroke information as well as the original device coordinates and is extensible through Microsoft’s support for arbitrary properties. In this case, Microsoft provides the extended property object for adding additional properties. For example, if you were storing a signature in ISF format, you could embed the client’s name and a security value in the extended property object. When you store and later reload the signature, your application can use this data to validate against the client’s user records to prevent possible fraud.
Applications whose documents contain only ink would use ISF. Applications that are concerned only with the “Ink as Ink” and not with handwriting recognition would fall into this category. For example, you would use ISF to for adding a layer of annotation to images without compromising the original image.
The low-level binary ISF format is the most compact representation of Ink, so it’s the format you should prefer for applications that support copy and paste Ink operations, using the ClipboardCopy(Rectangle), ClipboardCopy(Strokes), and ClipboardPaste() methods. The Tablet PC platform contains built-in methods for both collecting and copying-and-pasting ISF. ISF allows for the lowest level of ink interoperability. You must load data produced by an ISF Application into an ink-aware application to view it. Machines without the ink components installed (via the Microsoft Tablet PC SDK or your application’s install) will not be able to interpret the ISF format. In situations where interoperability between Windows platforms is essential, it is best to use a more commonly understood format?HTML.
HTML allows for many varied types of content. Ink data is represented in HTML as an image by applications that do not recognize ink. The Tablet PC Platform APIs provide the fortified GIF persistence format used for generating the image and, to an ink-enabled application, there is a rich set of data underlying the image. Fortified GIFs have ISF embedded within them.
It is possible to represent ink with other file formats as well. For example, you can embed OLE objects that contain ink within RTF files, meaning, you can paste ink into older applications, such as Microsoft Word, that do not natively recognize ink. You can use still other formats, such as binary or XML-based formats, in the appropriate circumstances. Unfortunately, HTML is not suitable for storing structured data. If your application gathers, stores, and manipulates structured data, you might find XML to be a more useful ink storage format. The Inking objects and controls of the Tablet PC platform SDK help you generate and interpret these formats whether you choose to implement using the Managed Library or the Automation API interfaces.
Persisting Ink Data
This sample code included with this article demonstrates how to save and load ink. It lets you save ink data in Ink Serialized Format (ISF), XML, and HTML format. The XML format stores ink as Base64-encoded ISF, while the HTML format stores the ink as a Base64-encoded fortified GIF image.
Using the sample you can open files saved using the XML and ISF formats, but the sample does not support loading from the HTML format. You can view the HTML output with your browser or MS Word. Using HTML can be a “one-way” street. Though the ink controls generate fortified GIF files, the application used to view the HTML is in charge of evaluating the .GIF, and the results can vary. For your HTML to invoke the ActiveX Inking Controls, you need to generate the HTML to include the
Author Note: As of this writing, Internet Explorer has no built-in capability for working with ink.
Collecting Ink
 | |
| Figure 2: Selecting the Microsoft InkEdit Control and/or InkPicture control will add these controls to your toolbox. |
To get started, create a new .NET Windows application, and then click the Add Reference item in the Solution Explorer. That brings up the Add Reference dialog box. Click the .NET tab, and then select the Microsoft Tablet PC API component to add the reference. Adding the reference gives your application access to the Tablet PC object model and controls. By default, the library installs to systemdrive:Program FilesMicrosoft Tablet PC Platform SDKIncludeMicrosoft.Ink.dll.
Unfortunately, adding the library reference doesn’t automatically add the available controls to the ToolBox. To do that, right-click within the Toolbox, and then click the Customize Toolbox item. In the Customize Toolbox dialog box, click the .NET Framework Components tab. Select the InkEdit and/or InkPicture controls from the list box (see Figure 2), and then click OK.
Add an Imports statement for the Microsoft.Ink library to the top of your code module.
Imports Microsoft.Ink
Then in the New() constructor for the form, create an InkCollector for the form and enable it.
myInkCollector = New InkCollector(gbInkArea.Handle) myInkCollector.Enabled = True
Saving Ink to Files
Example 1: Creating a Stream Object
The btnSaveFile_Click method displays a Save As dialog box. Initially the code creates a Stream object and opens a file with it. The code passes the Stream object by reference (byRef) to the save subroutine that corresponds to the user’s choice
Private Sub btnSaveFile_Click(ByVal sender As _ System.Object, ByVal e As System.EventArgs) _ Handles btnSaveFile.Click 'This handler lets users specify a filename, ' location, and type of format to save as. 'Create a stream which will be used to save ' data to the output file Dim myStream As Stream 'Create the SaveFileDialog, and present to the user Dim mySaveDialog As SaveFileDialog = New _ SaveFileDialog() 'Set the filter to suggest our recommended extensions mySaveDialog.Filter = "Ink Serialized Format " & + "files (*.isf)|*.isf|XML files (*.xml)|*.xml|" & _ "HTML files (*.htm)|*.htm" 'If the dialog exits and the user didn't choose Cancel If mySaveDialog.ShowDialog = DialogResult.OK Then '(the try...catch section in this function ' handles the errors for the helper methods ' called.) Try 'Attempt to Open the file with read/write 'permission myStream = mySaveDialog.OpenFile() If Not myStream Is Nothing Then 'Change the filename to Lower Case Dim filename As String = _ mySaveDialog.FileName.ToLower() ' Get a version of the filename ' without an extension ' This will be used to save associated ' Ink Data Dim extensionlessFilename As String = _ Path.GetFileNameWithoutExtension(filename) 'Get the extension of the file Dim extension As String = _ Path.GetExtension(filename) ' Use the extension to determine what form ' to save the data in Select Case extension Case ".xml" SaveXML(myStream) Case ".htm", ".html" ' The two HTML cases require a ' filename for saving associated images SaveHTML(myStream, _ extensionlessFilename) Case Else ' If unfamiliar with the extension, use ' ISF, the most "native format" SaveISF(myStream) End Select Else ' Throw an exception if a null pointer is ' returned for the stream Throw New IOException() End If Catch excpt As IOException MessageBox.Show("File error") Finally ' Close the stream in the finally clause so it ' is always reached, regardless of whether an ' exception occurs. SaveXML, SaveHTML, and ' SaveISF can throw, so this precaution is ' necessary. If Not myStream Is Nothing Then myStream.Close() End If End Try End If 'End if user chose OK from dialog End SubExample 2: Saving to an ISF File
The SaveISF() method serializes the ink to ISF format and writes the result to the file using the InkCollector.Save() method, specifying ISF as the PersistenceFormat.
Sub SaveISF(ByRef s As Stream) ' This function saves the form in ISF format. Dim isf As Byte() ' Perform the serialization isf = myInkCollector.Ink.Save _ (PersistenceFormat.InkSerializedFormat) ' Write the ISF to the stream s.Write(isf, 0, isf.Length) End Sub
Example 3: Saving to an XML File
 | |
| Figure 3: The application saves collected text data as XML along with the applicant’s signature persisted in fortified .GIF format. |
The SaveXML() method uses an XmlTextWriter object to create and write to an XML document. Calling the Ink object’s Save() method with the PersistenceFormat.Base64InkSerializedFormat enumeration value as a parameter converts the ink to a Base64 encoded ISF byte array. Finally, the code converts that byte array to a string and writes it to the XML file. You could also write any ancillary text data from the form to the XML file if you wished. For example, the application you were contracted to write for “Bob’s Mighty Mega Loans” accepts the entry of the applicant’s name, address, income and social security number. When users have reviewed their application, they sign on the dotted line (on the ink control), and click the submit button (see Figure 3). The text data, along with the client’s signature, is written to XML.
Sub SaveXML(ByRef s As Stream) ' This function saves the form in XML format. ' It uses the base64 encoded version of the ink, ' which is most suitable for use XML. ' This object will encode our byte data to ' a UTF8 string Dim utf8 As UTF8Encoding = New UTF8Encoding() Dim base64ISF_bytes As Byte() Dim base64ISF_string As String ' Create a new XmlTextWriter. Dim myXWriter As XmlTextWriter = _ New XmlTextWriter(s, System.Text.Encoding.UTF8) ' Write the beginning of the document including ' the document declaration. myXWriter.WriteStartDocument() ' Write the beginning of the "data" element. ' This is the opening tag to our data myXWriter.WriteStartElement("SerializationSampleData") ' Get the base64 encoded ISF base64ISF_bytes = myInkCollector.Ink.Save _ (PersistenceFormat.Base64InkSerializedFormat) ' Convert it to a String base64ISF_string = utf8.GetString(base64ISF_bytes) ' Write the ISF containing node to the XML myXWriter.WriteElementString("Ink", base64ISF_string) 'End the "data" element. myXWriter.WriteEndElement() 'End the document myXWriter.WriteEndDocument() 'Close the xml document. myXWriter.Close() End SubExample 4: Saving to an HTML File
The SaveHTML() method uses the bounding box of the Strokes collection to test for the presence of ink data. If any ink exists, the method calls Ink.Save() with the PersistenceFormat.Gif enumeration value to convert it to the fortified GIF format.
fortifiedGif = myInkCollector.Ink.Save(PersistenceFormat.Gif)
The code writes the GIF produced to a file using the GIF extension. To an application that is not ink-enabled, there is no difference between a fortified GIF and a normal GIF. An IMG tag in HTML must reference the fortified GIF. The HTML generated by the .SAVE method is in Office Clipboard format (CF_HTML), making the HTML available and visible to other applications even if they are not ink-enabled. Ink-enabled applications can evaluate the HTML and determine if the .GIF is fortified.
The following examples refer to a fortified GIF by using HTML tags:
or
Sub SaveHTML(ByRef s As Stream, ByVal nameBase As String) ' This function saves the form in HTML format. ' It also creates an ink fortified GIF which is ' referenced by the HTML. ' It is not possible to save an empty .gif, so ' ensure that the ink ' has a bounding box (i.e. the ink is not empty). ' Use bounding box for this check instead of ' checking if the stroke count is zero, since the ' ink could still be empty if the strokes ' don't contain any points. If (myInkCollector.Ink.Strokes.GetBoundingBox() _ .IsEmpty) Then MessageBox.Show("Unable to save empty ink in " & _ "HTML persistence format.") Else Dim gifFile As FileStream Dim fortifiedGif As Byte() Dim html As String Dim htmlBytes As Byte() ' This object will encode our byte data to a ' UTF8 string Dim utf8 As UTF8Encoding = New UTF8Encoding() ' Create a directory to store the ' fortified GIF which also contains ISF ' and open the file for writing Directory.CreateDirectory(nameBase + "_files") gifFile = File.OpenWrite(nameBase + _ "_files\myInk.gif") ' Generate the fortified GIF representation ' of the ink fortifiedGif = myInkCollector.Ink.Save _ (PersistenceFormat.Gif) ' Write and close the gif file gifFile.Write(fortifiedGif, 0, fortifiedGif.Length) gifFile.Close() ' Create the HTML output ' Note that the names are stored in HTML ' tags, rather than custom properties, so that ' these properties can be easily retrieved from ' the HTML. html = _ "" & _ "
Ink Save Test: " & nameBase & "
" & _ "" & _ "" ' Convert the HTML to a byte array for ' writing to the stream htmlBytes = utf8.GetBytes(html) ' Write the HTML to the stream s.Write(htmlBytes, 0, htmlBytes.Length) End If End Sub
![](" & _ nameBase + "_files\myInk.gif" & _ ")
Example 5: Saving to a SQL Server Database.
The btnSaveDB_Click() method also creates a Stream object; however, instead of a FileStream Object, the code uses a MemoryStream object. The SQL Server Image datatype is a binary datatype and is perfect for holding ink data. You have to convert the ink to an array of bytes using the InkCollector.Ink.Save() method. The following example uses the exiting “Picture” table in the northwinds sample database.
Private Sub btnSaveDB_Click(ByVal sender As _ System.Object, ByVal e As System.EventArgs) _ Handles btnSaveDB.Click ' The SQL Server Image datatype is a binary datatype. ' To save it to the database we must convert the ink ' to an array of bytes. We will use ' MemoryStream with the inkCollector.Ink.Save method. Dim Filename As String Dim isf As Byte() If inkEdFileName.Text.Length = 0 Then MessageBox.Show("You must enter a file name.", _ "Missing File Name", _ MessageBoxButtons.OK, _ MessageBoxIcon.Error) Else ' Perform the serialization isf = myInkCollector.Ink.Save _ (PersistenceFormat.InkSerializedFormat) 'append the .ink extension to denote .INK data inkEdFileName.Text = _ Trim(inkEdFileName.Text) & ".ink" Filename = inkEdFileName.Text Dim isConnecting As Boolean = True While isConnecting Try Dim northwindConnection As _ New SqlConnection(connectionString) Dim strSQL As String = _ "INSERT INTO Picture (Filename, Picture)" & _ "VALUES (@Filename, @Picture)" myCmd = New SqlCommand(strSQL, _ northwindConnection) With myCmd ' Add parameters required by SQL ' statement. 'PictureID is an ' identity field so only pass values for 'the two remaining fields. .Parameters.Add(New SqlParameter _ ("@Filename", _ SqlDbType.NVarChar, 50)).Value _ = Filename .Parameters.Add(New SqlParameter_ ("@Picture", SqlDbType.Image)).Value _ = isf End With ' Open the connection, execute the command, 'and close the connection. northwindConnection.Open() myCmd.ExecuteNonQuery() northwindConnection.Close() ' Data has been successfully submitted, 'so break out of the loop isConnecting = False MessageBox.Show(Filename & _ " saved to the " & _ "database.", _ "Ink Save Status", MessageBoxButtons.OK, _ MessageBoxIcon.Information) Catch sqlExc As SqlException MessageBox.Show(sqlExc.ToString, _ "SQL Exception Error!", _ MessageBoxButtons.OK, _ MessageBoxIcon.Error) Exit While Catch exc As Exception ' Unable to connect to SQL Server MessageBox.Show("Error connecting " & _ "to Database.", _ "Connection Failed!", _ MessageBoxButtons.OK, _ MessageBoxIcon.Error) End Try End While End If End SubNow that you know how to save ink in the various file formats, the next step is learning how to retrieve it.
Loading Ink from Files
Example 6: Opening Files Containing Ink Data
The btnOpen_Click() method handles the Open dialog. By creating a Stream object, you can call the load method that corresponds to the user’s choice, passing the Stream object ByRef.
Private Sub btnOpen_Click(ByVal sender As _ System.Object, ByVal e As System.EventArgs) _ Handles btnOpen.Click [. . .] If myOpenFileDialog.ShowDialog() = _ DialogResult.OK Then Try ' Attempt to Open the file with ' read-only permission myStream = myOpenFileDialog.OpenFile() If Not myStream Is Nothing Then ' Just a readability move Dim filename As String = _ myOpenFileDialog.FileName.ToLower() ' Get the extension of the file Dim extension As String = _ filename.Substring _ (filename.LastIndexOf(".")) ' Use the extension to determine what ' form to save the data in Select Case extension Case ".isf" LoadISF(myStream) Case ".xml" LoadXML(myStream) Case Else ' If unfamiliar with the extension, ' assume ISF, the most "native format" LoadISF(myStream) End Select inkEdFileName.Text = filename 'repaint the inkEdit area inkEdFileName.Invalidate() Catch 'Display an error and exit MessageBox.Show("An error occurred while " & _ "loading ink from " & _ " the specified file. Please try again.",_ "Serialization", MessageBoxButtons.OK) Finally ' Always close the stream in the finally ' clause so it is always reached, regardless ' of whether an exception occurs. If Not myStream Is Nothing Then _ myStream.Close() End Try [. . .] End SubExample 7: Loading Ink from an ISF File
The LoadISF() method reads a Byte array from a previously created file and converts the Byte array to ink with the Ink object’s Load() method. You should temporarily disable the ink collector while assigning the Ink object to it.
Private Sub LoadISF(ByVal s As Stream) 'Load ISF into the ink object Dim LoadedInk As Ink = New Ink() Dim isfBytes() As Byte = New Byte() {0} ReDim isfBytes(s.Length) ' read in the ISF s.Read(isfBytes, 0, CInt(s.Length)) ' Load the ink into a new ink object ' Once an ink object has been "dirtied" it can ' never load ink again LoadedInk.Load(isfBytes) ' temporarily disable the ink collector and ' swap ink objects myInkCollector.Enabled = False myInkCollector.Ink = LoadedInk myInkCollector.Enabled = True ' Repaint the inkable region gbInkArea.Invalidate() End SubExample 8: Loading Ink from an XML File
The LoadXML() subroutine loads a previously created XML file, retrieves data from the Ink node, and converts the data in the node to ink using the Ink object’s Load method. You should temporarily disable the ink collector while assigning the Ink object to it. Call the Invalidate method of the ink collection box to force it to refresh..
Private Sub LoadXML(ByVal s As Stream) ' This function will load XML into the ink object. ' This object will encode our byte data to a ' UTF8 string Dim utf8 As UTF8Encoding = New UTF8Encoding() Dim myXD As XmlDocument = New XmlDocument() Dim myNodes As XmlNodeList Dim loadedInk As Ink = New Ink() ' Load the XML data into an XMLDocument object myXD.Load(s) ' Get the data in the ink node myNodes = myXD.GetElementsByTagName("Ink") ' load the ink into a new ink object ' once an ink object has been "dirtied" it can never ' load ink again If myNodes.Count <> 0 Then loadedInk.Load(utf8.GetBytes(myNodes(0).InnerXml)) ' temporarily disable the ink collector and ' swap ink objects myInkCollector.Enabled = False myInkCollector.Ink = loadedInk myInkCollector.Enabled = True ' Repaint the inkable region gbInkArea.Invalidate() End If End SubExample 9: Loading Ink from SQL Server
To load ink data from the SQL Server Database, you can reuse a little code. In the btnDisplay_Click() method, using the FileName column as the lookup value, you can use a simple data adapter to retrieve the information. Then you can use a Byte array and a Stream object to convert the data to ink format using the LoadISF() routine outlined earlier.
Private Sub btnDisplay_Click(ByVal sender As _ System.Object, ByVal e As System.EventArgs) _ Handles btnDisplay.Click ' Handles the Display button click event, allowing ' the user to display an image stored in the database. ' When nothing is selected in the ListBox, ' the SelectedIndex = -1. If lstInk.SelectedIndex < 0 Then MessageBox.Show("There are no ink selections " & _ "in the database to display.", _ "Empty Database!", MessageBoxButtons.OK, _ MessageBoxIcon.Information) Else ' The SQL Server Image datatype is a ' binary datatype. ' To generate ink from it, you must first create a ' stream object containing the binary data. Dim arrInk() As Byte = CType(dsInk.Tables(0). _ Rows(lstInk.SelectedIndex)("Picture"), Byte()) Dim myStream As New MemoryStream(arrInk) Try LoadISF(myStream) inkEdFileName.Text = dsInk.Tables(0).Rows _ (lstInk.SelectedIndex)("FileName").ToString Catch 'Display an error and exit MessageBox.Show("An error occurred while " & _ "loading ink. " & _ " Please verify the data selected " & _ "contains valid serialized " & _ "ink and try again.", "Serialization", _ MessageBoxButtons.OK) Finally ' Close the stream object to ' release the resource. If Not myStream Is Nothing Then myStream.Close() End If End Try End If End SubYou can see that you handle ink in much the same way as you would handle image data. . You should review the Visual Design guidelines within the Tablet PC SDK before writing production ink application?especially those dealing with Tablet PC System Sizing Metrics for High DPI.
Since initial publication of this article, Microsoft has released an updated SDK as well as merged the Tablet PC API into the latest edition of Windows XP. Some features and functions in this article may therefore have changed and improved with the latest tools.
