Browse DevX
Sign up for e-mail newsletters from DevX


Scaling and Hit-Testing in Ink Applications

In addition to text recognition and similar capabilities, Ink can also be useful for image annotation and markup, such as in medical and insurance applications, where marking up images can be a valuable and critical form of input.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

t's common in imaging applications for images to be displayed at different sizes. If such images are annotated with Ink, it's necessary for the Ink to be scaled to the current size of the image. Developers just starting with the Tablet PC often find that this is trickier than they thought. This article covers the basics of scaling Ink, including a quick introduction to the concept of transforms, which are one of the underlying mechanisms for scaling Ink, and one about which most developers have never had to worry.

Annotation applications also need to know whether Ink has been entered into a particular area of a drawing. For example, an insurance application might annotate a picture of a car to indicate damaged places. The application might then need to detect whether an annotation had been added that indicated damage only to the door of the vehicle.

This capability is called "hit testing." The basic technique to do hit testing for Ink is pretty easy. It uses a method of the Ink object, which checks to see if Ink is within or around a given rectangle. But given that the image may be scaled to different sizes and resolutions, the concept of scaling also applies to hit testing, because the rectangle for hit testing must be sized and positioned properly for the currently displayed image.

Because these topics are related, this article also looks at the basics of hit testing, and demonstrates how to scale hit tests to various image sizes.

Author's Note: The examples in this article are in Visual Basic .NET, but can be translated to C# easily. The classes used are the .NET wrapper classes for Ink, so all examples must be run on the .NET Framework. The step-by-step construction of the examples assumes that you have a Tablet PC already set up with Visual Studio 2003, and that the standard Ink controls (InkPicture and InkEdit) are present in your Visual Studio toolbox. These examples should work fine with Visual Studio 2005.

Basics of Scaling
Scaling Ink becomes an issue in two typical scenarios. First, if an image dynamically becomes larger or smaller in an application, any Ink that overlays the image must scale as well. Second, if Ink overlay is stored separately from an image and later retrieved, it must be scaled to the current size of the image.

To see the scaling problem, let's start the example application. In Visual Studio, create a new Windows Application, and name it ScaleAndHitTestSample. When the application comes up, add a new form to the project named InkScaleForm. Drag the corner of the form to make it about twice as wide and twice as high as it appeared by default.

Drag an InkPicture control to the form, and name the control VehicleImage. Set the Image property for the InkPicture to some appropriate bitmapped image. In the downloadable sample application, you'll find a yellow car image as in Figure 1. Set the Anchor property for the InkPicture to anchor the image to all four sides of the form.

Figure 1: Non-resizable Ink: With the SizeMode property of the InkPicture control set to StretchImage, the ink doesn't resize with the image.
The InkPicture control inherits from the PictureBox control, and has a property derived from PictureBox called SizeMode. By default, this is set to Normal, which means that the image displays from the top left of the control, and that there will be no sizing of the image to accommodate the control's size. If the image is larger than the control, the image will be clipped.

The InkPicture control has Ink enabled by default, so you're ready to test it. Notice that the default color of the Ink is black, and you may want to change it. The sample screens for this article, for example, use red ink. To change the default Ink color, insert this line in the form's Load event:

VehicleImage.DefaultDrawingAttributes.Color = _ Color.Red

To start the test, change the properties for the project so that InkScaleForm is the startup form, and run the program.

You can resize the form and the InkPicture control should resize with it. Depending on your image size and the size of your form, you'll see more or less of the image. You can put Ink on the control with your pen, and the Ink will also be shown or hidden, depending on the size of the form.

Now stop the program. Change the SizeMode property of the InkPicture control to StretchImage. Run the program again.

Now the image is sized to the control as the form is resized. However, if you put Ink on the image, notice that the Ink is not resized. It remains the same size as it was when you entered it, regardless of the size of the image. Figure 1 shows this problem.

If an image dynamically becomes larger or smaller in an application, the Ink that overlays the image must scale as well.
Adding Basic Scaling
Let's add basic scaling so that the Ink adjusts to the image. First, you'll need several Imports statements (Using for C# folks) to easily access some classes you'll need. Add the following statements at the very top of the code for the InkScaleForm:

Imports System.Drawing Imports System.Drawing.Drawing2D Imports Microsoft.Ink

Now add the following subroutine to the program:

Private Sub SetInkTransform() Dim newTransform As New Matrix If VehicleImage.SizeMode = PictureBoxSizeMode.Normal Then ' the scale is one-to-one, so we don't ' need to change ' the matrix from the default identity ' matrix Else Dim xScale As Single Dim yScale As Single xScale = VehicleImage.Width / _ VehicleImage.Image.PhysicalDimension.Width yScale = VehicleImage.Height / _ VehicleImage.Image.PhysicalDimension.Height newTransform.Scale(xScale, yScale) End If VehicleImage.Renderer.SetViewTransform(new _ Transform) End Sub

Some of the available Matrix transforms are scaling (making everything larger or smaller), repositioning (moving all coordinates the same direction and distance), and rotation.
The Matrix class from the System.Drawing.Drawing2D namespace is the main .NET framework class you need to understand to do transforms. It helps if you recall a bit of your linear algebra. Transforms in a coordinate space can be done in several ways. Some of them include scaling (making everything larger or smaller), repositioning (moving all coordinates the same direction and distance), and rotation. I'll just look at scaling, which is the only transform that applies to Ink in this example.

When a new instance of the Matrix class is created, it represents the identity Matrix, which just means "don't change anything." This instance represents the case in which no scaling or other transformation of the coordinate data is needed.

Ink data includes coordinates. That means Ink data can be scaled using a transform. The identity transform applied to Ink says "don't do any transformation of the ink coordinates."

Figure 2. Scalable Ink: The figures shows how Ink on the image scales after applying a matrix transform.
An instance of the Matrix class can have its scale changed with the Scale method, which takes two arguments. One number tells how much to scale coordinates in the horizontal (X) dimension. The other tells how much scale in the vertical, or Y, dimension.

The SetInkTransform routine above calculates these scaling values by dividing the size of the control by the size of the image. That gives a factor by which the image has been expanded or shrunk. One such factor is needed for each dimension.

Once you apply the Scale method to the Matrix, it is capable of scaling other coordinates the same way as the InkPicture scaled the image. Of course, you want to apply the scaling to Ink. To do that, you need to tell the Ink rendering object what scale to use.

Figure 3. Scaling Ink: A translucent rectangle has been painted on the door with GDI+ to help see the area being hit tested.
The ink rendering object is an instance of the Renderer class, which is in the Microsoft.Ink namespace. The Renderer accepts a Matrix for transformation of the Ink it manages, using the SetViewTransform method.

Once a Matrix has been applied to a Renderer, any existing Ink is automatically scaled to the new transform. New Ink that is entered has its coordinate data adjusted to accommodate the new scale, so that all Ink managed by the Renderer is on the same coordinate system (see Figure 2).

The SetInkTransform subroutine should be called in the Form Load event, and in the Resize event of the InkPicture control. Once that's accomplished, run the program again. Now, any Ink on the image scales with the image. Ink that you input at one size will scale to any other size. Figure 3 shows the results in the downloadable sample application.

Thanks for your registration, follow us on our social networks to keep up-to-date