devxlogo

Scaling and Hit-Testing in Ink Applications

Scaling and Hit-Testing in Ink Applications

t’s common in imaging applications for images to be displayed at different sizes. If such images are annotated with Ink, it’s necessary for the Ink to be scaled to the current size of the image. Developers just starting with the Tablet PC often find that this is trickier than they thought. This article covers the basics of scaling Ink, including a quick introduction to the concept of transforms, which are one of the underlying mechanisms for scaling Ink, and one about which most developers have never had to worry.

Annotation applications also need to know whether Ink has been entered into a particular area of a drawing. For example, an insurance application might annotate a picture of a car to indicate damaged places. The application might then need to detect whether an annotation had been added that indicated damage only to the door of the vehicle.

This capability is called “hit testing.” The basic technique to do hit testing for Ink is pretty easy. It uses a method of the Ink object, which checks to see if Ink is within or around a given rectangle. But given that the image may be scaled to different sizes and resolutions, the concept of scaling also applies to hit testing, because the rectangle for hit testing must be sized and positioned properly for the currently displayed image.

Because these topics are related, this article also looks at the basics of hit testing, and demonstrates how to scale hit tests to various image sizes.

Author’s Note: The examples in this article are in Visual Basic .NET, but can be translated to C# easily. The classes used are the .NET wrapper classes for Ink, so all examples must be run on the .NET Framework. The step-by-step construction of the examples assumes that you have a Tablet PC already set up with Visual Studio 2003, and that the standard Ink controls (InkPicture and InkEdit) are present in your Visual Studio toolbox. These examples should work fine with Visual Studio 2005.

Basics of Scaling
Scaling Ink becomes an issue in two typical scenarios. First, if an image dynamically becomes larger or smaller in an application, any Ink that overlays the image must scale as well. Second, if Ink overlay is stored separately from an image and later retrieved, it must be scaled to the current size of the image.

To see the scaling problem, let’s start the example application. In Visual Studio, create a new Windows Application, and name it ScaleAndHitTestSample. When the application comes up, add a new form to the project named InkScaleForm. Drag the corner of the form to make it about twice as wide and twice as high as it appeared by default.

Drag an InkPicture control to the form, and name the control VehicleImage. Set the Image property for the InkPicture to some appropriate bitmapped image. In the downloadable sample application, you’ll find a yellow car image as in Figure 1. Set the Anchor property for the InkPicture to anchor the image to all four sides of the form.

?
Figure 1: Non-resizable Ink: With the SizeMode property of the InkPicture control set to StretchImage, the ink doesn’t resize with the image.

The InkPicture control inherits from the PictureBox control, and has a property derived from PictureBox called SizeMode. By default, this is set to Normal, which means that the image displays from the top left of the control, and that there will be no sizing of the image to accommodate the control’s size. If the image is larger than the control, the image will be clipped.

The InkPicture control has Ink enabled by default, so you’re ready to test it. Notice that the default color of the Ink is black, and you may want to change it. The sample screens for this article, for example, use red ink. To change the default Ink color, insert this line in the form’s Load event:

   VehicleImage.DefaultDrawingAttributes.Color = _      Color.Red

To start the test, change the properties for the project so that InkScaleForm is the startup form, and run the program.

You can resize the form and the InkPicture control should resize with it. Depending on your image size and the size of your form, you’ll see more or less of the image. You can put Ink on the control with your pen, and the Ink will also be shown or hidden, depending on the size of the form.

Now stop the program. Change the SizeMode property of the InkPicture control to StretchImage. Run the program again.

Now the image is sized to the control as the form is resized. However, if you put Ink on the image, notice that the Ink is not resized. It remains the same size as it was when you entered it, regardless of the size of the image. Figure 1 shows this problem.

If an image dynamically becomes larger or smaller in an application, the Ink that overlays the image must scale as well.

Adding Basic Scaling
Let’s add basic scaling so that the Ink adjusts to the image. First, you’ll need several Imports statements (Using for C# folks) to easily access some classes you’ll need. Add the following statements at the very top of the code for the InkScaleForm:

   Imports System.Drawing   Imports System.Drawing.Drawing2D   Imports Microsoft.Ink

Now add the following subroutine to the program:

   Private Sub SetInkTransform()      Dim newTransform As New Matrix      If VehicleImage.SizeMode =          PictureBoxSizeMode.Normal Then         ' the scale is one-to-one, so we don't          ' need to change         ' the matrix from the default identity          ' matrix      Else         Dim xScale As Single         Dim yScale As Single         xScale = VehicleImage.Width / _             VehicleImage.Image.PhysicalDimension.Width            yScale = VehicleImage.Height / _            VehicleImage.Image.PhysicalDimension.Height            newTransform.Scale(xScale, yScale)      End If      VehicleImage.Renderer.SetViewTransform(new _         Transform)   End Sub

Some of the available Matrix transforms are scaling (making everything larger or smaller), repositioning (moving all coordinates the same direction and distance), and rotation.

The Matrix class from the System.Drawing.Drawing2D namespace is the main .NET framework class you need to understand to do transforms. It helps if you recall a bit of your linear algebra. Transforms in a coordinate space can be done in several ways. Some of them include scaling (making everything larger or smaller), repositioning (moving all coordinates the same direction and distance), and rotation. I’ll just look at scaling, which is the only transform that applies to Ink in this example.

When a new instance of the Matrix class is created, it represents the identity Matrix, which just means “don’t change anything.” This instance represents the case in which no scaling or other transformation of the coordinate data is needed.

Ink data includes coordinates. That means Ink data can be scaled using a transform. The identity transform applied to Ink says “don’t do any transformation of the ink coordinates.”

?
Figure 2. Scalable Ink: The figures shows how Ink on the image scales after applying a matrix transform.

An instance of the Matrix class can have its scale changed with the Scale method, which takes two arguments. One number tells how much to scale coordinates in the horizontal (X) dimension. The other tells how much scale in the vertical, or Y, dimension.

The SetInkTransform routine above calculates these scaling values by dividing the size of the control by the size of the image. That gives a factor by which the image has been expanded or shrunk. One such factor is needed for each dimension.

Once you apply the Scale method to the Matrix, it is capable of scaling other coordinates the same way as the InkPicture scaled the image. Of course, you want to apply the scaling to Ink. To do that, you need to tell the Ink rendering object what scale to use.

?
Figure 3. Scaling Ink: A translucent rectangle has been painted on the door with GDI+ to help see the area being hit tested.

The ink rendering object is an instance of the Renderer class, which is in the Microsoft.Ink namespace. The Renderer accepts a Matrix for transformation of the Ink it manages, using the SetViewTransform method.

Once a Matrix has been applied to a Renderer, any existing Ink is automatically scaled to the new transform. New Ink that is entered has its coordinate data adjusted to accommodate the new scale, so that all Ink managed by the Renderer is on the same coordinate system (see Figure 2).

The SetInkTransform subroutine should be called in the Form Load event, and in the Resize event of the InkPicture control. Once that’s accomplished, run the program again. Now, any Ink on the image scales with the image. Ink that you input at one size will scale to any other size. Figure 3 shows the results in the downloadable sample application.

Hit-testing
Now suppose you want to detect whether or not the Ink annotations include any Ink in a certain area. For this example, I’ll use the door of the vehicle.

Normally, hit testing is done within a rectangle, which is represented by the Rectangle class in System.Drawing. Rectangle is just a structure that contains the position and size of the rectangle.

When working on hit testing, it helps a lot if you can see the rectangle that’s being used. That usually means painting the rectangle in the Paint event of the form or control on which the rectangle needs to appear.

It’s easy enough to paint an opaque rectangle, but it’s almost as easy to paint a rectangle that’s transparent, so that you can see what’s behind it. Let’s start the hit test example by placing a transparent rectangle on the vehicle door.

Change the SizeMode of the InkPicture control back to Normal. Then add a member variable to the InkScaleForm with this line of code:

   Private HitTestRectangle As New _       Rectangle(230, 200, 160, 120)

The size and position specified in this line work well for the door in the example image I used, but you might want to adjust those numbers for a different position in your image.

Next, create an event handler for the Paint event of the VehicleImage control. In the Paint event, place the following code:

   Dim brushHighlight As New _      SolidBrush(Color.FromArgb(80, _      Color.Magenta))   e.Graphics.FillRectangle(brushHighlight, _      HitTestRectangle)   brushHighlight.Dispose()

This demonstrates the use of GDI+ in Windows Forms to do some drawing directly to a control. First, you create a brush, which is the object used to fill an area. In this case, the type of brush is a SolidBrush, but that’s a little misleading. The color you specify using a special method of the Color class called FromArgb. This rather obscurely named method creates a color with transparency. The first argument in the method is a number from 0 to 255 that specifies how much opacity is desired, with 0 being none and 255 being total. I’m using 80 in the example above, which is a typical setting that paints some color but still shows much of the background.

The brush is then used to paint the area specified by the HitTestRectangle. That’s done with the FillRectangle method, which is available on an instance of the Graphics class. The Graphics class represents a rectangular area on which to draw. Paint events always expose a Graphics instance in the event argument, as e.Graphics.

After drawing, dispose of the brush, which is good practice because GDI+ objects are connected to underlying operating system objects.

It’s easy enough to paint an opaque rectangle, but it’s almost as easy to paint a rectangle that’s transparent, so that you can see what’s behind it.

If you run the program now, you’ll see the transparent rectangle painted, as in Figure 3. However, it’s always displayed in the same place. If the SizeMode for the InkPicture is changed back to StretchImage, the transparent rectangle will not move with the image as it resizes.

The next step, then, is to set the rectangle so that it will move with the image. To do that, you need to scale and reposition the rectangle. Add the three functions to the form shown in Listing 1.

Listing 1 creates a new rectangle at any necessary instant that is scaled to the current size of the image, much the same way the transform was scaled for Ink earlier. Note that while the Renderer automatically repositioned the Ink when you set the scale, you have to do that manually for the rectangle. That is, not only do you have to scale the size of the rectangle, but you also have to scale its location.

Now, you need to change the Paint event for the VehicleImage control to use the new scaled rectangle. The new Paint event should look like this.

   Private Sub VehicleImage_Paint(ByVal sender As _      Object, ByVal e As       System.Windows.Forms.PaintEventArgs) _      Handles VehicleImage.Paint         If VehicleImage.Image Is Nothing Then         Exit Sub      End If      Dim brushHighlight As New _        SolidBrush(Color.FromArgb(80, _        Color.Magenta))      e.Graphics.FillRectangle(brushHighlight, _         ScaledHitTestRectangle)      brushHighlight.Dispose()   End Sub

This is just like the previous Paint event code, except that you use ScaledHitTestRectangle instead of HitTestRectangle, and test to make sure there’s an image that can be used for scaling. (If there’s no image, the scaling operation will fail because it can’t fetch the image size.)

Now run the program and resize the form. The transparent rectangle scales with the image.

Hit-testing Logic
Next, it’s time to add the hit testing logic. First, you need a support function. Add this function to the form:

   Private Function GetInkRectangleFromPixelRectangle( _       ByVal g As Graphics, ByVal rect As _      Rectangle)      Dim topLeft As Point = rect.Location      Dim bottomRight As New _         Point(rect.Location.X + _         rect.Size.Width, _         rect.Location.Y + _         rect.Size.Height)      VehicleImage.Renderer.PixelToInkSpace( _         g, topLeft)      VehicleImage.Renderer.PixelToInkSpace( _         g, bottomRight)      Dim newRect As Rectangle      newRect.Location = topLeft      newRect.Width = bottomRight.X - topLeft.X      newRect.Height = bottomRight.Y - topLeft.Y      Return newRect   End Function

The preceding function, GetInkRectangleFromPixelRectangle, takes a rectangle specified in pixel coordinates, and translates it to a rectangle in terms of Ink coordinates. You’ll need that capability in the hit testing logic later.

The function uses the PixelToInkSpace method of the Renderer object. The method takes a point on the control, measured in pixel coordinate space, and translates it to the equivalent point in Ink coordinate space. The method also takes the Matrix that was previously applied to the Renderer into account. So this function converts pixel coordinates to Ink coordinates and also takes scaling into account.

Now, place a button named HitTestButton on the form, and add the following logic to the button’s click event:

   ' First translate hit test rectangle to ink space   Dim InkHitTestRectangle As Rectangle   Dim g As Graphics = VehicleImage.CreateGraphics   InkHitTestRectangle = _      GetInkRectangleFromPixelRectangle(g, _      ScaledHitTestRectangle)   Dim HitStrokes As Strokes   HitStrokes =       VehicleImage.Ink.HitTest(InkHitTestRectangle, 0.01)      If HitStrokes.Count > 0 Then      MessageBox.Show("Found ink on or around door")   Else      MessageBox.Show("Little or no ink around _         door")   End If

You already have the rectangle you want to hit test, but it must be changed to Ink coordinate space. The function you added earlier is available for that. The rectangle in Ink space is then hit tested using the HitTest method of the Ink object on the VehicleImage control. This method returns any strokes whose rectangle in Ink space overlaps the hit test rectangle.

A stroke close to the door, but not quite on it, might still generate a rectangle containing a little bit of the door, so there is a numeric factor to specify how much of the stroke is within the hit test rectangle. You can use a value of 0.01, or 1 percent. That fudge factor could be adjusted, of course.

The results of the hit test are a collection of strokes that satisfy the hit test. If that collection contains any strokes, the user gets a message that the hit test was successful.

Notice that one of the implications of this hit test technique is that a circle around the door that does not touch the door will still give a positive hit test. This is typically desirable, and that’s good, because a hit test that does not work that way requires a different and more difficult technique.

The HitTest method has overloads to do hit testing inside a circle, or inside an irregularly shaped area specified by a lasso. These overloads are used in a similar fashion to the rectangular hit test above. Remember that any coordinates used in these methods must be properly scaled and translated to Ink space coordinates before being used for hit testing.

Apply the Techniques
You’ve now seen the basics of scaling and hit testing of Ink on top of an image. These basic techniques can be applied to most applications that use Ink annotations on images.

Scaling is relatively simple once you understand how to create an appropriate Matrix object to specify a scaling transform. The Renderer object does a great job of handling Ink scaling tasks, once it has been told what scale to use.

Hit testing can be considerably more complex than shown here. For example, hit testing against non-rectangular regions might be needed. But understanding the simplest case of hit testing and how scaling affects it is a good grounding for further hit testing work.

Hit-testing
Now suppose you want to detect whether or not the Ink annotations include any Ink in a certain area. For this example, I’ll use the door of the vehicle.

Normally, hit testing is done within a rectangle, which is represented by the Rectangle class in System.Drawing. Rectangle is just a structure that contains the position and size of the rectangle.

When working on hit testing, it helps a lot if you can see the rectangle that’s being used. That usually means painting the rectangle in the Paint event of the form or control on which the rectangle needs to appear.

It’s easy enough to paint an opaque rectangle, but it’s almost as easy to paint a rectangle that’s transparent, so that you can see what’s behind it. Let’s start the hit test example by placing a transparent rectangle on the vehicle door.

Change the SizeMode of the InkPicture control back to Normal. Then add a member variable to the InkScaleForm with this line of code:

   Private HitTestRectangle As New _       Rectangle(230, 200, 160, 120)

The size and position specified in this line work well for the door in the example image I used, but you might want to adjust those numbers for a different position in your image.

Next, create an event handler for the Paint event of the VehicleImage control. In the Paint event, place the following code:

   Dim brushHighlight As New _      SolidBrush(Color.FromArgb(80, _      Color.Magenta))   e.Graphics.FillRectangle(brushHighlight, _      HitTestRectangle)   brushHighlight.Dispose()

This demonstrates the use of GDI+ in Windows Forms to do some drawing directly to a control. First, you create a brush, which is the object used to fill an area. In this case, the type of brush is a SolidBrush, but that’s a little misleading. The color you specify using a special method of the Color class called FromArgb. This rather obscurely named method creates a color with transparency. The first argument in the method is a number from 0 to 255 that specifies how much opacity is desired, with 0 being none and 255 being total. I’m using 80 in the example above, which is a typical setting that paints some color but still shows much of the background.

The brush is then used to paint the area specified by the HitTestRectangle. That’s done with the FillRectangle method, which is available on an instance of the Graphics class. The Graphics class represents a rectangular area on which to draw. Paint events always expose a Graphics instance in the event argument, as e.Graphics.

After drawing, dispose of the brush, which is good practice because GDI+ objects are connected to underlying operating system objects.

It’s easy enough to paint an opaque rectangle, but it’s almost as easy to paint a rectangle that’s transparent, so that you can see what’s behind it.

If you run the program now, you’ll see the transparent rectangle painted, as in Figure 4. However, it’s always displayed in the same place. If the SizeMode for the InkPicture is changed back to StretchImage, the transparent rectangle will not move with the image as it resizes.

The next step, then, is to set the rectangle so that it will move with the image. To do that, you need to scale and reposition the rectangle. Add the three functions to the form shown in Listing 1.

Listing 1 creates a new rectangle at any necessary instant that is scaled to the current size of the image, much the same way the transform was scaled for Ink earlier. Note that while the Renderer automatically repositioned the Ink when you set the scale, you have to do that manually for the rectangle. That is, not only do you have to scale the size of the rectangle, but you also have to scale its location.

Now, you need to change the Paint event for the VehicleImage control to use the new scaled rectangle. The new Paint event should look like this.

   Private Sub VehicleImage_Paint(ByVal sender As _      Object, ByVal e As       System.Windows.Forms.PaintEventArgs) _      Handles VehicleImage.Paint         If VehicleImage.Image Is Nothing Then         Exit Sub      End If      Dim brushHighlight As New _        SolidBrush(Color.FromArgb(80, _        Color.Magenta))      e.Graphics.FillRectangle(brushHighlight, _         ScaledHitTestRectangle)      brushHighlight.Dispose()   End Sub

This is just like the previous Paint event code, except that you use ScaledHitTestRectangle instead of HitTestRectangle, and test to make sure there’s an image that can be used for scaling. (If there’s no image, the scaling operation will fail because it can’t fetch the image size.)

Now run the program and resize the form. The transparent rectangle scales with the image.

Hit-testing Logic
Next, it’s time to add the hit testing logic. First, you need a support function. Add this function to the form:

   Private Function GetInkRectangleFromPixelRectangle( _       ByVal g As Graphics, ByVal rect As _      Rectangle)      Dim topLeft As Point = rect.Location      Dim bottomRight As New _         Point(rect.Location.X + _         rect.Size.Width, _         rect.Location.Y + _         rect.Size.Height)      VehicleImage.Renderer.PixelToInkSpace( _         g, topLeft)      VehicleImage.Renderer.PixelToInkSpace( _         g, bottomRight)      Dim newRect As Rectangle      newRect.Location = topLeft      newRect.Width = bottomRight.X - topLeft.X      newRect.Height = bottomRight.Y - topLeft.Y      Return newRect   End Function

The preceding function, GetInkRectangleFromPixelRectangle, takes a rectangle specified in pixel coordinates, and translates it to a rectangle in terms of Ink coordinates. You’ll need that capability in the hit testing logic later.

The function uses the PixelToInkSpace method of the Renderer object. The method takes a point on the control, measured in pixel coordinate space, and translates it to the equivalent point in Ink coordinate space. The method also takes the Matrix that was previously applied to the Renderer into account. So this function converts pixel coordinates to Ink coordinates and also takes scaling into account.

Now, place a button named HitTestButton on the form, and add the following logic to the button’s click event:

   ' First translate hit test rectangle to ink space   Dim InkHitTestRectangle As Rectangle   Dim g As Graphics = VehicleImage.CreateGraphics   InkHitTestRectangle = _      GetInkRectangleFromPixelRectangle(g, _      ScaledHitTestRectangle)   Dim HitStrokes As Strokes   HitStrokes =       VehicleImage.Ink.HitTest(InkHitTestRectangle, 0.01)      If HitStrokes.Count > 0 Then      MessageBox.Show("Found ink on or around door")   Else      MessageBox.Show("Little or no ink around _         door")   End If

You already have the rectangle you want to hit test, but it must be changed to Ink coordinate space. The function you added earlier is available for that. The rectangle in Ink space is then hit tested using the HitTest method of the Ink object on the VehicleImage control. This method returns any strokes whose rectangle in Ink space overlaps the hit test rectangle.

A stroke close to the door, but not quite on it, might still generate a rectangle containing a little bit of the door, so there is a numeric factor to specify how much of the stroke is within the hit test rectangle. You can use a value of 0.01, or 1 percent. That fudge factor could be adjusted, of course.

The results of the hit test are a collection of strokes that satisfy the hit test. If that collection contains any strokes, the user gets a message that the hit test was successful.

Notice that one of the implications of this hit test technique is that a circle around the door that does not touch the door will still give a positive hit test. This is typically desirable, and that’s good, because a hit test that does not work that way requires a different and more difficult technique.

The HitTest method has overloads to do hit testing inside a circle, or inside an irregularly shaped area specified by a lasso. These overloads are used in a similar fashion to the rectangular hit test above. Remember that any coordinates used in these methods must be properly scaled and translated to Ink space coordinates before being used for hit testing.

Apply the Techniques
You’ve now seen the basics of scaling and hit testing of Ink on top of an image. These basic techniques can be applied to most applications that use Ink annotations on images.

Scaling is relatively simple once you understand how to create an appropriate Matrix object to specify a scaling transform. The Renderer object does a great job of handling Ink scaling tasks, once it has been told what scale to use.

Hit testing can be considerably more complex than shown here. For example, hit testing against non-rectangular regions might be needed. But understanding the simplest case of hit testing and how scaling affects it is a good grounding for further hit testing work.

devxblackblue

About Our Editorial Process

At DevX, we’re dedicated to tech entrepreneurship. Our team closely follows industry shifts, new products, AI breakthroughs, technology trends, and funding announcements. Articles undergo thorough editing to ensure accuracy and clarity, reflecting DevX’s style and supporting entrepreneurs in the tech sphere.

See our full editorial policy.

About Our Journalist