Browse DevX
Sign up for e-mail newsletters from DevX


Scaling and Hit-Testing in Ink Applications : Page 3

In addition to text recognition and similar capabilities, Ink can also be useful for image annotation and markup, such as in medical and insurance applications, where marking up images can be a valuable and critical form of input.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

Now suppose you want to detect whether or not the Ink annotations include any Ink in a certain area. For this example, I'll use the door of the vehicle.

Normally, hit testing is done within a rectangle, which is represented by the Rectangle class in System.Drawing. Rectangle is just a structure that contains the position and size of the rectangle.

When working on hit testing, it helps a lot if you can see the rectangle that's being used. That usually means painting the rectangle in the Paint event of the form or control on which the rectangle needs to appear.

It's easy enough to paint an opaque rectangle, but it's almost as easy to paint a rectangle that's transparent, so that you can see what's behind it. Let's start the hit test example by placing a transparent rectangle on the vehicle door.

Change the SizeMode of the InkPicture control back to Normal. Then add a member variable to the InkScaleForm with this line of code:

Private HitTestRectangle As New _ Rectangle(230, 200, 160, 120)

The size and position specified in this line work well for the door in the example image I used, but you might want to adjust those numbers for a different position in your image.

Next, create an event handler for the Paint event of the VehicleImage control. In the Paint event, place the following code:

Dim brushHighlight As New _ SolidBrush(Color.FromArgb(80, _ Color.Magenta)) e.Graphics.FillRectangle(brushHighlight, _ HitTestRectangle) brushHighlight.Dispose()

This demonstrates the use of GDI+ in Windows Forms to do some drawing directly to a control. First, you create a brush, which is the object used to fill an area. In this case, the type of brush is a SolidBrush, but that's a little misleading. The color you specify using a special method of the Color class called FromArgb. This rather obscurely named method creates a color with transparency. The first argument in the method is a number from 0 to 255 that specifies how much opacity is desired, with 0 being none and 255 being total. I'm using 80 in the example above, which is a typical setting that paints some color but still shows much of the background.

The brush is then used to paint the area specified by the HitTestRectangle. That's done with the FillRectangle method, which is available on an instance of the Graphics class. The Graphics class represents a rectangular area on which to draw. Paint events always expose a Graphics instance in the event argument, as e.Graphics.

After drawing, dispose of the brush, which is good practice because GDI+ objects are connected to underlying operating system objects.

It's easy enough to paint an opaque rectangle, but it's almost as easy to paint a rectangle that's transparent, so that you can see what's behind it.
If you run the program now, you'll see the transparent rectangle painted, as in Figure 4. However, it's always displayed in the same place. If the SizeMode for the InkPicture is changed back to StretchImage, the transparent rectangle will not move with the image as it resizes.

The next step, then, is to set the rectangle so that it will move with the image. To do that, you need to scale and reposition the rectangle. Add the three functions to the form shown in Listing 1.

Listing 1 creates a new rectangle at any necessary instant that is scaled to the current size of the image, much the same way the transform was scaled for Ink earlier. Note that while the Renderer automatically repositioned the Ink when you set the scale, you have to do that manually for the rectangle. That is, not only do you have to scale the size of the rectangle, but you also have to scale its location.

Now, you need to change the Paint event for the VehicleImage control to use the new scaled rectangle. The new Paint event should look like this.

Private Sub VehicleImage_Paint(ByVal sender As _ Object, ByVal e As System.Windows.Forms.PaintEventArgs) _ Handles VehicleImage.Paint If VehicleImage.Image Is Nothing Then Exit Sub End If Dim brushHighlight As New _ SolidBrush(Color.FromArgb(80, _ Color.Magenta)) e.Graphics.FillRectangle(brushHighlight, _ ScaledHitTestRectangle) brushHighlight.Dispose() End Sub

This is just like the previous Paint event code, except that you use ScaledHitTestRectangle instead of HitTestRectangle, and test to make sure there's an image that can be used for scaling. (If there's no image, the scaling operation will fail because it can't fetch the image size.)

Now run the program and resize the form. The transparent rectangle scales with the image.

Hit-testing Logic
Next, it's time to add the hit testing logic. First, you need a support function. Add this function to the form:

Private Function GetInkRectangleFromPixelRectangle( _ ByVal g As Graphics, ByVal rect As _ Rectangle) Dim topLeft As Point = rect.Location Dim bottomRight As New _ Point(rect.Location.X + _ rect.Size.Width, _ rect.Location.Y + _ rect.Size.Height) VehicleImage.Renderer.PixelToInkSpace( _ g, topLeft) VehicleImage.Renderer.PixelToInkSpace( _ g, bottomRight) Dim newRect As Rectangle newRect.Location = topLeft newRect.Width = bottomRight.X - topLeft.X newRect.Height = bottomRight.Y - topLeft.Y Return newRect End Function

The preceding function, GetInkRectangleFromPixelRectangle, takes a rectangle specified in pixel coordinates, and translates it to a rectangle in terms of Ink coordinates. You'll need that capability in the hit testing logic later.

The function uses the PixelToInkSpace method of the Renderer object. The method takes a point on the control, measured in pixel coordinate space, and translates it to the equivalent point in Ink coordinate space. The method also takes the Matrix that was previously applied to the Renderer into account. So this function converts pixel coordinates to Ink coordinates and also takes scaling into account.

Now, place a button named HitTestButton on the form, and add the following logic to the button's click event:

' First translate hit test rectangle to ink space Dim InkHitTestRectangle As Rectangle Dim g As Graphics = VehicleImage.CreateGraphics InkHitTestRectangle = _ GetInkRectangleFromPixelRectangle(g, _ ScaledHitTestRectangle) Dim HitStrokes As Strokes HitStrokes = VehicleImage.Ink.HitTest(InkHitTestRectangle, 0.01) If HitStrokes.Count > 0 Then MessageBox.Show("Found ink on or around door") Else MessageBox.Show("Little or no ink around _ door") End If

You already have the rectangle you want to hit test, but it must be changed to Ink coordinate space. The function you added earlier is available for that. The rectangle in Ink space is then hit tested using the HitTest method of the Ink object on the VehicleImage control. This method returns any strokes whose rectangle in Ink space overlaps the hit test rectangle.

A stroke close to the door, but not quite on it, might still generate a rectangle containing a little bit of the door, so there is a numeric factor to specify how much of the stroke is within the hit test rectangle. You can use a value of 0.01, or 1 percent. That fudge factor could be adjusted, of course.

The results of the hit test are a collection of strokes that satisfy the hit test. If that collection contains any strokes, the user gets a message that the hit test was successful.

Notice that one of the implications of this hit test technique is that a circle around the door that does not touch the door will still give a positive hit test. This is typically desirable, and that's good, because a hit test that does not work that way requires a different and more difficult technique.

The HitTest method has overloads to do hit testing inside a circle, or inside an irregularly shaped area specified by a lasso. These overloads are used in a similar fashion to the rectangular hit test above. Remember that any coordinates used in these methods must be properly scaled and translated to Ink space coordinates before being used for hit testing.

Apply the Techniques
You've now seen the basics of scaling and hit testing of Ink on top of an image. These basic techniques can be applied to most applications that use Ink annotations on images.

Scaling is relatively simple once you understand how to create an appropriate Matrix object to specify a scaling transform. The Renderer object does a great job of handling Ink scaling tasks, once it has been told what scale to use.

Hit testing can be considerably more complex than shown here. For example, hit testing against non-rectangular regions might be needed. But understanding the simplest case of hit testing and how scaling affects it is a good grounding for further hit testing work.

Billy Hollis is author of the first book ever published on VB.NET as well as several other books. He speaks at many technical conferences, and has his own consulting practice in Nashville, TN.
Thanks for your registration, follow us on our social networks to keep up-to-date