 Standard Ink collection is incredibly easy using the Tablet PC SDK. For scenarios that require more control or non-default behavior, default Ink collection may not be the best choice. The Real Time Stylus API provides lower-level, higher performance access to stylus input, making this the API of choice for power-developers. Many Tablet PC applications can do everything they need to do using Tablet PC SDK controls and classes such as the InkCollector and InkOverlay. These objects fully automate the collection of Ink information received from the stylus (pen), rendering that Ink on the screen, and even collecting that Ink into memory using Ink objects, stroke collections, and data points. These objects are great for the most fundamental thing Tablet PCs can do: collecting digital Ink. | " |
The Real Time Stylus API is all about control and performance.
| " |
There are many scenarios where a developer might want to have more control over this process, or perhaps apply a different behavior and use the stylus for things other than collecting Ink, or maybe to collect and draw Ink in a non-standard fashion. There also are many scenarios where the performance of standard Ink collectors is not sufficient. Examples include games, 3D applications where the stylus is used to move objects around, and other real-time applications. For all these needs, the Tablet PC SDK version 1.7 introduced a new way of getting to pen data: the Real Time Stylus. The Real Time Stylus API is very simple in concept. The digitizer collects data from the hardware and passes the information on to a collection of registered objects. These objects are known as stylus plug-in objects. There are two kinds of those plug-in objects: those that implement the IStylusSyncPlugin interface, and those that implement IStylusAsyncPlugin. These two object types are very similar in functionality. For the first examples, I will focus on IStylusSyncPlugin. The Basic Implementation To create a project implementing an object that reacts to real-time stylus events, create a standard VB.NET or C# project, and include the Microsoft.Ink (also known as Tablet PC SDK) assembly/namespace in your references. The next step is the creation of an object that can be registered with the digitizer so it can be called whenever the pen triggers some action you are interested in. This object needs to implement the IStylusSyncPlugin interface, which is very simple in concept. It has a number of methods that get called whenever a certain action happens. These methods can conceptually be seen as event handlers. Keep in mind that the Real Time Stylus API is all about performance. For this reason, not all the available methods are called all the time. Instead, each plug-in object must implement a property that indicates which events you are interested in. For instance, if you are interested in all packets (data generated when the pen is used to write or draw) as well as pen up/down events, you can indicate this using the DataInterest property as shown in the following VB.Net example: Public ReadOnly Property DataInterest() _
As DataInterestMask _
Implements IStylusSyncPlugin.DataInterest
Get
Return DataInterestMask.Packets _
Or DataInterestMask.StylusDown _
Or DataInterestMask.StylusUp
End Get
End Property
Here is the C# version: public DataInterestMask DataInterest
{
get
{
return DataInterestMask.Packets |
DataInterestMask.StylusDown |
DataInterestMask.StylusUp;
}
}
A plug-in object configured in this fashion has its Packets(), StylusDown(), and StylusUp() methods called whenever those events occur. They can be used in the following fashion: Public Sub Packets(ByVal s As RealTimeStylus, _
ByVal data As PluginData.PacketsData) _
Implements IStylusSyncPlugin.Packets
Console.WriteLine("Packet: X " _
+ data(0).ToString() + _
"/ Y " + data(1).ToString())
End Sub
Public Sub StylusDown(ByVal s As RealTimeStylus, _
ByVal data As PluginData.StylusDownData) _
Implements IStylusSyncPlugin.StylusDown
Console.WriteLine("Stylus Down")
End Sub
Public Sub StylusUp(ByVal s As RealTimeStylus, _
ByVal data As PluginData.StylusUpData) _
Implements IStylusSyncPlugin.StylusUp
Console.WriteLine("Stylus Up")
End Sub
The C# version is very similar: public void Packets(RealTimeStylus sender,
PacketsData data)
{
Console.WriteLine("Packet: X "
+ data[0].ToString() +
"/ Y " + data[1].ToString());
}
public void StylusDown(RealTimeStylus sender,
StylusDownData data)
{
Console.WriteLine("Stylus Down");
}
public void StylusUp(RealTimeStylus sender,
StylusUpData data)
{
Console.WriteLine("Stylus Up");
}
Listings 1 and 2 show the complete implementation of this simple Real Time Stylus plug-in. Now that you have a simple plug-in ready to go, all you need is something to attach it to. In a very simple version, this could be a form. When that form instantiates, create a RealTimeStylus object and point it to the UI control you want to collect pen information on (in this example the entire window, but it could be an individual control, such as a textbox or a panel). Then, instantiate the plug-in object and add it to the Real Time Stylus’ SyncPluginCollection. The following snippet shows how this can be done in VB.NET: Private rts As RealTimeStylus
Public Sub New()
MyBase.New()
'This call is req. by the Win Form Designer.
InitializeComponent()
'We create the stylus
Me.rts = New RealTimeStylus(Me)
Dim plugIn As New SimpleStylus
Me.rts.SyncPluginCollection.Add(plugIn)
Me.rts.Enabled = True
End Sub
Here is the C# version: private RealTimeStylus rts;
public SimpleStylusForm()
{
InitializeComponent();
// We create the stylus
this.rts = new RealTimeStylus(this);
SimpleStylus plugIn = new SimpleStylus();
this.rts.SyncPluginCollection.Add(plugIn);
this.rts.Enabled = true;
}
Listings 3 and 4 show the complete form code for both VB and C# versions. | " |
The Real Time Stylus API can be used to filter and manipulate pen input before it gets processed or recognized.
| " |
You can now run this application and watch the output window to see stylus events being handled by the new real-time plug-in. The output you see might look similar to the following: Stylus Down
Packet: X 5133/ Y 4207
Packet: X 5133/ Y 4180
...
Packet: X 4577/ Y 3731
Packet: X 4577/ Y 3757
Stylus Up
Expect to see a lot of Packet lines, as those events fire very fast and very often during typical drawing operations. Using the Provided Data Now that you know how to write a Real Time Stylus plug-in and register it with a control that you want to use the stylus/pen on, you have knowledge of almost everything you need to use this API. There is one important tidbit of elusive information remaining: making sense of the data provided, especially in the Packets() method. There are a few key issues that force you to take a close look at the data passed to you. For one, the PacketsData object is a very bare-bones object that is an indexed list of information coming along from the digitizer. You might expect properties such as X and Y on each contained information object, but instead, the object is a very low-level list of stylus information. This is due to two simple facts. For one, performance is extremely important, so spinning up a very simple list of objects is preferable over a more complex list of objects. The second reason is that different tablet manufacturers, different system configurations, or different versions of the API could pass different data. The fact that two different tablets may produce different stylus data, even in the same version of the SDK, is a core consideration. So how exactly can you make sense of this data? The good news is that core data always appears at the same position within the list. The X position of the pen is always item number 0. Y is always item number 1. In a simple scenario, this might be all the data communicated to you, so each packet consists of two properties. This information is communicated through the data object’s PacketPropertyCount. Knowing the packet length is crucial because there is one more twist: every time the Packet() method is called and a data object is passed, you may receive more than one packet. In other words, if the pen is moved very rapidly, more than one pen position may be communicated in each pass. For instance, the data object may have a Count of four, which might mean that you have two sets of X and Y coordinates. In this case, the PacketPropertyCount is two. In a very similar scenario, the Count may be six. But this does not necessarily mean that you got three different sets of X and Y coordinates. You have to inspect the PacketPropertyCount property first. If it is still set to 2, you have received three sets of X and Y coordinates, but if it is set to 3, you have two sets of X and Y coordinates, as well as a third piece of information (pen pressure). The following code snippet shows how all this information can be used in a modified version of the Packet() method: Public Sub Packets(ByVal s As RealTimeStylus, _
ByVal data As PluginData.PacketsData) _
Implements IStylusSyncPlugin.Packets
Console.WriteLine("Number of Packets: " + _
data.Count.ToString())
Console.WriteLine("Properties per packet: " _
+ data.PacketPropertyCount.ToString())
Dim pc As Integer
For pc = 0 To
data.Count - data.PacketPropertyCount _
Step data.PacketPropertyCount
Console.WriteLine("Packet: X " _
+ data(pc).ToString() + _
"/ Y " + data(pc + 1).ToString())
If data.PacketPropertyCount > 2 Then
Console.WriteLine("Pressure: " _
+ data(pc + 2).ToString())
End If
Next
End Sub
(Because the C# version of the code is very similar, and in the interest of space, I will not list it from here on. You can download all the samples used in this article in both C# and VB.NET.) In this example, you run a loop over all data received, skipping forward by the number of properties that make up each packet. Here is an interesting thing to try: run this example and use your mouse as the pen. Simply click the mouse button, move around a bit, and let go of the button. You will see that the plug-in shows a lot of X and Y coordinates. The length of each packet will be 2. No information other than X/Y is provided. Then, try this with a real stylus (pen). You will see that the packet property count increases and all of a sudden, pressure information is available as well. Note that you have to account for this in code with an IF statement. There is one final twist when it comes to handling packet data. You may have noticed that the coordinates you received were somewhat unusual. My Tablet PC has a screen resolution of 1400 x 1050 pixels. When I use the pen on that very tablet, I receive X and Y coordinates that could be something like 4577/3575. The reason for this is that today’s digitizers operate at much higher resolutions than current displays. In many (but not all) scenarios, it is important to map the resolution of the digitizer to the resolution of the screen. For instance, you may want to draw a dot where the pen touched the screen. For that, you need to know what position 4577/3575 maps to on the display. | " |
The Real Time Stylus API gives you access to raw pen data, rather than just the Ink input resulting from pen operations.
| " |
This task can be achieved relatively easily as long as you have access to a GDI+ Graphics object, which provides important information such as horizontal and vertical screen dpi. In your current version, you do not have access to such an object, but you can easily create one, as long as you have access to the control the stylus plug-in is used with. I modified the code so that each plug-in can be handed a reference to the assigned control (see Listing 5). The following is the sub-section of the modified code that is responsible for the translation: Dim g As Graphics = _
Me.attachedControl.CreateGraphics()
Dim iX As Integer
Dim iY As Integer
iX = g.DpiX * data(packetCounter) / 2540
iY = g.DpiY * data(packetCounter + 1) / 2540
The important aspect is that you use the DPI assigned to the X and Y axis of the current display system and then multiply it by the X and Y coordinates passed to you, divided by 2540 (this value remains constant). This takes care of all the hard parts, and you can now start having some fun with this technology. Rendering Collected Information One of the simplest things to do with the information you get from the plug-in is custom rendering of the received information. The following example draws a small circle at the pen coordinates you received. Pressure information is used to determine the radius of the circle. Public Sub Packets(ByVal s As RealTimeStylus, _
ByVal data As PluginData.PacketsData) _
Implements IStylusSyncPlugin.Packets
Dim g As Graphics = _
Me.attachedControl.CreateGraphics()
Dim packetCounter As Integer
For packetCounter = 0 To _
data.Count - data.PacketPropertyCount _
Step data.PacketPropertyCount
Dim iX As Integer
Dim iY As Integer
Dim iPressure As Integer = 10
iX = g.DpiX * data(packetCounter) / 2540
iY = g.DpiY * data(packetCounter+1) / 2540
If data.PacketPropertyCount > 2 Then
iPressure = data(packetCounter+2) / 20
End If
g.DrawEllipse(Pens.Red, _
iX - iPressure, iY - iPressure, _
iPressure * 2, iPressure * 2)
Next
End Sub
The result of this can be seen in Figure 1. At this point, you have created a lot of what a normal InkCollector provides, and have replaced it with your own functionality. The advantage of this approach is that you have full control and can thus create completely different rendering of the Ink “written” or “drawn” with the stylus. 
Figure 1: This is a custom rendering of Real Time Stylus input.This can be used for much more sophisticated effects than drawing circles of different sizes. Charles Petzold has a nice example, where the “Ink” automatically has a drop-shadow (at http://www.charlespetzold.com/etc/RealTimeDropShadow/). A similar example adds a glow effect to the Ink. Larry O’Brien has an interesting example that uses pen information to draw Ink in 3D space, using pressure or elapsed time as the third dimension (at http://www.devx.com/TabletPC/Article/21943). Limits are set only by your imagination. Another option is to go beyond drawing “Ink” wherever the pen touches the display. The example in Listing 6 shows a simple kaleidoscope, where the drawing is automatically mirrored on three axes. The code change in this example is that you now store references to four controls rather than just one. The first is the one you actually draw on. It is a panel on a form, rather than the form itself, as in the previous examples. The other three controls are simply used to draw each drawing point at a mirrored position. Figure 2 shows the result. 
Figure 2: This is my attempt at drawing a turtle. The top/left quarter is drawn. The other three quarters are mirrored in real time.Modifying Collected Information One of the great advantages of the Real Time Stylus is that you have full control over the data received from the API. This includes the ability to modify data as it is being received. A simple example that demonstrates that capability is the restriction of Inking to a specific area within a control. To keep this example simple, use a rectangle as the allowed Ink area, although the same concept can be used for any shape. Figure 3 shows the example at work. 
Figure 3: This shows real time input limited to an arbitrarily defined rectangle within the window.Listing 7 shows the implementation of that functionality. The main difference between this example and the version that collected data and drew circles at the pen position is the ModifyData() method. This method is called from the Packet() method before any other processing happens. It compared the X and Y coordinates received from the stylus plug-in and checks whether they fall within the boundaries of a rectangle you define in the constructor of the plug-in (that’s what I did to keep things simple for this example-the confining rectangle could be created or modified anywhere). The tricky part is that in most scenarios, the Inkable area is more easily defined in screen coordinates. You first need to map the X and Y coordinates from digitizer resolution to screen resolution, then check whether they are in a valid area; if not, move them into a valid area and translate them back to digitizer resolution. Luckily, this sounds more difficult than it really is, as you can see in Listing 7. Note that you didn’t just translate the values before you drew on the screen, but you changed the data in the actual packet data object. This means that other code (plug-ins) using the same data gets the modified version. Collecting Ink If you have followed along with all the examples in this article, you may have noticed that they all work fine, but all the rendered output is somewhat-shall we say-fragile. Although all the rendering works well, it is not persisted. If you draw on any of the forms, and then minimize and bring them back again, the drawings are gone. That’s because so far, you only drew on the form whenever the pen caused events, but you do not listen for any re-draw events that the Windows operating system may indicate. In particular, each window raises a Paint event whenever its contents need to be re-drawn, and because you completely ignore that event on all the examples, they all end up with blank forms whenever they are refreshed. To solve this problem, you need to add one very fundamental step. You need to store the information you receive from the pen, so you can use it to re-draw all the Ink whenever it needs to be. There are a number of ways to accomplish this task. For instance, you could simply take all the data points you received and store them in an array or collection. You could then use the Paint event to iterate over all the collected points and draw them again. You can also take advantage of Ink storage provided by the Tablet PC SDK. This has the advantage that you end up with a standard Ink structure in an Ink object, including all the related functionality, such as Ink serialization and Ink recognition. In fact, there are even standard objects that help us to render Ink that is stored in an Ink object. Ink data stored within Ink objects is maintained in a collection of strokes. Rather than simply collecting all data points in a single large collection, Ink objects store a series of points on a stroke-by-stroke basis. This means that you need to determine when a stroke is started, which happens when the pen is put down on the digitizer. From that event on, you need to store all points (including the touch-down point of the StylusDown event) in a temporary collection of points until the StylusUp event occurs. You then take all the points and move them from the temporary collection into a new stroke on the Ink object. The tasks required to achieve this are straightforward and can be seen in Listing 8. The only part that requires extra explanation is the call to the CreateStroke() method, in particular the second parameter. This parameter is a collection of properties associated with the stroke. The easiest way to provide this parameter is to use the default set of parameters associated with the Real Time Stylus. At this point, you have collected the Ink but you have not displayed it anywhere. One easy way to render the collected Ink is to use a Renderer object, which is also provided by the Tablet PC SDK. You can simply call this helper object in the Paint event of whatever control you want to draw on: Protected Overrides Sub OnPaint(ByVal e As _
System.Windows.Forms.PaintEventArgs)
Dim rend As New Microsoft.Ink.Renderer
rend.Draw(e.Graphics, Me.stylus.Ink.Strokes)
End Sub
There are two problems that make this approach incomplete. For one, the Ink object’s Strokes collection only contains each stroke’s data once it has been completed. In other words, the information needed to render strokes is not available while the pen is still touching the display. It only becomes available when the pen is lifted up. This does not reflect a real-world writing experience accurately, as Ink can not be seen while writing. It only becomes visible when the writing process has been completed. The second problem is that the Paint event will not fire unless Windows detects a good reason to refresh the display, which is only the case if the window gets minimized and restored, or is partially or fully obstructed and brought forward again. You could force the window into a refresh by invalidating its content. However, this is not a good option performance-wise. These are clearly problems you need to solve. You have already solved the previous problem, which is Ink persistence and Ink re-rendering when the window or control refreshes. So this clearly is useful, and all you need now is a way to dynamically render the Ink while it is still being drawn. You could manually draw the same way you did in earlier examples, but there is an even simpler way. Microsoft provides a standard Real Time Stylus plug-in called the DynamicRenderer, which can be registered in addition to your custom plug-in. Here is the code that initializes both plug-ins: Public Sub New()
MyBase.New()
'This call is req. by the Win Form Designer.
InitializeComponent()
'We create the stylus
Me.rts = New RealTimeStylus(Me)
' We create the plug-in
Me.stylus = New InkStylus(Me, Me.rts)
Me.rts.SyncPluginCollection.Add(Me.stylus)
' We create a default dynamic renderer
Me.myDynamicRenderer = New DynamicRenderer(Me)
Me.rts.SyncPluginCollection.Add( _
Me.myDynamicRenderer)
' We enable everything
Me.myDynamicRenderer.Enabled = True
Me.rts.Enabled = True
End Sub
This already works pretty well, but you can enhance the code a little further by making sure the dynamic renderer never interferes with paint events. You can do this by calling the dynamic renderer’s Refresh() method in the paint events. The following snippet shows the completed Paint event: Protected Overrides Sub OnPaint(ByVal e As _
System.Windows.Forms.PaintEventArgs)
' Make sure the new Ink is refreshed properly
Me.myDynamicRenderer.Refresh()
' Render Ink in the Ink object
Me.myRenderer.Draw(e.Graphics, _
Me.stylus.Ink.Strokes)
End Sub
Listing 9 shows all the code required by the form that uses this combination of plug-ins. At this point, people could point out that you have spent a lot of effort on re-creating what the default InkCollector object would have provided (or in fact, even a little less). This is true for this example, but keep in mind that Real Time Stylus technology affords a lot more control than the default objects. For instance, you could combine the last example with the input-area restriction shown earlier. It is also possible to apply logic to the data that gets collected. For instance, Tablet PCs collect a lot of data very quickly and at a high resolution. This is great in most scenarios, but the sheer amount of data can also be overkill. Using Real Time Stylus plug-ins, you can decide to collect data at a lower rate by dropping packets that are very similar to previously collected packets, or you can apply other filters. Asynchronous Collection All the examples so far operate synchronously. This means that data is collected by the digitizer and then it is passed to the plug-ins that process it; once that is completed, more data is collected and then it goes back to the plug-ins again. This loop is continued as long as more data is available. It is the ultimate goal of the Real Time Stylus API to process data as efficiently as possible in real time. This is not always feasible or even required. Consider the most recent example that collected Ink in an Ink object. Was it really necessary to store strokes in real time? No. As long as you are able to dynamically render new Ink in real time, it is not overly critical how quickly Ink gets stored in the Ink object, as long as it is available when the Paint event fires the next time. So a slight delay in Ink collection is acceptable, and any performance gain in the real-time rendering improves the user’s experience drastically. This is where asynchronous plug-ins come into play. They are almost identical to the plug-ins you have seen so far. The only difference is that they implement the IStylusAsyncPlugin interface rather than the IStylusSyncPlugin interface. Both interfaces are identical and only differ by name. All implementations you have created thus far work in identical fashion as asynchronous plug-ins if you simply search and replace the interface name. The only other change is that the plug-in needs to be added to the AsyncPluginCollection collection, rather than the SyncPluginCollection you added it to before. Give it a try and run the same example in asynchronous mode again. You will notice that it is functionally identical, but you will probably have a smoother experience, as all the processing power is available for Ink rendering, and the process of storing Ink is handled asynchronously when the system isn’t busy. Note that this does not mean that the asynchronous plug-in will run minutes later. The delay will be minimal. But it means that the main processing thread is not blocked by non time-critical code. Conclusion The Real Time Stylus API is a great addition to the Tablet PC SDK. It provides more raw power, control, and flexibility than any of the default Ink collection controls and classes for a relatively small increase in complexity. This doesn’t mean that the Real Time Stylus API should be used as a replacement for other options in all scenarios, but whenever you desire this degree of freedom, the RTS API is a great way to go! Markus Egger | 
