Incremental Magic

The Magic: Shape "Recognition"

Tue, 12 Sep 2006 04:54:46 GMT

This is the post where I spill some of my magician's tricks.

Graphite does not use Vista's new InkAnalysis APIs. In fact, it doesn't really do true shape recognition. The truth is almost embarassingly simple:

public float ProbabilityStrokeIsShape(Stroke stroke)
{
    Point start = Utils.EndpointOfStroke(stroke, EndPoint.Start);
    Point end = Utils.EndpointOfStroke(stroke, EndPoint.End);
    Rectangle bounds = stroke.GetBoundingBox();

float diag = Utils.Distance(new Point(0, 0), new Point(bounds.Width, bounds.Height));
float dist = Utils.Distance(start, end);

return (1.0f - (dist / diag));
} // (adapted from the file Utilities\ShapeRecognizer.cs)

I'm taking a stroke, and calculating one minus the ratio of the distance between the stroke's start and end, and the length of the stroke's bounding box. That's the core of it - the el-cheapo heuristic that motivated two years of development work in my spare time. The key idea is that a typical shape's endpoints are near their starting points. This is true of circles, squares, triangles, and pretty much any shape that was scrawled out in a single freehand stroke. Contrast this with a line that connects two "boxes" - lines are drawn to connect two far-flung endpoints, so the starting points and ending points pretty much define the bounding box of the stroke. Thus, if dist/diag is small, the stroke is a shape, and if dist/diag is large, it's a line.

Then, there's connectors. If you draw a stroke that ends near a box, the way I calculate whether your stroke terminates inside that box is as follows: I create a small rectangle surrounding the box, and I see if that small rectangle intersects with the bounding rect of my "box" stroke. If there's any intersection, then the stroke might be a line that terminates inside my box/ Then, I take the bounding rect of the box and the bounding rect of the stroke, and I compute their intersection. I then calculate the area of that intersection divided by the area of the bounding box of the stroke. This gives me a rough idea of what percentage of the stroke's bounds overlap with my pre-existing box.

I then do the exact same calculation on the other endpoint of the stroke.

Truthfully, though, when I'm checking if your newly drawn stroke is a connecting line, I don't actually look at the endpoints. Instead, I look for a "smart" endpoint. Look at the connecting line in the picture, above. The "endpoint" of the stroke was one of the edges of the arrowhead, which is not the behavior I want. So, I use the stroke's PolylineCusps property to get a collection of all the cusps. I then look at each cusp, and I find the distance between each cusp and the starting point of the stroke. I then try to find a cust that is as early in the stroke as possible, and yet is as far away from the starting point as possible. If you look at the five blue numbers in the picture, you'll see that cusps 2, 3, 4, and 5 are the possible candidates. 3 is too close to the start, and 4 and 5 are too late in the order, so my heuristic settles on Cusp 2. The net result is that arrowheads work the way you'd expect them to.

Think you can do better? Why not download the source and give it a shot?

A few notes...

Tue, 12 Sep 2006 04:01:04 GMT

I forgot to mention that Graphite will only run on Windows XP Tablet PC Edition, or on Windows Vista. You'll need the .NET Framework 2.0 as well.

John Tokash says that the build that's available doesn't work properly on his Eo UMPC. Alas, I have no UMPC to test or develop on. UMPC devs, feel free to download the source and see what you can find!

Graphite source released!

Tue, 12 Sep 2006 03:20:06 GMT

You can download Graphite's full source code at the new Graphite Project homepage, courtesy of my friend (and Graphite contributor) Gordon McNaughton. The code is made available under a BSD-style license, so you can use it for any purpose you want, as long as you attribute the portions I wrote to me.

Non-developers can join in the fun, too. The site also has an already-compiled copy of the Graphite executable to play with.

So what are you waiting for? Go grab it! And if you do anything interesting with the source, do leave a comment or drop me an email.

Meanwhile, I'm off to find a new side-project...

Ready for release!

Sat, 09 Sep 2006 23:38:06 GMT

It's been over four months since I decided to release the source code for Graphite. Today, I finally found the time to prepare it for release. I removed some utility files that weren't in use any more, I commented-out the Search Dialog (which I never got around to finishing), I wrote a "redistribution license" file (basically the BSD license, with some attribution notes) and I replaced my various "borrowed" icons with el-cheapo, hand-drawn icons that I whipped up using Paint.NET. The Tango icons remain, thanks to their open-source license.

Behold my artistic talents.

Now, all that I need to do is find a place to host the source files, and then I'll put up a link.

Preparing the Graphite source for release

Sun, 07 May 2006 02:15:55 GMT

I've decided to publicly release the current Graphite source code, in the hopes that someone with more time on their hands might turn it into a finished product. I'm planning to use a BSD-style license, so anyone can take the code and freely adapt it, as long as they give me (and Gordon) credit for the portions they take.

I have three things to do before I can release it:

1) I want to thoroughly comment the codebase before I release it

2) I need to replace all the icons and images with new images that I can distribute without violating anyones copyrights

3) I need a place to to put the .zip file containing the project and source. It's super-small (<200K), so this shouldn't be a big issue.

I don't have a timeline for when I'll be done with this; hopefully "soon".

Inkanalysis APIs

Fri, 05 May 2006 03:57:02 GMT

Windows Vista introduces a new set of InkAnalysis APIs that can do shape recognition and detect box-and-line association. Dagnabbit, I wish I'd had those two years ago when I started working on Graphite. When Vista comes out (or at least when Beta 2 is released), I may have to rewrite Graphite from scratch to take these new tools into account.

In the meantime, work has been keeping me *very* busy lately. Gordon has been keeping the flame alive with a few tweaks and bugfixes, but development has basically ground to a halt.

Avalon? Sure, why not?

Thu, 16 Mar 2006 07:15:10 GMT

Someone who appears to be Shawn Van Ness left the following comment:

Man, you should use a UI platform with a retained-mode graphics stack, like Avalon. Let the MIL compositing layer will handle all that dirty-region updating for you.

InkCanvas even supports selection and editing of arbitrary child elements (not just ink strokes) so you might be able to save some work there too...

</ShamelessPlug>

Shameless plug or no, Shawn, you're right on the money. The next time I go to completely refactor Graphite, I'm going to want to port it to Avalon while I'm at it.

From what I've read about Avalon/WPF's new Ink object model, it's a really nice, streamlined refinement over the existing Tablet PC 1.7 APIs. I particularly like that Strokes objects stand alone, without their the tricky Ink object identity/lifetime context stuff that caused me so much trouble in 2004 when I was just getting started working on Graphite.

And when I go to implement the PaperBox UI, with touchable controls that cause large, finger-sized tool overlays to appear and disappear, you'd better believe that I'm going to want something richer than WinForms at my disposal.

But not right this second.

I've played the "beta API game" with Graphite once already, when VS2005 came out. Beta IDE, beta frameworks, beta language... it all sounded rather risky, but eventually the seductive lure of generics won be over. I also got burned for a while when the Tablet 1.7 APIs didn't play nice with 2.0 System.Data DLL, breaking cut/copy/paste and various internal data operations that I was dependent on. Plus, ever since my day job at Microsoft took a detour into Windows Vista, I've had more than my fill of beta environments for a little while.

And even once I am game for it, switching platforms will require doing a lot of investigation into the Avalon/WPF libraries; hello-world projects, fishing through incomplete documentation, and whatnot. And when the due diligence is done, I'll still have 6000 lines of existing C# code to work my way through adapting and modifying. That sounds an awful lot like work.

Right now, the code is in a state where I can make measurable and satisfying improvements in as little time as half-an-hour. That's an ideal state for a hobby project to be in.

Because ultimately, Graphite/PaperBox is a hobby project for me.

It was bound to happen.

Thu, 16 Mar 2006 02:23:15 GMT

I *knew* I couldn't have been the first person to decide to use the name "Graphite" in connection with a cool drawing application. These guys appear to have a head start on me. :-)

Looks like I'll be officially changing my project's name to Paperbox.

Coding my way out of a wet paper bag

Sun, 12 Mar 2006 01:11:32 GMT

Until about 20 minutes ago, if you were doing something interactive with the graph (drawing, resizing something, whatever), the only reason that the UI repaints is because your MVC Control was telling the View to Invalidate(). This meant that there were redundant Invalidate() calls all over the place. I decided that it would make for a nice afternoon's project to remove these external Invalidate() calls, and instead have the Graph object itself kick up change notifications whenever its internals change. The GraphPanel would listen for these notifications, and respond by calling Invalidate().

This was all well and good until I got to the Move Selection tool. This tool works by capturing new "packets" (InkCollector-speak for "mouse-dragged events") and keeping track of the x/y delta between the first packet seen and the last packet seen. It then registers itself as a "pre-paint handler" for the Graph; that is to say, the next time the Graph is asked to paint itself, the Move Selection Tool gets to so some work first. It then calls Invalidate().

Invalidate() causes a repaint event to be enqueued in the thread's message queue, if there isn't one enqueued already. In between me and that message, there might be more queued NewPackets events; the tool handles them by just updating the x/y delta. Notice that no real actually-change-the-internals-of-the-Graph work has been done yet. The idea is that since we'll be getting spammed with Mouse Moved events, and even if we update the work

When the Invalidate() hits, we enter the GraphPanel's paint routine, which sets up the appropriate coordinate transformations and invokes the Graph's paint routine. Right then, the Move Selection Tool gets the PrePaint callback, and it uses the opportunity to actually apply the x-y deltas. Then, the PrePaint callback exits, the graph paints itself into the GraphPanel, at which point the user's changes are visibly reflected.

Except that I'm not calling Invalidate() unless and until the Graph changes. So I'm never invoking the PrePaint() handler so the Graph itself doesn't change. Since the Graph doesn't change, there's no change notification to prompt an Invalidate().

Whoops.

Workaround #1 involves putting the Invalidate() call back in, but that doesn't sit right somehow.

Workaround #2 involves revising the pre-paint handler mechanism; it'll probably turn into "SetDeferredChange(ChangeObject)", which would kick up a change notification and then call the ChangeObject

Workaround #3 involves revisiting the whole threading model, which I just don’t feel up to right now.

PaperBox Concept Sketch

Sat, 11 Mar 2006 04:16:01 GMT

I've added two new pictures to the gallery. The first shows the current Graphite UI running in the UMPC Display Emulator. As you can see, there isn't a lot of space to work with.

The second is a mockup of the PaperBox UX. I'm no artist, so squint for a second and pretend that the orange and blue semicircles have bevelling and texturing that clearly demarks them as grabbable widgets. By including panning widgets, by collapsing the Document Title area, and by putting the different tools right near your left thumb, this new UI attempts to put all of Graphite's tools in a place that's more accessible to a touch-screen user. However, this new UI takes up much less screen real estate.

See the 4-way switch on the left side? I plan to put that to work for zoom control. Your left thumb will be near it all the time, so you can quickly zoom in on a box (for editing its internals) and then quickly zoom back out to see the whole document (for adding new boxes and redoing their layouts.)

Why Graphite won't be satisfying on an Origami UMPC

Fri, 10 Mar 2006 08:44:50 GMT

John Tokash wants to run Graphite on an Origami UMPC. John, you're welcome to try, but I don't think you'll be satisfied with the experience.

First of all, touchscreen and EMR styli are very different beasts. Graphite's UI is optimized for use with a pen with a barrel button and an eraser on the flip-side. This gives you inking, erasing, selection (right-click to lasso), selection-movement, selection-resizing, deletion (via gestures); all without having to move your pen away from the content. Graphite's toolbar, for those few occasions when you need it, is tucked away on the right side of the screen, occluded by a typical right-handed user's hand when the focus is on the content.

On a touchscreen, you have only one flavor of click at your immediate disposal. No eraser, no right-click, and no hover. This means that any selection, manipulation, or anything else that's not drawing, requires a trip to the toolbar or some other piece of auxiliary UI. Here, Graphite's UI causes more problems. Putting the toolbar in an occluded region of the screen is fine when you rarely need to access it, but if you need it frequently, your hand has to constantly leave the screen so you can see - and thus effectively target - the toolbar.

Also, the combined window titlebar and Document Title Field in Graphite are 100 pixels high, which isn't a big deal when you have 1024 pixels of vertical screen real estate (as on my Thinkpad), but that same 100 pixels suddenly seems an exorbitant price to pay when you only have 480 pixels to work with. (And the Windows Taskbar eats another 30 pixels.) Because there's so much less space in between, it will be difficult to put a substantive amount of boxes and lines on the screen while still leaving room for interesting details inside the box.

Put it all together, and you'll find yourself frustrated. Once the novelty wears off, you'll probably go back to just doing one-off sketches in Windows Journal or OneNote.

Fortunately, I have a solution in mind.

My working name for it is Paperbox.

(More to come this weekend...)

Wed, 08 Mar 2006 04:31:23 GMT

"because ink rendering can be surprisingly - even dismayingly - slow."

Expecially when you're forcing two repaints of the visible document for every stroke that's drawn. (That would be once on pen-up, and then again when the background thread reports that it's done classifying that stroke and incorporating it into the data model.)

Excuse me while I make a one line code change.

...

Okay, I'm back. And that definitely improved matters; it takes a lot more stuff on the page before the performance-cliff shows up.

Now, on to the substance of my gripe. Jarrett & Su's book Building Tablet PC Applications says that the Tablet PC Team benchmarked a 1.4GHz P4 reference system as being able to render 600 typical strokes per second. That's not a whole heck of a lot, when you consider that "hello world" is ten strokes in my handwriting. From that, let's assume that 600 strokes is approximately 120 short words. Then, consider that a second is an achingly long time in the context of an interactive pen-based application. Since most painting occurs in the main message loop, that's a long time when your app is unresponsive to further input.

And while some may quibble that a P4 is cycle-by-cycle slower than the Pentium M processors in most modern Tablet PCs, I'll note that when running on batteries, my sprightly 1.5GHz Thinkpad likes to slow to 500MHz, or even to a a 1997-esque 300MHz.

Sure, there are tricks you can do. Right now, I'm changing the ThreadedGraph class so that if it's falling behind on drawing, it doesn't commission an invalidate until it finishes catching up. Since drawing is expensive, it's worth trying to do it as little as possible.

Origami?

Wed, 08 Mar 2006 03:30:46 GMT

John Tokash was my manager at my first coding gig. He's got a great set of thoughts on device form-factors and on the current rumors and fragments about the impending MS Thing codenamed "Origami". I wish I knew more about it. Of course, since I work for Microsoft, if I *did* know something about it, I wouldn't be able to say anything until the official announcements.

Still, this much is clear: I want one.

Also: If it will run Graphite, I really really want one.

Chances are, the Origami device will be a fairly low-power device with a (physically) small battery. Right now, Graphite sucks CPU - and thus batterly life - like a ravenous beast. I need to find ways to make my recognition algorithms less expensive, even if it means being less accurate; I also need to find ways to reduce the amount of time I spend repainting the graph, because ink rendering can be surprisingly - even dismayingly - slow.

February 4th Update

Sun, 05 Feb 2006 01:27:36 GMT

Three hours' hacking yielded two new screenshots in the gallery.

- When you select a group of elements, you can now resize them by grabbing on the ugly orange grab handles.

- There's now a floating search dialog with an InkEdit control; type or write in your search text, and it will find the next element that matches your search. Keep tapping "Find" to cycle through all the different candidates that were found.

The code for both of these was pretty hacked-on, but still - two features in three hours? I'm very pleased with the results of my last significant cleanup pass on the code.

Of course, I'm only working on these features because I'm discouraged about the amount of work it will take to get Visio export working. I'll have to really richly understand Visio's object model and worldview before I'll be ablt to grok its XML schema.

January Update

Wed, 25 Jan 2006 04:58:34 GMT

Nothing but rain outside. It's a good time of year to stay indoors and hack on things.

Recent changes:

- Copy/paste between Graphite instances and other applications

- Recognition is offloaded onto one or more worker threads, which helps keep ink input smooth and responsive, even when you're running on battery and your processor throttles itself back to 500MHz

- Commented and cleaned up much of the code

Like I said, a good time of year for hacking.

Happy Thanksgiving

Fri, 25 Nov 2005 00:15:30 GMT

About a third of the work that I do for Graphite happens in airports and on airplanes. Tedious waits, no internet connectivity, cramped spaces, and poor lighting? Bah! Five hour battery life (I *heart* Thinkpad), bright screen, and the full MSDN documentation.

Latest changes:

- Proper cursors for the current tool/action

- Drop shadows have been removed because they take a long time to draw, replaced by simple outlines. Drawing speed is my #1 performance issue; no amount of clever threading will mask the fact that complicated diagrams will take upwards of 500ms to draw. I met a fellow on the Avalon On Tablets team at a training class last week, and grilled him about the issue, and he remarked that it was much faster to draw an entire Strokes collection at once than it was to draw each Stroke individually, and anecdotal evidence would seem to support that. With that in mind, my next thing to try will be to take the strokes from all my elements and put them into one uber-collection, and draw the uber-collection all at once.

- In the meantime: Elements that are outside the window's viewable area are not drawn. Such a simple thing, with dramatic performance improvements when dealing with sprawling, multi-screen documents.

- There are now two different mechanisms for deforming Connectors - one uses polar coordinates to rotate and scale, while the other independently scales the x- and y-axis. The decision about which to use is made on a connector-by-connector basis, according to some appalingly crude heuristics that try to guess which of the two will produce better results. The result? Connector deformation is incrementally more magical. Which is the point of the whole exercise.

- Code quality remains high; changes are nicely compartmentalized, performance is improving bit by bit, regressions are few and far between. All told, the the project is up to 5593 lines. About 500 of those are autogenerated *.cs.designer files, and another ~500 are helper routines from Jarrett & Su.

Some initial stuff

Mon, 14 Nov 2005 07:51:56 GMT

- Added a small gallery of screenshots from various points in the development process

- Populated the current list of work items

Welcome

Mon, 14 Nov 2005 07:46:10 GMT

Welcome to Hans Andersen's official Graphite Blog.

Photo Album: Graphite Screenshots-In-Progress

Sat, 11 Mar 2006 04:06:31 GMT

Graphite Screenshots-In-Progress

(July 2004) Earliest working version

(July 2004) Some progress - one tree is highlighted based in a graph traversal

(July 2004) First cut at the toolbar

(April 2005) Final version of the original codebase

(November 2005) "Graphite2005" - complete rewrite with Smart Arrow Detection, Multiple Undo, Save, and Load.

Lasso-selection in action.

(January 2006) Screenshot showing a high-level diagram of the new threading model

(February 2006) New ugly orange grab-handles for resizing

(February 2006) First rough-cut at a search feature

Graphite_in_UMPC_Emulator

More Photos...

Custom List: Graphite FAQ

Tue, 12 Sep 2006 03:30:30 GMT

Graphite FAQ

What is Graphite?
Graphite is the Spiffy Codename for a tablet PC application fordrawing and manipulating flowcharts and other box-and-line graphs.
Why not use Visio?
Good question. I use Visio at work all the time to create system architecture diagrams. The trouble with Visio is that it's not really designed for real-time note taking. I wanted a fast and fluid app that I could use while in a meeting.
So, wouldn't it be cool if you could export your Graphite document to Visio for final tweaks and publication?
That's the plan.
Why don't you update more often?
Graphite is a personal hobby project; I have a "real" job as a software engineer. I update Graphite (and this blog) when I have spare time.
Where can I get Graphite?
Head on over to the Graphite Project homepage!

Custom List: Graphite Work Items

Thu, 16 Mar 2006 07:26:31 GMT

Graphite Work Items

Optimize painting - dirty rects, etc.
Implement printing
Help Files and Documentation
(done)Re-work toolbar to make it smaller and more pen-amenable
(done)Implement cut, copy, & paste of both Graphite format and raw Ink
(semi-done)Scrounge up some nice icons
(done)Offload input stroke parsing onto a second thread so thatthe main thread - and thus the InkCollector - remains responsive