HOMEWORK 4: Interpreting and Processing Digital Ink

This is an INDIVIDUAL assignment.

Objective

In this assignment we'll continue our focus on digital ink. Here, we'll explore how to use and extend the recognizer we created in the previous assignment in order to tell the Courier application to "interpret" pre-existing regions of digital ink in certain ways.

The learning goals for this assignment are:

Description

In this homework, we'll extend the SiGeR recognizer built in the previous assignment to do some new things. In particular, we'll add some new gestures that tell the system to interpret pre-existing content in some interesting ways:

List Behavior

Drawing a "left square bracket" gesture ([) to the left of a region of ink indicates that that ink should be interpreted as a list of lines, such as a to-do list. Such an "interpreted" list should appear then appear differently, and should acquire additional behaviors related to lists. For example, imagine that you have a shopping list written in your journal. You want the system to treat these lines of digital ink as items in a list, so you draw the square bracket gesture to the side. This causes the list to change appearance to indicate that it is now being treated as a list, not just "dumb" digital ink (this might be done by a special border around the list, or shading its background, or some other graphical effect). Once the region has been interpreted as a list, new gestures become available that let you do list-like things on it. These should include:

Sticky Note Behavior

Drawing a box around a region of content turns it into a "sticky note" (see this on the Courier demo video at around 0:35). This means that the content (ink, shapes, and strokes) within the box is extracted from the display list and put into a "sticky note" component that is then embedded within the journal page. Once turned into a sticky note, the content can be dragged around within the journal page, and appears over the top of other content that's written directly onto the page background. (The sticky note itself doesn't need to respond to any input other than being dragged--it's ok if you can't draw on a sticky note.)

Handling the Content Interpretation

Just as in the previous assignment, we're using a mode to distinguish ink (left mouse button) input versus gesture input (right mouse button). Gesture input should be drawn on screen while the gesture is being made, so that it provides feedback to the user. The gesture should disappear once the mouse is released. The basic interaction pattern with these new gestures should thus be the same as in the previous assignment.

There are two aspects of content interpretation we'll be working on here: the list behavior and the sticky note behavior.

For the list behavior, once you recognize the bracket gesture you then need to determine what region of the inked content the list behavior should be applied to. You don't have to be fancy with this: assume that the user will draw the bracket to the left of the content to be "listified;" the bracket establishes the top and bottom (Y axis) bounds of the list, as well as the left (X axis) bounds of the list. To get the right (X axis) bound, simply go until you encounter a band of whitespace that runs the vertical dimensions of your region. You can figure this out by examining the bounding boxes of everything contained within the vertical region marked out by the bracket. Once you have the bounds of the contained ink, you can render it differently to indicate that it is now being interpreted as a list.

NOTE: You *don't* have to worry about what happens if the bracket's bounds cut through one or more strokes--in other words, you don't have to be concerned with splitting a single stroke, shape, or block of text into "listified" and "non-listified" pieces. We won't test for this.

Next, you need to examine the ink in this region to break it out into items. This is called "segmenting," and is a common algorithm in lots of digital ink-type applications.

While ideally, users would write in clean, evenly-spaced lines (so that you could just search for spans of horizontal whitespace that delineate lines), in actuality people write much more sloppily. Hand-drawn ascenders and descenders, for example, may mean that there is no clean whitespace between items. So, the best way to segment the ink into lines is to walk across each horizontal band of pixels in the contained area, counting ink pixels versus whitespace pixels, and then say that lines with ink pixels below some threshold are part of the gaps between lines. Rather than having an absolute threshold in terms of numbers of pixels--which won't cope well with lines of varying length--you'll likely want to have some sort of ratio threshold (perhaps: "all lines with only 10% of the inked pixels of the most inked line are considered whitespace," although you'll want to play around with this).

NOTE: One way to do this pixel counting is to render the region to an off-screen image, then simply iterate over it examining color values of the pixels. There are lots of tutorials on the web about how to render swing components to offscreen images, such as here).

Once you've done this you'll be able to set bounding boxes that indicate your best estimate of what the individual items are in the list. You should save this in a data structure so that you don't have to keep recomputing this. Once you've computed this data structure, it's pretty easy to do the rest of the list manipulations: recognize the up and down caret, and the deletion gestures. When a gesture happens over a list item, you implement the recognized action by simply translating the bounds of the inked items (to move up or down), or take the ink out of the display list (to delete).

The sticky note behavior is actually a little easier than the list behavior. I'd suggest starting by creating a separate, simple "sticky note component." This component will draw itself like a sticky note, and render a list of strokes, shapes, or text that are passed to it when it is constructed.

When you recognize the box gesture, simply take the ink contained within its bounding box, remove it from the display list, and then create a new sticky note to contain it. The sticky note can then be made a child of your journal page component, positioned at the X, Y coordinates that originally contained the content. You'll need to implement some basic dragging behavior, where a mouse drag in the sticky note moves it around the journal page.

Note that you do NOT have to allow the content in the sticky note to be editable, nor do you have to worry about occluded (covered up) content in the journal page.

Extra Credit

As usual, there are a lot of ways you might make this assignment much fancier than described:

Deliverable

This is an INDIVIDUAL assignment; while you may ask others for help on Java or Swing details, please build your application on your own.

Here are the details for how to turn in the assignment. We'll be using this structure for all of the turn-ins:

1. Create an executable JAR file named courier.jar that contains your runnable application. (See here for details on how to create executable JAR files.) We should be able to run your program by typing "java -jar courier.jar" on the command line; please be sure that your application runs correctly using ONLY this command, and that it doesn't require any additional CLASSPATH or other environment variables, no additional parameters, no classfiles or images located outside the JAR file, etc.

2. Create a new directory using your last name as the name of the directory.

3. Put the courier.jar file into the top level of this directory, and all of your source files into a "source" subdirectory inside this directory.

4. Put a README.txt file into the top level of this directory. This file should contain your name and email address, the version of Java you used (Java 1.6.x only, please) as well as any special info we might need to know about your program (let us know if you did extra credit, for example).

5. ZIP this directory and submit via T-Square (instructions are here).

Please take care to remove any platform dependencies, such as hardcoded Windows path names or dependence on a particular look-and-feel that may not exist on all platforms. Also, if you use any images in your application, please make sure that you include these in your JAR file and that your code will refer to them and load them properly when they're in this JAR file (see this page for some details on how to include and load images from within a JAR file).

Grading for this assignment, and future assignments, will roughly follow this breakdown:

Please let the TA or me know if you have any questions.