Augmented Paper System: A Framework for User’s
Personalized Workspace
Kavita Bhardwaj, Santanu Chaudhury, Sumantra Dutta Roy
Dept of EE, IIT Delhi
New Delhi - 110016, INDIA
Email: kavitabhardwaj.iitd@gmail.com, schaudhury@gmail.com, sumantra@ee.iitd.ac.in
Abstract—In this paper, we are presenting a framework for
“User’s Personalized Workspace” by augmenting the physical
paper and digital document. The paper based interactions are
seamlessly integrated with digital document based interactions for
reading as a activity. For instance when user is involved in reading
activity, writing becomes complimentary. In a academic system,
paper based presentation mode has facilitated such exercises.
Despite rendering the annotation on digital document and store
it onto the database, the content of the paper encircled or
underlined is used to hyperlink the document. Synchronizing a
physical paper and those of digital version in seamless fashion
from a user’s perspective is the main objective of this work.
We have also compared the existing systems which focus on one
activity or the other in our proposed system.
Keywords:
Personalized workspace, Augmented paper, Hyperlink-
ing, Annotation rendering
I. INTRODUCTION
In [1] The Myth of the Paperless Office authors observed
that despite the popularity of computers (desktop or tablets)
paper has not lost its advantage of the writing process. For
example, paper documents are easy to search as well as easy
to annotate while reading and still accepted in many social
contexts e.g. in learning, official meetings etc.
A vast research in this field has endured that the activities
which can be performed with physical papers can be easily
performed with digital documents e.g. drawing annotation,
hyperlinking etc.
Some systems only focus on augmenting annotations with
digital documents made on physical paper and presents the
system for active reading task. Capturing handwritten anno-
tations on a printed document and augmenting them with a
digital version of the document has been one of the most
frequent issue in pen-and-paper computing. PADD [2] paper
augmented digital document system has introduced the main
principle of paper-based annotation. This system uses a Anoto
digital pen comes with a plug-in for Adobe Acrobat to annotate
PDF document. Proofrite [3] is paper-augmented system used
to annotate word documents and applies the PADD concept.
These annotations are synchronized with digitally stored word
documents. The pen-and-paper based interaction is also used
for creating and following hyperlinks from printed documents
to web pages and the pens used in these systems are ultrasonic
pens. PaperLink [4] is one of the system of this class, use
a specific kind of pen on which a camera is attached. This
camera serves for creating and detecting hot spots and these
hots spots are the visual content of the document which serve
as link anchors. The Interactive Multimedia Textbook [5] and
PapierCraft [6] is also similar example of the same. There
are various systems which are developed for one feature or
the other. For this work, we collaborate such features with
novel techniques and present a system sufficient for a user
in academic system. Consider a scenario, a user in academic
system deals with lots of paper-based information. When user
is involved in reading, writing becomes a complimentary pro-
cess. These writings are in fact annotations which is considered
as a part of reading. We are here proposing this workspace
of a augmented paper based reading system to meet these
requirements.
The different primitives has been developed for familiar
mode of presentation of physical paper and digital document
augmentation for reading as a activity. The objective of propos-
ing User’s Personalized Workspace is to permit the user for
following activities:
• annotate a hard copy of paper while reading and same
is rendered on digital document
• as a part of reading, the user may have to find difficult
to understand words, the guided usage of the language
displays the meaning of such words on desktop by just
pointing the word
• during reading by underlining or encircling some
text, these annotations are used for content based
hyperlinking the document to other related documents.
Steimle et al. [7] designed an integrated framework for
various activities but that was specifically proposed for a team
work. Compared to this system, our proposed framework is
particularly developed for a user’s activity in academic system.
In our proposed work, we contribute a novel paper based
activities augmented with digital document based activities
in a seamless integration of both media. In section IV, we
discuss a novel annotation technique and the hyperlinking of a
paper to its related documents using annotations is described
in section V.
Section III explains how the system helps in learning
difficult to understand words on paper.
Finally in section VI we conclude the interaction design for
our proposed framework “User’s Personalized Workspace”.
II. A FRAMEWORK FOR USER’S PERSONALIZED
WORKSPACE
The grounds for this research is to create a perception
driven collaborative workspace which can be used for active
reading, learning and referencing to web-pages for a user
working on his Desktop. These primitives are synchronized
to develop a complete learning environment. The problem fo-
cused on incorporating the handwriting (drawing annotations)
as a temporal media can be synchronously observed on digital
version of the document. The next part is aimed at presenting
a realistic augmented learning environment using paper and its
digital version. In which a user can learn a language by just
pointing the word and the meaning of the word are displayed
on dialogue box and in last, on the paper by just encircling
and underlining a word the paper can be linked to different
documents lying on other portals.
So the overall proposed system has following features:
• Handwriting (drawing annotation) data as a temporal
media
• Learning difficult words of a language by pointing the
words on the paper with a pen
• Hyperlinking by encircling or underlining the words
on the paper with other documents lying on different
portals
A. Interactions between Entities
Figure 1 shows the user interface and the functionality of
the system at the back-end. The user’s desktop is attached
with a digital Inkpad on which physical paper is clipped. User
can draw annotations on the digital document and store that
annotated document, can select the word for hyperlinking the
document and can refer the meaning of difficult to understand
words.
Fig. 1. Interactions occur between entities
III. GUIDED USAGE OF A LANGUAGE
This application of interactive augmented digital document
aimed to help a user by providing meanings of difficult words
in his native language, while reading or working on a physical
paper. Figure 2 shows the setup used for augmented paper
system specifically used for this application.
Figure 2 shows the setup used for the system.
Fig. 2. Setup for augmented paper Framework
This application allows the users to access words on a paper
document with a pen gesture based command. The hardware
used for this setup are camera, projector and paper computer
coordinator. Towards the goal of fine-grained interaction with
paper documents, we used content-based physical-digital in-
teraction mapping. The physical-digital interaction mapping
basically translates a pen-pointing action on a paper document
into an equivalent digital counterpart.
A. Methodology used
For this SIFT algorithm is used as a content based ap-
proach. Scale Invariant Feature Transform(SIFT, hereafter) is
an approach for detecting and extracting local feature de-
scriptors that are reasonably invariant to changes in illumi-
nation, image noise, occlusion(partially), rotation, scaling, and
small changes in viewpoint especially geometric distortions.
The scale-invariant feature transform, or SIFT algorithm is
among the most well-known and widely-used invariant local
feature methods. This method is used for precise coordinate
matching between the physical paper and the corresponding
digital document by constructing a coordinate transform for
the camera and document frames referred. This approach does
not require any special markers or modification on physical
paper. A SIFT keypoint is a circular image region with an
orientation. This is described by a geometric frame of four
parameters: the keypoint center x-y coordinates, its scale (the
radius of the region), and its orientation. The SIFT detector
uses keypoints as image structures which resembles ”blobs”
and as it is scale invariant it can search the blobs at multiple
scales and positions. The keypoints are searched at multiple
scales which is obtained by constructing a Gaussian scale
space. The scale space is just a collection of images ob-
tained by progressively smoothing the input image, which is
analogous to gradually reducing the image resolution. As the
SIFT features are obtained between the captured image of the
paper and the stored reference image, these are matched to
get the initial correspondence. For finding the correspondences
circular normalized cross-correlation coefficient is used and the
aim is to measure the local intensity similarity between each
corner in both images. This circular cross correlation method
does not guarantee that all the correspondences are correct,
thus introducing outliers. These outliers can severely disturb
the estimated transformation and are required to be identified.
A RANSAC algorithm is used to handle this problem by
introducing a classification of the data into inliers (valid points)
and outliers while estimating the optimal transformation for
the inliers. A threshold t is used, which ensures that none
of the inliers deviates from the model by more than t. A
small number of data points are selected randomly among the
correspondence points and they should satisfy the least square
approximation criteria. A transformation model is generated
using the least-squares approximation method. All the other
correspondence points are transformed with the generated
transformation model and classified into inliers and outliers
using the threshold t. This process is repeated for a specific
number of times and the model is re-estimated using the inliers.
The camera captures and analyzes the frames to recognize
a portion of the document to detect and trace the user’s pen
tip. Once the image features of captured image and the pre-
registered stored image are matched, the correspondences thus
formed are used to compute the desired transform between
the physical paper and digital document. Now this requires
the physical paper to act like a touch screen when accessed
with the pen tip. A Meta data file is associated with the user’s
work in his database at the backend and referred for obtaining
the meaning of the word selected by the user. Instead of
including information about all the words in the document, the
size of the file can be maintained by allowing only selected
typical words which are relatively unfamiliar for a language
learner. For detecting the pen-tip, there are various approaches
but we have used the colour segmentation technique. For
our implementation, we have chosen light blue colour for
the colour of pen tip, assuming that the pen tip colour is
distinguishable from the background. The implementation is
done in C with OpenCV library and a hue based colour
segmentation scheme is used for pen tip detection.
Figure 3 shows the images of color segmentation for pen
tip detection.
(a) Original image
(b) Color segmented binary image
Fig. 3. Colour segmentation method for pen tip detection
The result of projecting meanings of the words on the user’s
desk is shown in Figure 4.
Fig. 4. Meaning of the word projected on user’s desk
IV. PERSONALIZED ANNOTATION
User can make handwritten annotation like underlining
something important, make synopsis of the read document on
physical paper while reading and can use it for future referral.
A digital Inkpad is used for making annotations. The user can
annotate the physical paper clipped on the Inkpad and these
annotations are drawn synchronously on its digital version and
can be seen on computer. Figure 5 depicts how annotations
rendered on digital document.
A. Capturing Handwritten (drawing annotations) with Tempo-
ral Information
This problem is divided in two parts, first is generation
of the annotations on a document and second is rendering
of the annotated data on the digitally stored document along
with temporal information. To facilitate the use of temporal
handwritten data, also termed as ink data, an InkML i.e. Ink
Markup Language is used. It supports a complete and accurate
representation using time-stamps. The media stream for ink
data includes 3 channels x-y coordinate of each pixel of the
stroke and T the time-stamp (Time information e.g. when the
annotations made). Each stroke of the handwritten data is
denoted as a trace and identified by a unique trace identifier.
The time offset contains the time-stamp when rendering of
this trace is to be started. Apart from the InkML data we
have one more table which includes the set of valid points
along with the pixel value. This table is combined with the
InkML to generate a set of 4-tuples. Each 4-tuple consists of
x-y coordinate, time-stamp, pixel value. This set is mapped to
the corresponding digital document on which annotation was
made.
B. Mapping of Ink data
Mapping of inkdata onto the digital document is a two step
process. 1) mapping of pixel data having x-y coordinate and the
RGB value to the temporal data which has x-y coordinate of
the pen locations and the time stamp. This mapping is obtained
by a shift matrix between temporal and pixel data. Shift
matrix is calculated using 4 point correspondence and best
fit approximation. Then the time-stamp value was calculated
using an elation approach where the pen locations (x-y coord)
are elated and the handwritten data is mapped to them. After
this mapping, we have a data which contains x-y coordinate,
time-stamp and pixel value. 2) the data obtained from the
prior is x-y coordinate, time-stamp and pixel location and it is
used for mapping onto the digital document. A transformation
matrix is calculated with the correspondence between ink data
coordinates to the document’s coordinates.
(a) Annotations made on docu-
ment using Digital Inkpad
(b) Generated mask of the anno-
tations
(c) Annotated digital document
Fig. 5. Annotations rendered on digital document
V. HYPERLINKING THE DOCUMENT
The annotations made by a user on paper are extracted and
inputted to hyperlink module to link the paper with related
documents on the web. For reading as a activity, user can
select the word on the paper by underlining or encircling (e.g.
Keyword of the paper) and the words/text are extracted using
OCR and used as a input information to link the paper. The
two primary sub modules work in synchronized way to fulfill
the task are:
1) marking or highlighting the word on the paper
2) select the highlighted word, recognize using OCR and
use it as input to search the related document on the
web.
A Digital Inkpad used to highlight a word on the physical
paper and the underlined or selected word is extracted from
the digital document and used as inputted to the hyperinking.
The overall process takes following steps:
1) The document is first divided into horizontal blocks.
Each line in the document represents a block.
2) Within each horizontal block the words are separated
and the information about each word is stored.
3) Using OCR the selected word is converted into text.
4) When a user writes on a document the number of
colored pixel within a word block is counted.
5) If the number is greater than a certain threshold, then
the word is marked selected and it is shown on a
panel.
A. Hyperlinking of the selected word
The selected word as described above is linked to a site
using ActionListener and Desktop class of java. The selected
word is shown on a panel and the user can go to a certain
link by clicking on that panel. We have linked the search of
the documents for the selected word on the panel with google
search engine.
(a) Annotated digital document (b) Selected word ex-
tracted from the digital
document
Fig. 6. Selecting the word for hyperlinking the document
VI. CONCLUSION
In this paper we handle different problems on augmenting
the physical paper and the digital document. The activities that
can be performed by the author while reading the physical
paper like drawing annotation, further using the annotations
for hyperlinking and obtaining the meaning of difficult to
understand words etc. can be augmented with digitally stored
document. A Digital Inkpad is used for rendering the anno-
tations and selecting the words from the digital document.
Integrating these technology and bring it as a Personalized
Workspace for a user is the objective of this research.
REFERENCES
[1] A. J. Sellen and H. Richard, “Harper (2001). the myth of the paperless
office.”
[2] F. Guimbreti
`
ere, “Paper augmented digital documents,” in Proceedings
of the 16th annual ACM symposium on User interface software and
technology. ACM, 2003, pp. 51–60.
[3] K. Conroy, D. Levin, and F. Guimbreti
`
ere, “Proofrite: A paper-augmented
word processor,” Demo Session of UIST, 2004.
[4] T. Arai, D. Aust, and S. E. Hudson, “Paperlink: a technique for
hyperlinking from real paper to electronic content,” in Proceedings of
the ACM SIGCHI Conference on Human factors in computing systems.
ACM, 1997, pp. 327–334.
[5] W.-C. Lai, P.-Y. Chao, and G.-D. Chen, “The interactive multimedia
textbook: Using a digital pen to support learning for computer program-
ming,” in Advanced Learning Technologies, 2007. ICALT 2007. Seventh
IEEE International Conference on. IEEE, 2007, pp. 742–746.
[6] C. Liao, F. Guimbreti
`
ere, and K. Hinckley, “Papiercraft: a command
system for interactive paper,” in Proceedings of the 18th annual ACM
symposium on User interface software and technology. ACM, 2005,
pp. 241–244.
[7] J. Steimle, O. Brdiczka, and M. Muhlhauser, “Coscribe: Integrating
paper and digital documents for collaborative knowledge work,” Learning
Technologies, IEEE Transactions on, vol. 2, no. 3, pp. 174–188, 2009.
