View this PageEdit this PageUploads to this PageHistory of this PageHomeRecent ChangesSearchHelp Guide

Heather Richter

IEEE Pervasive Computing Review Form (V2.0)

Paper number: 114

Title: Using the Experience Sampling Method to Evaluate Ubicomp Applications

Submission type: Research Paper
/Research paper
Project overview

[All submission types]
Relevance to magazine: 4
/5 = Burning issue of the day; 4 = Timely and relevant;
3 = Worth reporting;
2 = Yawn; 1 = Much ado about nothing/

[All submission types]
Presentation and clarity: 3
/5 = Lucid and eloquent; 4 = Easily understandable;
3 = Could be improved;
2 = Hard to understand; 1 = Dense and impenetrable/

[All submission types]
Reviewer expertise: 3
/5 = I'm an authority on this topic; 4 = My work overlaps this topic;
3 = I'm know this stuff well, but don't work on it;
2 = I'm somewhat familiar with this stuff; 1 = This is new to me/

[Research papers only]
Depth of contribution: 3
/5 = Deep and Impressive; 4 = Good, but could be improved;
3 = Useful contribution, but not earthshaking;
2 = Ho Hum; 1 = Seriously flawed/

Detailed comments for the authors:

Contribution: Introduce the Experience Sampling method as a key formative evaluation technique in ubiquitous computing. Introduce issues involved in using this technique.

I find the intented contribution of the paper important and interesting. This method has been used in several studies recently and I think a paper presenting the method in more detail for use in ubiquitous computing is worth publishing.

The authors do an acceptable job introducing the ESM method and placing it within the context of other HCI evaluation methods. They then discuss issues in alerting, delivering, and capturing the information, as well as their experience in using ESM in a PDA based study. My concern with this paper is that the important issues and concerns in using ESM in this particular domain are not laid out well enough, or even deep enough. The bulk of the general issues is covered in two pages, with the case study of the personal server study spanning three. The case study does show the method in real use, and is a necessary part of the paper. However, the general issues need to be pulled out and discussed separately and need to make up the bulk of the paper. The paper needs to be organized such that the reader can more easily pull out the choices they have when preparing the study, and the issues and decisions they need to make when performing an ESM. ESM has been presented elsewhere, even in regards to computer support for ESM. The authors could summarize what those references cover (I'm having trouble finding electronic copies of the references, so I haven't been able to read them yet) and then add issues that are particular to the ubicomp domain. I believe that there's more that can be summarized, presented, and discussed that what is on page 3. I would encourage the authors to reorganize the general issues they have, perhaps adding additional lessons learned from other ESM studies and their own, and make that a more prominent contribution of the paper.

The final paragraph in the Alerting section does not make sense to anyone who is not already familiar with the study they reference. Please provide a better explanation.

The Delivering paragraph seems a little brief. Is there more discussion that could be included here?

Comments for EIC and AEICs only:

The contribution of the paper is important, and I would like to see such a paper published. However, as it stands, I don't believe the paper makes enough of a general contribution to be worth publishing in this forum at this time.

Reviewer: Heather Richter