Filetype PDF | Posted on 01 Feb 2023 | 3 years ago
Authentication
digital resources
282x Filetype PDF File size 2.67 MB Source: www.diva-portal.org
ARTICLE IN PRESS
Interacting with Computers xxx (2008) xxx–xxx
Contents lists available at ScienceDirect
Interacting with Computers
journal homepage: www.elsevier.com/locate/intcom
Setting the stage – Embodied and spatial dimensions in emerging
programming practices
Martin Jonssona, Jakob Tholanderb, Ylva Fernaeusb,*
aSödertörn University, School of Communication, Technology and Design, SE-141 89 Huddinge, Sweden
bStockholm University, Swedish Institute of Computer Science, Forum 100, SE-164 40 Kista, Sweden
article info abstract
Article history: In the design of interactive systems, developers sometimes need to engage in various ways of physical
Received 14 October 2008 performanceinordertocommunicateideasandtotestoutpropertiesofthesystemtoberealised.Exter-
Accepted 14 October 2008 nal resources such as sketches, as well as bodily action, often play important parts in such processes, and
Available online xxxx several methods and tools that explicitly address such aspects of interaction design have recently been
developed. This combined with the growing range of pervasive, ubiquitous, and tangible technologies
Keywords: adduptoacomplexwebofphysicalitywithinthepracticeofdesigninginteractivesystems.Weillustrate
Interaction design this dimension of systems development through three cases which in different ways address the design
Embodied interaction of systems whereembodiedperformanceisimportant.Therstcaseshowshowbuildingaphysicalsport
Physical user interfaces simulator emphasises a shift in activity between programming and debugging. The second case shows a
Embodied performance
Programming practice build-once run-once scenario, where the ne-tuning and control of the run-time activity gets turned into
anactofinsituperformancebytheprogrammers.Thethirdexampleillustratestheexplorativeandexpe-
riential nature of programming and debugging systems for specialised and autonomous interaction
devices. This multitude in approaches in existing programming settings reveals an expanded perspective
of what practices of interaction design consist of, emphasising the interlinking between design, program-
ming, and performance with the system that is being developed.
2008Elsevier B.V. All rights reserved.
1. Introduction cial performance in physical spaces, but also as the tools that
developersusehavedevelopedfromtext-based,throughgraphical,
The area of human–computer interaction has increasingly towards systems based on multimodal, and different forms of
recognised the physical and material dimensions of acting with physical interaction. Early programming resources did, for in-
and around computational artefacts. This is mirrored through the stance, take the form of holes in physical punch cards, while more
central theme of much contemporary research concerned with recent formats include textual codes, graphical rewrite rules (Can-
understanding the social and physical interdependencies in rela- eld Smith et al., 2001), or even animated (Kahn, 1996) and tangi-
tion to peoples interaction with technology. This has most prom- ble (Horn and Jacob, 2007) program representations. Currently, a
inently been conceptualised through notions of embodied rangeof physical and tangible resources for various aspects of pro-
interaction and in studies of situated conduct, e.g., by work of Dou- gramminghasbeendeveloped,includingtangible forms for repre-
rish (2001) and Heath and Luff (2000). Notions of physicality and senting programming constructs (McNerny, 2004), physical tools
embodiment are also central in research on pervasive computing for construction of code structures displayed on a computer screen
andtangibleinterfaces, in which the particular physical manifesta- (Fernaeus and Tholander, 2006a), and methods of tracking and
tion of a computational artefact and its consequences for peoples recording physical manipulation of actuated devices (Frei et al.,
interaction with and through the artefact is commonly brought to 2000;Rafeetal.,2004;Hartmannetal.,2007).Thereisalsoagen-
discussion (see Hornecker and Buur, 2006; Klemmer et al., 2006). erally increasing focus on social and situated aspects of system
The premise of this paper is that this development involves a development, manifested, for example, in the form of resources
number of new challenges for the designers and programmers of for collaboration such as version management software, open-
interactive artefacts and systems. This is partly since interactive source tools and public class libraries.
systems increasingly involve artefacts that support bodily and so- In our earlier work, we have explored how physical and visual
programmingtoolscontributetoareshapingoftheactivityof pro-
gramming, from intellectual and individual, towards getting the
* Corresponding author. Tel.: +46 708763301. character of a largely social activity relying heavily on physical ac-
E-mail addresses: martin.jonsson@sh.se (M. Jonsson), jakob.tholander@sh.se (J. tion and material resources. To describe this shift, we studied so-
Tholander), ylva@sics.se (Y. Fernaeus).
0953-5438/$ - see front matter 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.intcom.2008.10.004
Please cite this article in press as: Jonsson, M. et al., Setting the stage – Embodied and spatial dimensions in emerging ..., Interact. Comput.
(2008), doi:10.1016/j.intcom.2008.10.004
ARTICLE IN PRESS
2 M. Jonsson et al./Interacting with Computers xxx (2008) xxx–xxx
cial performances and their role in programming activity (Ferna- Laurels Computers as Theatre (Laurel, 1993), where the computer
eus and Tholander, 2006a), emphasising how visual and tangible programitself is looked upon as a live performance”, experienced
forms of representations allow people to involve bodily actions in use by its users. The role of the developer is then characterised
such as pointing and gesture in a more direct sense than is pos- as a theatre director orchestrating the acts to be performed by the
sible with traditional text-based or symbolic programming repre- computer program. A second view is through the performance of
sentations. In analyses of children making animated games users, who together and with the system act in a social setting, to-
together on a PC, a central aspect of the negotiation did, for in- wards a real or imagined audience” (e.g., Grint and Woolgar,
stance, concern how objects in the nal game should move and 1997). A third perspective concerns the communicative acts of
behave. To discuss such aspects, the children made extensive designers and programmersin the making”.Examplesofmethods
use of gestures, and also external resources such as sketches on that explicitly address such matters include test-running of low-
a piece of paper. The dynamic and spatial properties of the nal delity prototypes through collaborative role-plays (Rettig, 1994)
application thus presented a number of challenges that the par- and methods of bodystorming (Oulasvirta et al., 2003), where
ticipants addressed through bodily performance within the pro- designers explore a given context of use through acting out every-
gramming activity. Similarly, extensive use of bodily action has day activities in the environment of their target users. Recently,
been observed among adults building administrative software such bodily aspects have been increasingly brought into the con-
systems when talking about less tangible aspects of their designs ceptual discussions of interaction design at large, especially in
(Tholander et al., 2008, see Fig. 1). the design of physical and mixed-media interactive environments
In this paper, we further explore how spatial and physical as- (Jacucci, 2004; Ciol and Bannon, 2007). A related aspect, and
pects, not of the tools but of the intended interactive setting affect which we will focus on here, concerns the general development
the character of a programming activity. This further emphasises of newtechnicalsolutionsthatextendthequalitiesoftheresulting
themultitudeofunderstandingsofwhatprogrammingisandwhat systems to allow for richer forms of manipulation, perception and
it should be. We focus specically on the activity of building inter- technology use. An important aspect of such technologies is that
active systems that have physicality and spatiality as central qual- they invite users, and also the developers, to engage in more phys-
ities of the target setting, i.e., system that support interactions that ical, social, and bodily forms of interaction.
include enacted and embodied performances in shared physical To further understand how the concept of performance can be
spaces. A basic assumptionis thatanalinteractivesettingthatre- understood in these respects, we have chosen to explicitly draw
quires enacted and embodied performances shapes the nature of attention to the activity of computer programming. We nd this
the programming activity to hold a rather different character from relevant as this activity has traditionally been viewed as a primar-
whenother forms of interactions are designed. ily intellectual activity – a view that has recently been challenged
After a short overview of the perspective of embodied perfor- by a range of empirical studies (Button and Sharrock, 1995; Dow-
mances in computer programming, we present three brief cases ney, 1998; Sharp, 2004; Chong and Hurlbutt, 2007). A view that
that illustrate how developers manage the situation of building puts its primary focus on the intellectual aspects of activity has
systems for different kinds of physical and performance-based also been substantially criticised in social and cognitive science
interactions. The examples are used to emphasize a general shift in general, emphasising the socially and physically situated nature
in how to understand programming practice, and how it gets of creative work in professional practices (see, e.g., Schön, 1983;
shaped by the variety of ways that emerge through users interac- Suchman, 1987, 2007; Lave, 1988; Schatzki et al., 2001).
tion with digital technology that increasingly become physical and This perspective of studying programming as a socially and
spatial in character. We end by discussing how an increased physically situated activity, strongly shaped by the character of
acknowledgement of physical aspects of programming suggests theavailableresources,haselsewherebeenreferredtoasembodied
new challenges for the broader development of new resources programming,drawingonDourishsnotionofembodiedinteraction
for software development. (Fernaeus and Tholander, 2006b). This includes a general concern
for bodily actions performed in order to interact with and through
2. Embodied performance in computer programming a programming environment, and also to the physical aspects of
discussions, negotiations, and perceptionthatgoesonwhenpeople
Performanceininteractiondesigncanbeinvestigatedthrougha engage in construction of computational artefacts. Note however,
range of different perspectives. One is the classic view of Brenda that this is not to be mixed up with embedded programming, which
concerns more specically the sensing, behaviour, and control of
physical appliances.
In the eld of HCI, much workhasattemptedtodevelopwaysof
bringing social and material aspects of use settings to the centre of
concernofinteractiondesign practices, e.g., through contextual in-
quiry, participatory design methods, technology probes, and quick
and dirty ethnography. Bridging the gap between use and design
settings is, however, not only a question of providing information
about users and use settings to the designers. A recent focus on
physicality in programming can be found for example in the moti-
vations for pair programming, where two developers work to-
gether in front of a shared screen. In this way, errors are more
easily detected, and problems are immediately discussed and ad-
dressed collaboratively. Other physical aspects noted in ethno-
graphic accounts of extreme programming practice include the
importanceofacommonworkspace,sothatparticipantsmoreeas-
ily can create a shared understanding of what everybody in the
team is currently engaged in. Instead of storing information about
plans, goals, requirements, etc. in digital les, information is made
Fig. 1. Interaction designers engaged in a modelling activity. publicly available as physical representations in the forms of phys-
Please cite this article in press as: Jonsson, M. et al., Setting the stage – Embodied and spatial dimensions in emerging ..., Interact. Comput.
(2008), doi:10.1016/j.intcom.2008.10.004
ARTICLE IN PRESS
M. Jonsson et al./Interacting with Computers xxx (2008) xxx–xxx 3
ical story cards and sticky notes on whiteboards (e.g., MacKenzie ests in their work, we asked the developers to perform and explain
and Monk, 2004; Sharp, 2004; Chong and Hurlbutt, 2007; Martin typical activities in their programming practice, and especially
et al., 2007). how they dealt with the very physical aspects of the interactive
In this paper, we discuss how experiential properties of specic technologies that they developed.
end-user settings are addressed concretely by programmers in
their practice. This could be seen as the reverse problem as that ex- 3.1. Designing for full body interaction
plored in palpable computing (Büscher et al., 2007), which is con-
cerned with making the invisible” internal aspects of a A range of interactive technologies requires users to engage in
computationalsystem,e.g.,systemarchitectureandsoftwarecom- extensive bodily engagement to interact with a computational
ponents, easier for end-users to talk about and relate to in partic- system. Such systems include sports arcade games, simulators,
ipatory design practices. Similarly, tangible programming and training applications for different settings. Physical golf sim-
projects are commonly addressing how physical representational ulators is an example of this class of systems, allowing people to
forms may make abstract programming concepts more concrete. practice their hobby by hitting real golf shots using real clubs and
Hereinstead,focusisonhowthephysicalend-usercontextisdealt balls into a screen projection of a golf course. Naturally, building
withinconventional,PCandtext-basedprogrammingsettings.The such a system will take on a different character than the more
intention is to start identifying some of the key challenges in commonly observed cases of developing systems for PC and lap-
designing and realising systems for physical action and bodily top-based modes of interaction. To explore these aspects further,
performance. we visited a company that focuses on building such golf
simulators.
3. Three cases of programming for embodied performance Theobservedsettingissituatedinagolftrainingcentreconsist-
ing of a number of small booths each containing an interactive
Thissectionprovidesthreeexamplecasesofdesigningforinter- golf simulator environment that are used by individuals or small
active settings where embodied performance is a central property, groups of players. Each booth consist of a large wall projection of
andhowthisinuencestheactivityofprogramming.Theaimisto a3D-renderingofagolfcourseincombinationwithasensor-based
explore the multitude of use settings that developers may need to deviceforcapturingthephysicalpropertiesofthegolfshot,suchas
relate to. Rather than documenting the practices in detail, we have ball speed, direction, and spin, to calculate the trajectory and indi-
chosen to describe specic themes of the three different settings, cate where in the simulated environment the ball will end up
pointingathowtheyilluminatecertainaspectsandchallengesthat (Fig. 2).
programmers have to face when designing for such settings. Thesetupofthesystemprovidesaninteractive contextthatin-
Throughthesecasesweattempttocontributetoanunderstanding volves two distinctly separate modes of interaction, rst, hitting
of some of the dimensions involved in design of technologies golf shots to actually play the game, and second, interacting with
involving physical experiences, and in particular, the interplay be- the simulation to carry out the actions necessary to play in the
tween programming practice and its resources, the physical con- manner one wants. This involves indicating to the system what
texts, and the material properties of the interactive devices. club one is using, aiming the screen projection in the direction
First, we examineagolfsimulatorsetting,whichisacasewhere one wants to hit the shot, perceiving information presented on
the target activity is heavily based on physical action and full body the screen regarding distance to the hole, speed, and direction of
interaction. Our second case concerns the design and control of the wind, elevation, etc. This requires designers to support the
interactive performances in public spaces, in the form of live med- users possibility of shifting orientation between viewing the
ia-rich presentations at large exhibitions. Finally, we examine the screenprojectionasagolfcourseandviewingtheinformationpre-
activity of programming a certain type of autonomous embodied sented to control the progression of the simulation. The rst being
interactive devices called GlowBots. An overview of the three cases a highly physical mode of interaction while the second being more
is provided in Table 1. of an information seeking mode of interaction.
Each setting was studied through a light-weight ethnographic Collaborative and social dimensions are important both to how
study, based on observations and interviews with programmers golf is generally played in real-life settings, but was also prominent
at their respective work places. After explaining our research inter- in our observations of how the game is played in the golf simula-
tion centre where it was set up. Addressing these dimensions re-
quires the developers to consider how the physical and spatial
Table 1 setting supports ofine aspects of the interaction around the sys-
Characteristics of the three cases. tem such as observing others shots, commenting on them, com-
Use setting Programming setting paring them to those of oneself. This is supported both by how
Golf simulator the spatial and physical conguration of the simulation cater for
Full-body interaction including non- PC-based interaction in conjunction watching by audience and co-players. It is also addressed in the
computational artefacts as interface” with a physical stage for debugging simulation technology by leaving traces of the trajectories of the
elements, i.e., real golf clubs and balls, and ne-tuning the simulator shots of the other golfers that remain on the screen projection. In
and physical space our observations of the developers building the system they use
Multimedia show a conventional PC-based programming environment to control
System tailored for a specic location, Programming activity is the simulation technology. Their golf simulation technology is
interaction includes a complex intertwined with the creation of a based on models and principles from virtual game-worlds used
collaborative interplay between people physical environment, as well as in traditional desktop settings. A key challenge in this development
on and behind a physical stage with the nal end-user activity
setting is hence to use a traditional programming environment in building
GlowBots atechnologythatinvolvesahighlyphysicalactivityconductedina
Handheld physical interaction, where Programming and testing happens constrained physical space.
multiple devices in combination work to on very different devices, Programming and testing in this environment naturally in-
frame the larger social setting, e.g., to emphasising loops of testing, volves addressing a number of issues in which physical and spatial
turn off the lights and perform debugging, and tuning the qualities become salient. As non-computational physical objects
physically with the displays interactive properties such as club and ball are central interactive resources testing,
Please cite this article in press as: Jonsson, M. et al., Setting the stage – Embodied and spatial dimensions in emerging ..., Interact. Comput.
(2008), doi:10.1016/j.intcom.2008.10.004
ARTICLE IN PRESS
4 M. Jonsson et al./Interacting with Computers xxx (2008) xxx–xxx
Fig. 2. An illustration of the physical use setting of the golf simulator and the two different modes of interaction with the system while playing, as well as when programming
anddebugging.Theleftpictureshowsinteractionusinggolfballandclubs,andtherightshowsthePCwithwhichtheusershavetointeracttoforexampleselectcourseand
clubs.
e.g., the correctness of the physical measurements cannot be sim- a stage where one or several presenters give speeches accompa-
ulated on the PC only. Testing and debugging thereby requires the nied by visual multimedia material displayed on large displays
programmer to use a physical setup of the system and hit actual (see Fig. 3). The multimedia material can be composed of static
golf shots to generate different test values. The opposite of such text/image slides in combination with video streams and/or com-
technical testing procedures involves addressing how the system puter animations. An important difference to other multimedia
supports the imagined experience and the appropriate feeling of presentations is that this material is spread out over several
actually hitting real golf shots on a golf course. A central issue here screens, is often combined with live video footage of the presenter,
regards addressing how users will experience the spatial qualities andtypically also includes a light show with directed and ambient
of the setup so that it gives a realistic sense of how the course pro- light accompanying the presentation as well as spatial audio
jection is perceived. Achieving this requires extensive ne-tuning accompanying the various video streams. There are also several
of the specic placing of the sensors and the position of the hit- points of interaction both with the system and between the differ-
ting-mat hit in relation to the room and the screen projection. entpeopleinvolved.First,theshowisperformedcollaborativelyby
the people on stage and a number of people (who are also the
3.2. Programming for performance in large public spaces designers/programmersofthepresentation)locatedinabackstage
control room. The presenter has control over what slides and video
Performances in public spaces have been a popular domain for that are currently being displayed, either directly or indirectly by
exploring novel interaction forms and technologies. Such settings subtle cues to the control room staff. Except for interacting with
include musical performances (Jordà et al., 2007), place-specic thepresenterthecontrolroomstaffalsohastocoordinatetheirac-
artworks (e.g., Cofn, 2008) and one-off pervasive game experi- tions between them. At certain types of events the presentation is
ences (Jonsson et al., 2006). One kind of system intended for such also combined with interactive features to, e.g., allow for voting or
a public interactive space that we have chosen to investigate here other feedback from the audience, and in other ways of engaging
is multimedia presentations at larger exhibitions. The activity of the audience in the activity.
setting up and controlling such systems provides an interesting The presentation activity can be divided into two phases, the
case to examine due to the various performative, embodied and setup (programming) phase, and the performance (execution)
spatial factors affecting the activity. phase. The setup phase focuses on the creation of the stage, both
The systems specialised on at the second development com- in a literal physical sense, but also in a more general sense as to
pany that we visited, are aimed for settings that typically include create a framing of the performance that will take place. The
Fig. 3. Physical setting of a multi-screen multimedia presentation accompanying an on-stage presentation (left), and the control room used to set up and control the various
elements in the presentation during the show (right).
Please cite this article in press as: Jonsson, M. et al., Setting the stage – Embodied and spatial dimensions in emerging ..., Interact. Comput.
(2008), doi:10.1016/j.intcom.2008.10.004
no reviews yet
Please Login to review.
