Facing the future: Natural user Interfaces and transmedia spaces in museums

Scott Gillam, Canadian Museum for Human Rights, Canada, Liz Neely, Georgia O'Keeffe Museum, USA, Corey Timpson, Corey Timpson Design Inc., Canada


In 2016, the Canadian Museum for Human Rights (CMHR) began development of the exhibition "Rights of Passage." This exhibition explores human rights history in Canada over the 150 years since Confederation—how the rights and freedoms we now enjoy were fought for by those before us who refused to stay silent in the face of inequality and injustice, examining concepts and stories of courageous actions that inspire us to continue to work for the rights of all. The challenge undertaken with this exhibition was to immerse visitors within content in a way that encouraged relevance. Design direction focused on two primary tactics—medium and material— both creatively used to provoke visitor action, participation, and create relevance. Experience design through the "Rights of Passage" exhibition builds on CMHR’s reputation of mixed media presentation, immersive and layered storytelling, and integrating physical artifact, text, and image within transmedia storytelling. Included in the presentation are traditional media interfaces such as a radio and television, gesture and audio activated experiences, culminating in augmented spaces that include user-contributed content, wearable technology, and 3D-virtual artifacts (holograms). The boundaries of progressive storytelling and education are expanded through this design, placing the visitor at the center of a shared experience that is both physical and virtual in nature. Exhibition design is increasingly under the obligation to create hybrid spaces that merge the real and the virtual. The opportunities and drawbacks of new technology are creating seismic changes in how museums balance storytelling, scholarship, collection of physical objects, digitization, and enabling visitor engagement in both physical and digital environments. This paper explores the locus of innovation in relationships between human computer interaction, virtual artefacts, and the internet of things.

Keywords: storytelling, emerging technology, digital experience, augmented reality, wearables, education, interpretive interactive installation

In 2017 Canada celebrated its 150th birthday. To mark this occasion, the Canadian Museum for Human Rights (CMHR) set upon designing, developing, launching, and running a series of programs. One such program was the Rights of Passage exhibition. Rights of Passage: Canada at 150 examines 150 years of Canada’s human rights history. It does so by exploring the key debates and issues that have been central to the evolution of the Canadian conversation on rights, freedoms, justice, and equality.

These debates, issues, stories, and movements are organized through the following eras, represented as distinct zones within the exhibition:

  • 1867-1914: Foundations and Dislocations
  • 1915-1960: Transformations and Interventions
  • 1960-1982: Towards the Charter
  • 1982-Present: Human Rights in a Contemporary Canada
  • Past and Present: Defending Sovereignty

The problem

One of the challenges of this project was the focus on producing an exhibition that would encourage reflection and dialogue. How to ensure audiences would connect with sometimes difficult subject matter that was potentially removed from personal connection and relevance by generations? How to bring meaning and understanding, and provoke empathy for the debates, struggles, and actions of our ancestors? How to engage all visitor demographics in the historically didactic material?

The solution, born through interpretation and design, creating an immersive environment that would complement and make relevant the content being presented.

Design approach

In the case of Rights of Passage, the exhibition design direction is focused on two key elements: material and medium. Through this direction, the intent is to immerse visitors in the period of the content being presented, having visitors use mediums/materials of that time that would have actually been used to present these stories. This approach was carried through each zone of the exhibition as follows:

  1. 1867-1914: wood, printing press, magic lantern (figure 1)
  2. 1915-1960: steel, radio (figure 2)
  3. 1960-1982: plastic, television (figure 3, figure 4)
  4. 1982-future: light (figure 5)
  5. Past and Present: oral tradition and the intangible (figure 6)
Wall containing raised text, cut of wood on the left-hand side, and a projection of a young boy on the right.
Figure 1: Zone 1 – Foundations and Dislocation. Material: wood. Media: Printing press, pulp magazines and posters, magic lantern.
Evening shot of installation wall featuring printed text, images, and a physical vintage radio.
Figure 2: Zone 2 – Transformations and Interventions. Material: Steel. Medium: Radio.
Evening interior image of walls containing text, with coloured plastic bars in a "U" shape pattern.
Figure 3: Zone 3 – Towards the Charter. Material: Plastic.
Three faces of television sets emerge from a wall covered in museum text. A blurred figure is turning one of the dials on the television.
Figure 4: Zone 3 – Towards the Charter. Medium: Television.
An elongated curved wall stretches from the bottom right to the top left. The wall is made up of text and images that are lit from behind.
Figure 5: Zone 4 – Human Rights in Contemporary Canada. Material and Medium: Light.
A wall with a depressed red circle in the center containing exhibition text on the left. To the right, a pair of show shoes in a glass case.
Figure 6: Zone 5 – Reclaiming Sovereignty. Material: 4 sacred plants (red cedar, sage, tobacco, and sweetgrass). Medium: intangible, audio

In each of these zones, the environment is designed and built to emote the era it depicts, with an emphasis on the material and the natural interface being used to move the experience design from passive (read, watch) to active (perform a function), and eventually to interactive (cause and effect, reciprocity), all the while contributing to immersion.

  • Zone 1: Foundations and Dislocation. Canada was established as a country in 1867 and in this zone, which runs through 1914, visitors find themselves in an environment of wood and paper. Typefaces of the era are used for text, while images are framed in wood, and letter blocks are used scenographically. The media relied on to present content is the printing press, letterpress, and magic lantern projections. The passive experience is immersion in material and aesthetic of the period–reading, watching, and viewing in an environment of muted colors, paper posters, and wooden construction. The active experience in this zone, the natural interface, is as simple as period-recreated news and paper prints which visitors flip through.
  • Zone 2: Transformations and Interventions. Entering the era of 1915 through 1960, wood gives way to steel and rivets, and newspapers, pulp magazines and posters give way to radio. Just as visitors accessed content in zone 1 by flipping pages, here they access content by tuning radio dials to different media clips, in an environment of dark colors and industrial steel.
  • Zone 3: Towards the Charter. The era of 1960 through 1982 is one of colourful plastics, protest, and television. It is vibrant, messy, and alive. The active experience has visitors accessing content by turning knobs on televisions.
  • Zone 4. Human Rights in Contemporary Canada. Ranging from 1982 through the present and into the future, this zone sees medium and material converge into the one element that both delivers information and is presentation: light. This zone contains passive, active, and interactive experience. Visitors read, watch, and view; they also provoke changes to their environment through a natural interface that is in this case intangible. Three individual spotlights each contain a hashtag, and based on the tag, invoked by standing in the spotlight, the environment responds to the specified prompt by altering the color and behavior of lights, as well as the content that is presented through environmental projections, wearable technology, and 3D virtual artifacts.
  • Zone 5: Defending Sovereignty. The Past and Present zone is somewhat unique in context, as it is less “era” specific and examines the concepts of sovereignty and reconciliation of indigenous nations and their interactions with the Canadian state over the 150 year context of the exhibition. In this zone, the materials are the four sacred plants (red cedar, sage, tobacco, and sweetgrass), and the designed environment emotes a longhouse. The natural interface in this zone is also intangible. The visitor can affect environmental change and vary the content presentation via use of sound, reflecting the oral traditions of the cultures whose stories are being presented.

With all exhibition design and development in-house at the Canadian Museum for Human Rights, a transmedia and mixed-media storytelling approach is employed. It is never about the object or the media installation, it always about how all media and elements work together. Offering a variety of experiences, interfaces, and entry points to the content increases the appeal to various audience demographics (each member having their own personal areas of appeal and interest) and increases engagement overall.

The design intent of “Rights of Passage” is clear and purposeful–immersion would help engage the audience in the stories being presented, contribute to relevance, vary the experience design scenario between passive, active, and interactive experiences, and ultimately facilitate learning objectives. The execution of this design into production, fabrication, software development, and finally into presentation, was ripe with challenges.

Production considerations

The Canadian Museum for Human Rights employs a strict inclusive design methodology (Bahram et al., 2016). The development of the the exhibition and each of its unique, natural interfaces presented an opportunity for the design teams to imagine how inclusion and access would be executed in such one-off scenarios.

Given this exhibition exists within the continuity of the Canadian Museum for Human Rights 4700 m2 (50,000 ft), attention had to be given to this context. Accessible solutions could be new or different, but the impact on visitors having to learn how to access something again, once they have already learned how in previous galleries, would have to be mitigated, and considered in detail. There might be a good reason to do something differently, so long as this decision was deliberately made with ample consideration.

As such, how to make a newspaper, a 3D virtual artifact, a social media enable dress, a 1970s television, or a 1930s era radio accessible, were all good challenges to engage in from the outset.

In order to address inclusive/universal design and accessibility, along with seamless integration of media installations within the fabric of the exhibition, an iterative design approach was followed that, like all CMHR created installations, ensures several rounds of design development, prototyping, and testing.

Natural interfaces

The natural interfaces in this exhibition served the roles of creating active and interactive experiences, increasing personalization, and reflecting the transmedia storytelling approach now characteristic of the CMHR’s exhibition style.

The 1800s Newspaper

While other content within Zone 1 is passively accessed (posters or magic lantern content viewed at a glance) the newspaper demands an action–flipping the pages. The active experience, a different behavior, immediately provokes an opportunity for further immersion in the content.

The newspaper was a simple interface to produce. Most people, even younger audience members, would be familiar with how to access content contained within the pages of a newspaper or pulp magazine.

Newspapers have become smaller over time, but this object remains true to the common size and style of the time.

Attention was paid to the typefaces used, size, and contrast, in order to be representative of the era yet also meet the strict graphic standards of the CMHR. Content of the newspaper is also available in text-to-speech via the museum’s mobile app. A Universal Access Point (Bahram, et al., 2016) is included in the installation.

A blurred hand is turning the pages of a reproduction of a newspaper.
Figure 7: tangible interface of period-recreated newspaper

The 1930s Radio

Zone 2’s 1930s radio interface, like the newspaper in Zone 1, demands that the visitor perform an action in order to access content. In contrast to the content of the newspaper, which is viewed, the content of the radio is heard. In addition to mixing the experience design, a new type of content (audio versus text and image) is presented.

Functional requirements led to the consideration of how tactile radio components would be replaced, along with providing media storage, language selection, volume control, and how closed captioning for audio content would be presented to meet inclusive design standards.

Procurement of a period-specific radio (circa 1939) led to rapid prototyping the replacement of the original radio components with an iPad to serve as the media player. The tubes, potentiometers, and other parts of the radio were removed leaving an empty chassis, which would accommodate the installation of speaker housing, along with a 9.7 inch iPad. While only a portion of the iPad’s screen would be visible, the mounting of the iPad within the radio would replace the glass panel that formerly displayed a printed graphic of the radio stations/frequencies, along with housing the needle indicator.

The three tactile buttons on the face of the radio would be replaced with new switches connected to an arduino controller, which was then connected to the iPad. A toggle to switch the language of the media from English to French, a dial to control volume, and a potentiometer to move the now virtualized needle displayed on the iPad.

Technical challenges included properly displaying captions for the audio. The face of the iPad needed to be a persistent image of the radio facade with indicators for stations/frequencies, and a needle graphic with a dynamic drop-shadow that responded to turning the potentiometer. The solution, which was to display closed captions of the audio files, meant that video files were served in the background to provide the necessary .srt files to dynamically display over the persistent background. With the ability to provide dynamic text using this method, we were also able to provide instructional text in a time-out state to ensure that when the radio was not being used, an attract mode would be present for visitors encountering the device.

By replacing the display and media server with one device, attached to tactile controllers, we were able to perfectly recreate a period-specific digital experience and provide a greater level of accessibility to the content.

A hand reaches in from the left to turn the knobs of a vintage radio.
Figure 8: tangible interface is a 1920s radio with iPad integration

The 1970s Television

The television interfaces of Zone 3 present video news clips of the era. While the viewing of the media is a passive experience, the visitor control and access to various content is done through the dials of vintage televisions. Like the natural interfaces of Zones 1 and 2, the visitor has to play an active role in accessing content, while true to the period itself, content is now available via moving images and audio.

Much of what was learned from prototyping the radio was reapplied in the research of creating a multi-screen interface, incorporating the partial facades of vintage television sets. The design requirements led to the selection of three television screens of various sizes, materials (wood, plastic, metal), and time periods, in an array that would allow one set of controllers to change the media program on all three screens at the same time.

After testing various modes of controls, it was decided that only two control knobs would be required. One to toggle language select, and the other to change the channel between ten media clips. Using only one controller that affected change across the exhibit element (multiple screens), would prove to create a more inclusive experience.

One of the largest challenges was how to recreate the look and feel of cathode-ray televisions with modern technology, along with considerations for inclusive design. Physical accommodation needed to be considered in the fidelity of the knobs and tactile feedback. Display of captions, as well as inclusion of sign language in English (ASL) and French (LSQ) also required consideration, given the form factor. Another challenge was recreating the curved distortion of the glass of vintage televisions.

Numerous attempts to replace the screens with high-resolution monitors behind various forms of plexiglass met with failure to deliver a satisfactory result. The fabrication team then tested a medium-scale vacuum forming machine to mold plexiglass to the original tubes of the television sets. The plexiglass was then sprayed with a reflective coating that would accept projection, and mini-projectors were placed inside the housing of the sets. The rear-projected plexiglass screens provided a glow and resolution that matched the period- specific television media perfectly, while still providing high enough resolution for closed captioning and sign-language interpretation.

By leveraging a 3D printer, we tested various switches that would meet CMHR’s accessibility requirements, ensuring that someone with mobility or strength issues would be able to turn the knobs effectively.

Three television screens take up most of the image. A hand reaches in from the left, turning the dials on the left-most television.
Figure 9: The tangible interface is a classic dial of the 3 period specific, synchronized televisions. These TVs were created using rear projection integration and computer control.

Hashtag hotspots

The hashtag hotspots are, in a way, one of the two most natural interfaces in the exhibition. This is given there is no tangible interface, only position-based interaction. As a visitor steps into one of the three designated spotlit areas, the environment around them responds to this action.

By creating a pedestrian interface, we do not require the visitor to use a personal device to engage, while still reinforcing the metaphor of the environment reacting to a relational network. The visitor can be present within the experience, and still understand its implications in the way we use personalized technology.

Leveraging an xBox Kinect for blob detection, we used an array of built and virtual objects to deliver the experience:

  • approximately 6500 LEDs woven throughout the walls of the space;
  • one projector directed at the floor to provide the illuminated hashtag hotspots;
  • one projector directed above the object array to display social media images;
  • two smart watches;
  • one network-enabled dress;
  • one hologram display (“pepper’s ghost” case);
  • one directional speaker.

For the digital experience in Human Rights in Contemporary Canada, we wanted to capture the imagination of the visitor. Creating an environment, multimodal in nature, that responded to a single input. The three hashtags projected on the floor are #reconciliation, #equality, and #environment.

For example, activating one of the hashtags would precipitate the following outcomes for stepping on the #environment hashtag:

The ribbon of light circling the hashtag sweeps across the floor to the dress, which in turn changes to the color green. The other hashtags on the floor disappear temporarily. The ribbons of light within the walls glow green along with the display case housing the watches. The smart watches display notifications (one english, one french) of a “march for science” near you. The hologram display lights up with a floating image of a bee colony buzzing within the artifact case. A short musical clip of a pop song related to respect for the environment plays from a directional speaker. On the faceted wall above the dress, a cascade of projected images from Instagram users leveraging the same hashtag appears.

The Facing the Future dress

The focal point of Zone 4’s Facing the Future installation is an interactive dress that responds to the hashtag hotspots. With Apple Watches and Fitbits, wearable technology is already in the mainstream. The Facing the Future dress intends to help the visitor playfully imagine how wearables may evolve in using fashion and technology to communicate how we feel, sense our surroundings, or speak up on behalf of something. Clearly, fashion has had the capability of accomplishing these things, but how could technology enhance this? Using open-source hardware, there is also a subtext of the importance of free and open access to sensing and communicating the world around us.

The dress incorporates manifestations of light through several materials. The skirt portion is a textile woven with fiber optic threads providing subtle illumination; fiber optic thread is a flexible, transparent fiber made from glass or plastic that serves as the backbone to the Internet by transmitting data via light over long distances (Hayes, 2015). While the dress uses the fiber optic purely aesthetically, it is a symbolic nod to the importance of this material to modern communications.

The dress incorporates close to 700 colorful LED lights. These lights respond to visitors activating the hashtag hotspots, with each hashtag activation designed to evoke a different emotion. For #equality, the dress explodes into a colorful rainbow marquee, representing the joy of freedom of self-expression. For #environment, the LED lights twinkle diffused green hues to evoke a connection with the calm beauty and expansiveness of the aurora borealis while also activating “roots” in the skirt made of green illuminated laser-wire to further this sense of connectedness.  For #reconciliation, the dress glows red with a subtle pulse of a heartbeat. This symbolism is inspired by The REDress Project, a public art installation by Jaime Black where red dresses, serve as an aesthetic response to thousands of missing and murdered Indigenous women in Canada (Black, 2014). The dress technology is driven by an Arduino Yun with a REST API that allows for easy integration within the overall gallery system.

The dress is intended to be beautiful and playful, but also to implicitly elicit the design thinking technique of “what if?” for what the future might bring in terms of wearable technology and a response to the human condition. What if it’s used to better connect creatively? What if wearable technology is used to more effectively speak out for what people believe in? What if it’s used to measure and respond to the environment by tracking data and information around the world?

Three circles of light on the center of a cement floor, surrounding by wall featuring exhibition text.
Figure 10: natural interface is spotlights with motion tracking and hashtag projection
A dress in the right hand side is glowing with lights that are embedded in the fabric.
Figure 11: Zone 4’s responsive environment. Depicting “#equality” mode. The dress, watches, 3D virtual artifact, projections, and lighting within the environment reflect the hashtag choice (#equality) by changing content, color, and style
A dress made of lights and fabric on the right-hand side is glowing and casting a green light on the wall behind it.
Figure 12: Zone 4’s responsive environment. Depicting “#environment” mode. The dress, watches, 3D virtual artifact, projections, and lighting within the environment reflect the hashtag choice (#environment) by changing content, color, and style
A dress made of lights and fabric on the right-hand side is glowing and casting a red light on the wall behind it.
Figure 13: Zone 4’s responsive environment. Depicting “reconciliation” mode. The dress, watches, 3D virtual artifact, projections, and lighting within the environment reflect the hashtag choice (#reconciliation) by changing content, color, and style
Two watches with computer screens for faces display dates and times.
Figure 14: close-up view of the watches responding to the #equality hashtag
Wide view of the exhibition space, with two curves walls receding into the center of the frame.
Figure 15: view of Zone 4’s lighting responding to #equality hashtag
A bee colony is displayed on a holographic device surrounded by printed exhibition text.
Figure 16: view of Zone 4’s 3D virtual artifact responding to the #environment hashtag


Audio beaded projection

Like the hashtag hotspots, the sound-enabled beaded projection is the very definition of natural interfaces.

The Speaking Volumes projection interactive reveals artworks created in collaboration with the Omazinibii’gig Artist Collective from Children of the Earth High School.

Within Zone 5, a graphic of a wampum belt is printed on the wall, weaving in and out of exhibit elements. The belt is made up of blue and white beads. When the beads enter the space where the audio beaded projection is, the beads transition from being printed on the wall to appearing as projected elements.

In the creation of the interaction within this zone, special consideration was given to the relationship of oral tradition within Indigenous perspectives—how oral history applied to the stories contained within this portion of the exhibit related to themes of land, identity, and environment. For thousands of years, Indigenous peoples have transmitted knowledge by speaking and listening. Oral history continues to be an important way of communicating teachings and preserving community histories. It was essential to design an experience that demonstrated a link between sound and the expression of ideas.

The projection is made up of approximately 5,000 individual beads. Based upon sound received by a microphone located above the projection, iconography is revealed as the beads change color and shape to display artworks based on the pitch, volume, and tone of the sound that is being emitted. Visitors can create sound through voice and through physical mechanisms such as clapping, encouraging experimentation and collaboration in triggering different artwork. The interface is pedestrian, intuitive, and immediately responsive.

A projection of coloured dots in the shape of a stylized turtle are present on a curved wall.
Figure 17: Zone 5’s audio responsive environment. Visitors make or speak sound and the projected beaded work responds by visualizing an expression. Visuals were created in partnership with indigenous youth working groups. (Turtle depicted in this photo)

Final thoughts

While summative evaluation is underway and has yet to be completed, early results have demonstrated an average dwell time in the exhibition of almost three times that of other installations exhibited at the Canadian Museum for Human Rights.

Mild concerns of user familiarity were pre-mitigated through directional and instructional text and signage. Observation has revealed some visitors don’t realize the televisions are actively controlled (visitors don’t realize they can “change the channels” through use of the knobs). This is something that will be addressed through added instruction.

The multimodal nature of the exhibit and universal design approach continues to push the boundaries of storytelling through the combination of tactile, audio, visual, descriptive, and virtualized interfaces.

Overall, the natural interfaces in Rights of Passage: Canada at 150 play an important role in building visitor engagement, and therefore facilitating the accomplishment of learning objectives. Their primary goal of contributing to the delivery of transmedia storytelling was realized through a careful but rapid design development processes and a direction to immerse visitors in the period relevant to the stories being exhibited. Having users play an active and interactive role in accessing content has so far yielded positive responses, longer visitation times, and greater immersion in the content being exhibited.


Bahram, S., S. Gillam, C. Timpson, & B. Wyman. (2016). “Inclusive design: From approach to execution.” MW2016: Museums and the Web 2016. Published February 24, 2016. Consulted January 13, 2018. Available https://mw2016.museumsandtheweb.com/paper/inclusive-design-from-approach-to-execution/

Black, J. (2014). The REDress Project. Consulted January 31, 2018. Availablt http://www.theredressproject.org/

Hayes, J. (2015). “Optical Fiber.” The Fiber Optic Association, Inc. Consulted January 31, 2018. Available http://www.thefoa.org


Cite as:
Gillam, Scott, Neely, Liz and Timpson, Corey. "Facing the future: Natural user Interfaces and transmedia spaces in museums." MW18: MW 2018. Published February 1, 2018. Consulted .