NDC's MapIt : Game based learning for deaf kids
April 2017 - Present
"Deaf children need to know how their rights and identity blend into real life scenarios. NDC's Mapit is a game of the visual novel genre; that provides a safe space for players to learn to self-advocate, fail and build assertiveness. It is also the first game to be fully captioned in American Sign Language (ASL)."
  • Team size
  • 9
  • My role
  • User reasearch and experience design.
  • Timeline
  • April 2017 - Present
  • Methods used
  • Exploratory concept (convergent & divergent) research, qualitative user research, storyboarding, game design, low and high fidelity protoyping, accessibility engineering, UI design, play testing
  • Client
  • National Deaf Center

The Challenge

Pepnet2's 'Mapit: What comes next' is an online educational resource for deaf children where they can learn about deaf rights, self-advocacy, self-empowerment, career planning and job readiness. Pepnet2 resources have now been decomissioned, this project is designed to serve as a replacement for Mapit's mission.

Pepnet2 is essentially a repurposed compliance training module with little thought towards its young deaf audience.

The challenge was to design a game based learning offering that would teach deaf kids their rights and self-advocating these rights in a fun and engaging way. Pepnet2's MOOC style teaching methodology with long video content and quiz based assesments make it feel a lot less engaging.

The user experience with Pepnet2 is impersonal and objective; does not capture the different voices and social nuances of real life interactions.

The team

The project Mapit is being built by the Simulation and Game Application (SAGA) Lab at University of Texas at Austin. The project is advised by the principal of the lab Prof. Paul Toprac and the co-principal of the lab Dr. Sara Abraham and Matt O'Hair (ex-co principal). The team consists of 4 artists, 2 writers, 2 technologists and myself on the user-experience and research. I also liaise with the National Deaf Center to manage stakeholders, provide updates and involve the NDC in key decisions.

A promotional shot of the Mapit game.

Early exploration

A good chunk of early exploration consisted of divergent thinking exercises. The objective was to explore different genres of games and educational games that were fun to play and taught the user something. Our explorations involved conversations about games members of the labs had previously played and liked, going over the past project the lab worked on and surfing the internet for interesting games.

Some serious doodling and sketching, I love sketching ideas indiscriminately on paper and pen/pencil.

The most important part of this exploration was to identify game mechanics that promote self-expression and agency for the players. Self-advocacy is a central focus of NDC’s mission.

After considering a variety of games that may provide an inspiration for possible game mechanics, zeroed down on a blend of RPG style world based and visual novel style games. With enough examples to draw from, we switched gears to convergent thinking. The goal now was to define our vision with examples.

Two games stood out as good examples to inform Mapit’s vision. Spent, a game of surviving on minimum wage in face of economic hardships - living paycheck to paycheck was an inspiration for Mapit’s non-judgemental tone in its in-game choices.

Fair Play, a game about discrimination and racial bias was an experience we drew from, for framing our game scenarios. It inspired the intensity and depth of Mapit - pushing the players to make tough choices independently.

We prepared a pitch-deck and presented it to the NDC. The concept was well received and we moved ahead with presenting a rough playable prototype in the weeks that followed.

Agency and self-advocacy

Games are a great tools for providing a safe environment for players to try different approaches to a problem, make stupid choices or even informed one. Games eliminate the real-life time gap between action and reaction, this is helpful when we are trying to teach abstract topics like social-skills, assertiveness which may have very slow and mixed consequences in the real-world. This video by Extra Credits succinctly explain why games are a great learning tool for developing agency, the effort behind Mapit is based on a similar philosophy towards learning.

A commitment to Inclusivity and diversity

From the very beginning, it was decided that the game will be designed to be inclusive to all audiences and work towards representing characters that reflect the diversity in our society. All game art would be developed with this under consideration, giving out realistic and relatable art; which meant no opulent mansions or private jets.

The representation of the player themself inside the game was a very challenging problem. We tried to obscure as many physical details of the player character as possible. As a result, it was relatable to no one; this a problem game designers often face.

Limits of personas

The hard of hearing and hearing impaired community has an added dimension of uniqueness over on top of the traits that make you and me different from one another. Small differences in hearing-ability have very large impact on the life-experiences and self-image of a person and it is very difficult to lump this population into brackets. Often designers use personas to represent homogenous user populations that share a common goal, trait or some point of reference - this homogeneity is very difficult to apply on populations with impairments since minor variations in the nature of impairment may lead to very different persons.

A hard of hearing person who uses an amplifier headset is very different from a completely deaf individual who again is different from a deaf individual who has been deaf since birth; in terms of both impairment and life experiences!

The user personas may lie anywhere on the 3-dimensional spectrum. There are the primary factors that influence our target user base.

We look at our target population on a spectrum of hearing impairment, socio-economic standing and age. Unifying the diverse experiences of deaf users by distilling their experiences into common themes such that the game covers the representation of deaf individuals across the hearing impairment spectrum. As such, we are not designing the user experience based on personas; we instead use a fluid representation of a deaf individual. The nature of the player character’s impairment or their physical representation is not discussed in the game - leaving this to the player’s own imagination.

"NDC is using "deaf" in an all-inclusive manner, to include people who may identify as Deaf, deaf, deafblind, deafdisabled, hard of hearing, late-deafened, and hearing impaired. NDC recognizes that for many individuals, identity is fluid and can change over time or with setting. NDC has chosen to use one term - deaf - with the goal of recognizing experiences that are shared by all members of our diverse communities while also honoring all of our differences." - National Deaf Center

Concept prototypes

Building upon our exploratory research up until this point, the next steps were to create a concrete representation of the vision for the game so far. The game is based on a choose your own adventure style RPG where the player must collect all badges while maintaining a healthy level of money, GPA and social score. Each of these metric were selected to ensure that the player makes well rounded choices and to demonstrate that a balance of work and leisure is needed. Badges strewn across challenges are awarded when a challenge is successfully completed, think of them as in game collectibles.

Simple concept art

You can see the three major metrics for money, GPA and social score. The score on the right hand side is for the badges you have collected so far.

The player wanders into the open world, seeking adventure.

The player can navigate into buildings and interact with the people inside.

Interactions are conducted in a classic RPG style interface.

Similarly, non player characters (NPCs) interact in an RPG style as well.

Players are rewarded for making the best choices.

Functional prototype

After the initial concept art was critiqued and improved on, we moved on to build a functional prototype of the game. This would be used to demonstrate the working vision for the game to stakeholders and also test the idea of the game with deaf children. The functional prototype would serve as a very early litmus test for a long term development of the game.

A screen grab of a rough-cut pilot game for demonstration and early user testing.

A visual novel style interaction.

Early testing and elicitation device

This functional prototype was vital in getting stakeholder opinion, it acted as an elicitation device for the stakeholders who would then critique and suggest improvements to the game. We also wanted to continuously test the game mechanics and interface on our target user group.

Early qualitative research

Early qualitative studies were designed to gather as much rich qualitative feedback as possible. Over all two extensive qualitative studies were conducted with deaf children, we were helped by staff at NDC in conducting and administering the tests.

We collected screen recording and webcam video of the participants to understand the emotional response to the game. This was followed by a short written survey and a focus group.
Early research did yeild some interesting results and these results are explained in the following section.

Problems with player representation

A common theme with player representation, also explored in focus groups.

Players wanted more customization and relatable graphics for the player. We have seen how this is very complicated in a game world based in real life - where we must consider all varieties of deafness and human traits to achieve this. As a solution, all player representations were removed from the game. Navigation through the over-world will be done in a first person view only and RPG interactions will not show the player’s own character.

Cognitive load in text heavy gameplay

Because the game was heavily text based, children found it difficult to follow the complex dialogues and make accurate decisions. NDC recommended that the text in the game could be supplemented by American Sign Language videos. These ASL based video captions would let players with poor reading skills grasp the concepts being discussed in the game.

This would set the stage for making Mapit the first ASL captioned game.

User research : ASL integration

American Sign Language is a form of language practiced and standardized in the deaf community that expresses English in the form of sign language. ASL is made up of composite words, phrases, expressions and body-language that conveys meaning. It does not translate 1:1 to an equivalent English sentence - thus needs to be translated not captioned. Referring to the ASL videos in the games as ‘translations’ rather than ‘captions’ would be more appropriate.

Tradeoff between dialog length and ASL

ASL translations can pack a lot of information in a very short clip. It was decided that the ideal length of a dialog should be about 250 characters, which leads to a long enough ASL video while keeping the text only dialog digestible. This is rough pseudo-scientific estimate and has worked well in our experience.

Testing ASL integrated interfaces

To understand the how we can best re-distribute the screen real-estate to include ASL videos, I prepared a set of four ASL integrated prototypes. These prototypes demonstrate varying levels of real-estate dedicated to ASL translations.

Prototype 1: Video content is right oriented to suggest secondary importance at an offset to allow larger game art.

Prototype 2: Video content is centered in a more movie like representation.

Prototype 3: Video content is right oriented occupies smaller space and is strictly a secondary element.

Prototype 4: Video content is right oriented with its own special real estate and large size.

Prototype 4 wins, with a few tweaks

After testing these prototypes with users, prototype #4 was chosen as the best available representation of ASL videos in the game.

The users interacted with the game in a left to right reading pattern and preferred ASL translations over text. making the ASL video left aligned would ease this preference, as ASL (not text) was the primary source of information now.

User research : ASL narration styles

NDC was toying around with the idea of ASL translation in two ways.

  1. Translations per line of dialog : Every dialog in the game has an associated ASL translation that is played along with the dialog.
  2. Long form narratives about whole scenario : Every individual scenario in the game would have a single long form narrative that plays along as the user goes through the dialogs.

Having ASL translations per dialog would seem like an obvious choice to able-hearing persons, this is because we are not accustomed to using ASL the way deaf people do. Long form narratives help keep the cohesiveness of the ASL translation, where multiple breaks between ASL dialogs can be distracting. Another distinction is that ASL translations per dialog would need a unique interpreter to sign the ASL per game character. Also, the editing and packaging of these multiple videos (up to 30 dialogs in a single scenario) is an expensive exercise. NDC wanted to explore if narrative approaches were as effective as having ASL translations per dialog.

Informed by user research conducted, we choose to use per dialog translations instead of long form narratives in ASL. Even if this mean higher production costs and management overhead, this would provide the players the best user experience.

"I think the dialogs and videos should be in sync. If you had, make sure you had one sign vignette per dialog box. So you might have to edit her sign vignette to match the dialog." - Partcipant #11

Play testing and focus groups

Play testing is a user testing paradigm which is closer to the game design world than the user experience world. The objective of play testing is to uncover any design flaws in the mechanics of the game, any loopholes, any glitches in behaviour etc. This is a blend of usability testing along with an attention to the element of fun. The primary purpose of a game is have fun playing it. Regular play testing sessions are conducted on both deaf and regular audiences, because access to deaf minors or even minors is difficult to set up - we conduct playtests with young adults and older children. This is to ensure that storyline of the game is entertaining and keep you hooked.

A regular playtesting setup for small groups, each playtester plays the game followed with a focus group.

I run playtests in a casual setting, participants usually fill out a short written survey after playing the game to jot down any immediate feedback. It also acts as a primer for the focus group that would follow, where participants can use their answers to these qualitative surveys to participate in the focus group. I conduct focus groups in a professional friendly manner, this involves probing why player found certain aspect dull, difficult etc. The focus group are also a great way to understand the first time impressions of the game and we have gathered some very revealing research from these.

Written surveys on completion of playtests help prime participants on things to think about for the focus group.

Accessibility challenges in game development

We use Unity3D as the game development engine and environment. Unity has poor accessibility support, especially for text to speech and accessibility for native applications. Accessibility features have to be explicitly implemented by the game developers. The project is also federally funded and must comply with section 508 and Texas ADA laws, it is an enormous responsibility on my part to ensure that all kinds of players are able to enjoy and learn from this game.

Making the game accessible to people of all kinds of impairments is a priority. I am also working on building a re-usable open-source Unity plugin to facilitate accessibility in WebGl distributions of games.

Current state and future work

The game in its current state is far from done. While the storylines are being continuously worked upon and integrated into the game, the game needs a lot of art assets and renderings. On the overworld front; the game is not yet connected between the overworld and the RPG narrative. There is a lot of work remaining for the scoreboards and fine tuning the game mechanics for an exciting gameplay.

The game in its current state.

Future work also involves tangential projects like the Unity WebGl accessibility plugin, which is a moonshot project to help game developers around the world build more accessible games. As we go along, we would like to know more about the deaf community and design stories and in-game tasks that capture the daily struggles and victories of deaf people. I thank SAGA Lab and the National Deaf Center for giving me the opportunity to work on such an awesome project.