IoT Devices + Mobile AR for Immersive Experiences
Research Engineer @ Snap Inc. | Project Lead
2022
Skills
HCI Research
Augmented Reality
IoT
UX Design
HTML & CSS
JavaScript
User Studies
Rapid Prototyping
Intern Mentorship
Study Analysis
How can we leverage the growing usage of smart home IoT devices to enhance AR experiences?
The recent increase of IoT devices in people’s homes, such as smart lights or smart fans, presents an opportunity to use smart home devices to extend augmentation to the physical world for co-located experiences.
In this project, I explored using a smart light as a core aspect of an AR experience and novel form of interaction by building an AR experience and running a user study.
What did I build?
Spooky Spirits is a mobile AR game for 2 user's to play together in person with a smart light. The system allows users to easily connect their smart light to the mobile experience via a web interface.
In this Oujia board inspired experience, users will direct their partner on how to complete the “summoning ritual” by guiding them through different full-body gestures. After their full-body gestures are detected through the body tracking ML system, users will ask a question aloud and the smart light will turn green or red representing the spirits answer to their question as yes or no respectively. For example, users might as “Will I win the lottery?” and the “spirits” will communicate their answer through the light.
To make the light a salient part of this experience, a key design decision was using the light not only as an enhancement to ambiance, but also as a source of information in the narrative. Additionally, when considering device arrangement, I chose to have only one user hold the phone so that the smart light acts as the sole source of immersion and augmentation for player 2. This design encourages users to work together and leverages the physicality of users’ space by augmenting their environment beyond pixels on a screen.
To make the light a salient part of this experience, a key design decision was using the light not only as an enhancement to ambiance, but also as a source of information in the narrative. Additionally, when considering device arrangement, I chose to have only one user hold the phone so that the smart light acts as the sole source of immersion and augmentation for player 2. This design encourages users to work together and leverages the physicality of users’ space by augmenting their environment beyond pixels on a screen.
How did I build it?
This experience was built as a Snapchat Lens using Lens Studio. Additionally, the system involves a web app for users to connect to their smart lights.
The details of the system consist of AWS Amplify running a Next.js app with 1) a server-side component that handles API calls to get and set the settings of the smart light and 2) a client side component that displays the website to the user (see Figure X). The Next.js app also polls from a DynamoDB database periodically to check for changes for each device, represented by the generated 5-digit code, and if there are changes, it sends the new data to the Kasa Cloud using the Kasa public API, which modifies the state of the physical devices. The Snapchat Lens will make get and set calls to the AWS Remote API using the entered 5-digit code to change the state of the device.
The details of the system consist of AWS Amplify running a Next.js app with 1) a server-side component that handles API calls to get and set the settings of the smart light and 2) a client side component that displays the website to the user (see Figure X). The Next.js app also polls from a DynamoDB database periodically to check for changes for each device, represented by the generated 5-digit code, and if there are changes, it sends the new data to the Kasa Cloud using the Kasa public API, which modifies the state of the physical devices. The Snapchat Lens will make get and set calls to the AWS Remote API using the entered 5-digit code to change the state of the device.