ABOUT WAYMO ESSENTIAL+
The Google master challenge is a design contest for graduate students majoring in human-computer interaction. The theme of this contest is to design a new mobility-based AI system for Waymo based on the methodology in Google's "AI Guide Book". Within three month, we designed a concept display plan including user research, machine learning database and interactive prototype, then we participated in the review of the Waymo user experience team. In the end, my design team and I won the best user experience award. I am very grateful to Waymo's UX leader Ryan, who has given me a lot of help during his time as a project mentor
Project background：
People + AI Research (PAIR) is a multidisciplinary team at Google that explores the human side of AI by doing fundamental research, building tools, creating design frameworks, and working with diverse communities.
Challenge & Outcome:
The research topic of this project focuses on how to use machine learning to solve a wide range of social problems. Under the influence of COVID-19, the safety issue of going out to buy necessities has become a burden for the people around the world. My design team chose this topic for the innovative design and conducted in-depth research on this topic.
My role：
I am responsible for the user research, interface and interaction design of this project. In addition to two interaction designers, there is an industry expert who has worked in Toyota's marketing department for more than ten years. The mentor for this project comes from Waymo's interaction design team.
AI-powered grocery delivery service keeps your kitchen stocked with just enough essentials
WHEN WE TALK ABOUT GROCERY STORE SHOPPING...
Keeping essentials stocked at home accounts for most time spent grocery shopping.
But it’s the least enjoyable part of the shopping experience.
MOST of the time is tedious and boring to grab necessities as usual
“It's boring to spend most of your time in the grocery store picking up the same things from the same shelves every time.” --- Jesse
FEW of the time is delightful to explore new foods at the grocery store
“Exploring new items in the grocery store with your family is a ritual that makes life more interesting.” ---- Hugo
COVID-19 PANDEMIC
72% of people are still hitting the grocery store;
83% of people have had issues with grocery delivery.
People need to minimize the time in stores and improve efficiency.
--- Bankrate.com & YouGov
USER NEEDS
Based on our research, we think it comes down to these 4 key needs, prediction, delivery, eating healthier, and controlling spending.
PAINPOINTS
Despite the best efforts of stores and retailers, it's still a pain to keep your kitchen stocked with the essentials.
AT HOME, it’s hard to…
-
Control different consumption rate of essentials
-
Remember what items are already used up
-
Plan when and how much to stock
ONLINE, it’s hard to…
-
Compare attributes (i.e.nutrition, price) of different items
-
Know accurate delivery times to avoid parcels being stolen
-
Always get the right item and brand
IN STORE, it’s hard to…
-
Go to same aisles to get same things as always
-
Track how much has been added
-
Stand in line
-
Balance overall nutrition
Now you have Essential, your AI-powered grocery delivery service that keeps your kitchen stocked with just enough essentials.
What do we mean by “just enough”? Perfect quantity, accurate delivery time, healthier contents, and prudent budget management. Every service that we provide will be just what you want, no data mining, no over selling.
BEFORE
Before Essential, grocery shopping, as demonstrated from user pain point, it was an exercise in boredom at best and frustration at worst.
NOW
Now you no longer need to experience that tedium. Instead, you can get exactly all the essentials you need every week at home, and then have an efficient and delightful time to explore new stuff in the grocery store.
How does Essential get us there?
First, there are two crucial pieces of information we need in order to set-up our model and better serve you--- your food consumption rate and available time schedule.
STEP 1/4
Tell us about your household structure
First off--- household structure, which is the most direct factor that determines your food consumption rate.
ONBOARDING-QUICKLY
-
Household structure
-
Eating habits
-
Purchase history
-
Smart home devices
-
Food consumption rate
-
Available time schedule
Welcome to the primary features of Essential:
Perfect quantity prediction and just-in-time delivery.
CORE USER SCENARIOS
-
Food consumption rate
PERFECT QUANTITY
Home page -
Essentials’ status check (current, past, future)
• Incoming delivery
• Remaining essentials
• Future consumption
PERFECT-QUANTITY
-
Perfect quantity
-
Perfect delivery frequency
CORE USER SCENARIOS
JUST IN TIME DELIVERY
Time scheduling
Besides the automated delivery time we showed earlier from your smart home devices, we also provide quick access to manually adjustment which is also a co-learning interaction between you and our model.
JUST IN TIME-DELIVERY
-
Available time periods
-
Easy time scheduling
-
Deliver whenever it is convenient for you
CORE USER SCENARIOS
HEALTHIER SUBSTITUTION
Biometrics & Diet profile
We want Essential to be a light-weight counselor that can embed to your daily life naturally, rather than a health monitor delivering too much pressure. Therefore, you can turn on or turn off the diet profile feature whenever you want.
HEALTHIER
SUBSTITUTION
-
Health condition
-
Food preference
-
Goal (optional)
-
Biometrics & Diet profile
-
Healthier substitution
recommendations
CORE USER SCENARIOS
PRUDENT BUDGET
MANAGEMENT
-
Budget range
-
Food expenditure
-
Budget reference and evaluation
PRUDENT BUDGET MANAGEMENT
Budget overview & evaluation
By referencing your budget information and our overall dataset, we can provide an average reference in terms of similar household structures’ food expenditure every month.
You can have a general idea of how you spend money on essentials without any pressure.
I think you wonder why we choose AI to provide these features. We think to provide better quality of service, we have to deal with varieties of food consumption your availability and health conditions.
AI is the best tool because it is good at recognizes and categorizes food consumption and storage by user group, personalizing foods recommendations and predicting when you should refill the stock.
HOW AI HELP US ACHIEVE CORE SCENARIOS?
ROADMAP
Toward a delightful grocery shopping journey, as phase 1 , we will help you get essentials more efficiently online. As mentioned in phase 3 and 4, in the future, we plan to improve in-store shopping experience with AR and offer more fun virtual shopping with VR. Let’s go back to ESSENTIALS and let me explain the system and data side.
.png)
POSSIBLE RATERS
-
Understand how to estimate food consumption rate
-
Possess a background in nutrition-related disciplines.
Nutritionist, Dietitian, Food Technologist, etc
HOW WE LEVERAGE AI
This is the overview, how we’ve leveraged AI including raters and dataset, starting from “4 user needs” from the left side and shows data collection, labeling, processing and feedback.
User Healthiness
Datasource
-
Exercise/lifestyle IoT/App
(Apple Health, Peloton, Shopwell, etc) -
USDA Datasets
(Food nutrition facts, Dietary guideline)
We also create a food healthiness dataset to recommend appropriate healthier substitutions for each of you.
Based on the nutrition data, the raters label the food health level and particular concerns of each food for certain types of people. Why do we label both of them? Because even though a food, for example milk, has health level A, it is not healthy for people who have lactose intolerance.
Food Healthiness
According to the dataset, the raters should be specialists of foods and nutritions such as nutritionists, dietitians and food technologists.
POSSIBLE RATERS
-
Understand how to estimate food consumption rate
-
Be familiar with North American foods and how they are stored
American homemaker, Chef, Tidying expert, etc
HEALTHINESS DATASET
We plan to use two datasets. First one is user healthiness for the purpose of personalizing our service in line with the your health. Even though there is lots of medical data, we decide to collect age, BMI, blood pressure and calorie consumption because they are minimum but many enough for the rater to assess the health level.
Food storage and consumption
-
User input
-
Partnering with Samsung to detect when items in the refrigerator have been depleted
-
Weight data from the refrigerator or food containers to see how fast foods are eaten
In addition, to grasp the food consumption and storage, we also plan to utilize images in a fridge that come from smart fridges or pictures that our user takes, because image recognition is one of what AI is good at.
Food storage and consumption
However, here is a problem. Situations in the fridge are very different in each household. As you can see in the 2 images on the slide, some people prefer to preserve foods in a container while others do not, like me. So,who can help?
The raters can help. They teach which foods are stored and how enough they are, considering your household structures.
FOOD QUANTITY DATASET
We would like to deep down into the details of the dataset and raters of food quantity and healthier foods.
First, these tables are 2 of 3 datasets of food quantity prediction. We need these dataset to calculate food consumption rate and predict quantity of next purchase. About purchase history on the top, we get when, how many and how much you buy each food. We assume that labeling is not necessary, because the machine learning model can mathematically predict when the foods would be depleted.
For example, for user A, the model can predict he buys one pack of eggs every 7days on average, based on the purchase history from Amazon.
Purchase history
Datasource
-
Purchase history from Amazon
-
Quantitative survey of 10,000 households across the U.S.
-
Qualitative research -observing different types of families
So far we have collected sufficient data to run our model, then how can we know if the machine made the right decision for each single user? And how to keep engaging users to input feedback so the future service can be improved. Here comes to the last part of the whole process---Getting feedback.
At Essential, we have two main principles to gather informative data from users.
First, we introduce the features within the relevant context. Second, we collect the feedback in a simple and engaged way.
FEEDBACK
-
Introduce features at the right moment
-
Collect user feedback in easy and gamified ways
IMPLICIT
Feature disclosure
Introduce healthy substitution and budget management features in-context
