4U

Background

4U was conceived during the 2017 TartanHacks at Carnegie Mellon University. I was in a group with three other CMU students. We wanted to create a personalized AI assistant that would have profiles for individual users saved so the assistant could tailor its suggestions and help depending on who was using it. The features we added included: food suggestions, budget planning, and facial recognition.

We each divided up the tasks so we could work on different parts individually, then combine everything at the end. I primarily worked on using voice control to ask for top-rated food suggestions.

Technologies

We used the IBM Watson Python SDK for its text-to-speech and and speech-to-text capabilities, the Yelp API to access top rated restaurants, and OpenCV for facial recognition.

For my part of the project, I used the Watson SDK for the text-to-speech and speech-to-text functions so users could communicate their intentions verbally, and so the app could verbally communicate back as well. I also used the Yelp API to search for what the user wants to eat, then filter the related restaurants by rating and suggest the top five restaurants. Currently, the location is limited to Pittsburgh.

Challenges

The hardest aspect of this project was getting the audio functionality up and running—I spent the most time trying to get pyaudio to record and save the voice commands, as well as play back saved audio recordings and gracefully shutting down.

One of the other challenges we were unable to solve by the end of the hackathon was that OpenCV would only work on one of our group member’s laptops. This meant only that member was able to successfully run the finished project on his laptop.

The Final Product

Upon running the main script, the user must take a picture of themselves to either add themselves to the database as a new user, or for the application to recognize the user as a current user and ask them what they would like to do. Users can mention “budget” to log the amount of money they’ve spent that day, or say, “I want to eat __” and the app will suggest related restaurants at which the user may dine.

Source Code

Can be found on Github. (Added Disclaimer: As this was a hackathan project, the code is not the neatest or most organized! Please tread carefully.)