Chelsea's Updates

Getting Started

The main goal for me this week was to become more comfortable using TensorFlow through various tutorials. I began research by first completing a simple tutorial for image detection. This tutorial retrained Google's Inception V3 model or MobileNet model to classify different types of flowers in an image. At the end I was able to give the model a photo of a sunflower and have it classify it correctly with 98% certainty.
The other tutorial I worked on is the one provided by our customer. The tutorial helped me learn how to use the Google Cloud Platform in addition to an introduction into object recognition. This one was a bit tougher to get working, as it was written to run on an earlier version of the git repo I had downloaded for TensorFlow. Luckily Donald found a way around this by resetting the model directory back to the correct version. With the help of this fix I was able to get it off the ground and training. I used TensorBoard to monitor the model as it trained. An example of how TensorBoard can be used to monitor learning progress has been provided below.

All in all this was a successful week and I look forward to seeing how we can build off of the provided tutorial to tailor it to our needs!

Data Collection and Experimentation

On Monday (2/12) I installed LabelImg, a program for annotating images. I fed it a few jpgs to test, drew some boxes around some playing cards, labeled them with the class "playing card", and then inspected the resulting xml files that the program output. I then started collecting data for our project by scraping about 70 images of the queen of hearts from the web and by taking about 30 images of the card with my phone. I've shared a screenshot below of what annotating looks like on LabelImg.

As just a small experiment, I fed this small data set into the RCNN model we used for the pets tutorial last week just to see what it did. I succeeded in getting it running and training, however it was having difficulty drawing boxes with the test images. I believe this is due to the small data set, I also only let it run for about 20 minutes before stopping it to save credits.

I believe this was a great week! The group is going to start with a data set of 4 cards and then test locally on it to learn more about what goes into feeding a custom data set to a model. Once we get those four cards working we plan to add more until we can get a full deck and train with a full data set of a deck of cards on the model using GCP.

Teaching an Old Model New Tricks

This week we worked further on data collection. Each of us collected 60 images each of 4 different cards (Queen of Hearts, Jack of Hearts, King of Hearts, and Ace of Diamonds to throw something different but similar in). From initial experimentation of training the dataset on the model, we ran into memory issues and learned that our input image size was too large. After some discussion about the possibility of an eventual mobile or web application, we decided to size down to about 300 pixels for the images to optimize speed. We resubmitted the new resized images and their annotations to our shared Google Drive folder and then focused the rest of our time on the upcoming midterm presentation. Adam got a draft of our powerpoint together this past weekend for us and Rupal sent out some edits today. We plan on meeting some time tomorrow (2/26) to discuss further changes to the presentation and to practice before our practice presentation at Presentation U.

Presenting: Project Changes and More Data!

This week we went to Presentation U! to get some feedback for our slideshow presentation. This went well, we received a helpful note from them to take one of our textually heavy slides and to try to make it more visual. I was able to create a more colorful and very visual slide that I think communicates to an audience better than the previous version. We also made some changes to our requirements and schedule this week. After receiving approval from our customer, we decided to scale back the amount of cards in our dataset so that we could also build a GUI. We now plan on training our model on 24 cards- 9, 10, Jack, Queen, King, and Ace of every suit. This set will allow us to change our original idea of a poker hand recommendation system to a euchre one since a euchre deck only requires the top five cards of each suit. We made changes to the Schedule and Milestones and the Calendar (If you look at the description for each week we give a list of that week's milestones and use an X to denote if that goal has been completed yet) portion of our website to reflect this change, as well as changes in our presentation. In addition to the presentation work we did this week, we continued to add playing cards to the card dataset: the Queen, King, and Jack of Diamonds and the Ace of Hearts.

Data Madness ft. Model Serving

These past couple weeks were spent preparing for our midterm presentation followed by a relaxing spring break! During our spring break, we each upped our usual 4 card annotation goal to 8 (all face cards of both spades and clubs), so now all that we have left to annotate is the 9's and 10's of each suit. In addition to this, I got to work on serving up a frozen model so that we could test it and start some application development. I successfully was able to create a server using flask that allowed me to upload via an html form a photo of a card that then consults our temporary frozen model and outputs an image with the predicted boxes along with saving an array of all the cards detected in the image to use later for recommendation functions. The next step is to clean up my code and document it, and then we will get started on furthering front end development and testing out the best coding strategy to optimize prediction speed. We should be nearing the end of our data collection (maybe a week away) which will allow us to begin training and then testing our final model that we'll use for our web app.

Finishing up Data Collection

This week the data collection chapter of our project officially came to an end for me. I wrapped up taking/collection the rest of my images and annotating them. It should just be about testing and app development from here out! I was hoping to get my server code up on our github this week, but I wasn't feeling too well so I look forward to wrapping that up this next week.

App Development

The bulk of this week was spent learning about Flask and how to set up jinja templates to use for our website. Since my very simple webpage/server I wrote using Python and Flask with some jinja templates in html was able to successfully use the frozen model, I built off of that this week. Some of my group members started writing some front end code to jazz our website up, so I have been integrating that in with the simple webpage as we go. So far we have an app that allows you to upload an image (the main page), then there's a waiting period while it performs the recognition (we plan to add a loading page for this time), then it outputs the image with boxes around it and a dropdown table of the cards it detected. The reason we chose the dropdown was so that the user could manually correct the cards if the model guessed wrong. I also spent time translating an About Me and How To Use page over to jinja from a mix of HTML, CSS, and JS.

Testing Plans

This past week we had our meeting with Paul (our instructor) regarding testing for our project. We received some really good feedback and were able to get a good idea of what needs to be done for testing our project thoroughly. To make sure that everyone is always aware of what needs to be done for our project at all times, I created a google doc with a checklist for the different components of our project that I'll be embedding in the website under the calendar. This allows everyone to be up to date with who is working on what and also for everyone to add things when we think of them. In addition to this, I've spent some time looking into the training settings we're using for our model. After we began training with our full set, Donald noticed some issues with our accuracy that we need to address. In particular, we were getting around the 80's for accuracy percentage when training on the full dataset, which is less than what we expected. We wondering if the model was over generalizing because we scraped images from the web that included some that were VERY different looking from our bicycle deck. To experiment, we removed those images and Don trained on the smaller set. As expected, we saw a rise in accuracy (I am assuming due to the fact that we were now training on only one type of deck (Bicycle) with just photographs, so it is easier for the model to generalize with) but after the accuracy rose to the 90's, we saw a steep drop off. These two training sessions concerned us a bit so we've been looking into our dataset and our training conditions. Since we are using mostly the same settings as with the tutorial we did, we're not actually sure it those settings are optimal for our project. Hopefully some research into what settings are available to us will give us an idea of what training conditions are best for us.

Preparing for Delivery and Presentation

This week was mostly spent discussing what is left to do for our project before we deliver it to our customer. We wanted to ensure that our expectations for what we expect to finish in the next couple of weeks are realistic so that we don't end up drowning in work finals week. We decided that for the most part we want to focus on documentation so that the next group can have a good idea of what's been done and what needs to be done still, with us also throwing some ideas in for them of how to complete some things. We finished everything that was required and even got into the extra web app development so we feel good about our project. Along with going over our loose ends we need to tie up, we also scheduled some times to meet next week to go over our presentation slides before our practice presentation this upcoming Friday.

Coming to an End

This past week we had our practice presentation at Marksbury which went pretty well! I plan to write out a few talking points before the final presentation this upcoming Friday so that I can be sure I'm going to go over everything that I want to-I feel like I always forget something important that I want to mention when I'm nervous! In addition, our team is planning on tying up our project's loose ends by Thursday night so that we can deliver our project to Paul and our customers at Lexmark on Friday after the presentation. I'm really excited to see what our customers have to say about our finalized project, I feel like we really got out what we put in with it. I really had such a good experience this semester working on this project, I feel like I learned so many new things and I can't wait to use all these new skills for projects and work in the future. In fact, I liked it so much that I am going to be looking into implementing more machine learning projects over the summer. I hope that our project is continued in the following semesters because I'd love to see it grow! Overall I've really had a good time developing this project with the rest of my amazing team.