Machine Learning

My passion for generating models and understanding data has naturally lead to my interest in the field of machine learning. While pursuing my physics PhD, I also took several graduate level courses to learn the fundamentals of data science and machine learning. To explore beyond the mathematical background and toy problems that I learned in my courses, I worked on several projects during my free time to solidify my knowledge of the practice of data science. These hobby projects were inspired by my other hobbies, primarily playing board games with friends. Below you can browse several of these projects and the accompanying code repositories.

Incan Gold: Deep Counterfactual Regret with Monte Carlo Tree Search

Incan Gold is a very simple strategy game that requires players to balance risk and reward. Importantly, the rewards of future cards depend strongly on the actions of other players, since rewards are split among all of the active players. Although there is a large amount of randomness in the game, players rapidly discover a strong strategy due to the simplicity of the rules. But how strong is this strategy? Is there an optimal strategy for this game?

Steep: Imitation Learning to Complete Ski Races

Steep is an open-world winter sports game that allows players to participate in downhill skiing races, among other activities. Because of the open-world design, I was interested in trying to create a neural network that could complete the different races. I trained a neural network using imitation learning, creating training data from my own gameplay and augmenting it to create a sizable training set. How close to human performance can we achieve from a neural network trained using imitation learning?

Steep Part 2: Imitation Learning to Complete Rocket Suit Races

After spending considerable effort developing a data collection and processing pipeline, the ski neural network was able to perform well on a simple course. There exists a separate race mode in Steep using a rocket suit that allows a set of guidelines that can be toggled. However the rocket suit races require more controls to navigate the course and are much less forgiving. Is the additional visual information sufficient to allow a network trained through imitation learning to navigate the course?

Lost Cities: Using Deep Q-Networks to Learn a Simple Card Game

Lost Cities is a simple two-player card game that is not unlike competitive solitaire. Players are forced to invest in playing cards while lacking crucial information about future cards that may be drawn. To determine optimal play, I developed and trained a deep Q-network that learned to maximize its collected score through self-play. How well can a simple neural network perform with so much information unknown?

Lost Cities Part 2: Learning from Deep Q-Networks

Now that we developed a deep Q-network that achieved a mean score of 45 points in Lost Cities, what can we learn from it? We probe this network to understand the high-level strategy choices it makes, and we quantify the distributions of important metrics governing its strategy. To better evaluate its performance, we want to play against the network ourselves. Can we train an object detection architecture so that we can play against the deep Q-network using a game with physical cards?

Probing Bacterial Parasitism Using Multi-Agent Reinforcement Learning

Bacteria in their natural environment do not exist in isolation but with several other species. Much richer dynamics can be observed with additional species, such as commensalism and parasitism, which cannot be observed with a single species. In this work, we used multi-agent reinforcement learning to find the optimal policies of these bacterial agents in different environments. In particular, we try to quantify and understand the optimal level of antagonism for a given environment.