Team:
Date:

December 2025

Course:
Skills:
Tools:

This project focused on implementing a Deep Q-Learning (DQN) agent to solve the Taxi-v3 environment by teaching a taxi to pick up and drop off passengers as efficiently as possible.

Summary

We worked through the assignment code step-by-step, starting with the class-provided code and filling in each required component of the DQN process. This included building a small neural network to approximate Q-values, implementing an epsilon-greedy action strategy, and using the Bellman equation to update the model's predictions during training. We also generated plots to visualize rewards, epsilon decay, average steps, and loss, and built an interactive HTML dashboard to display the model's progress.

After testing both FAST and FULL training modes, we compared their performance, experimented with hyperparameter adjustments, and evaluated how changes in learning rate, gamma, batch size, and number of episodes affected the model's ability to learn the most efficient taxi routes. Overall, the project helped us better understand how each reinforcement learning component contributes to learning and how training time, hyperparameters, and neural network design influence real-world performance.

Model Code

Approach

Our group worked through the assignment chronologically to ensure that we understood each step of the DQN process. The first step was reading through the provided code from Professor Prescher to understand which parts of the problem were already completed and where we needed to fill in information.

The overall goal was to train a taxi model to pick up a passenger and drop them off as efficiently as possible. The taxi structure gets +20 points for dropping off correctly, -10 points for illegal pickup/dropoff, and -1 point per move — so over time, the model optimizes by maximizing long-term reward through fewer mistakes and shorter routes.

To create the model we builta small Keras neural network with Dense layers, ReLU activations, and an Adam optimizer with MSE loss to approximate the Q-function; implementing the epsilon-greedy policy to balance exploration and exploitation; predicting Q-values for the current and next states and using the Bellman equation to update them; and generating plots for rewards and epsilon decay per episode.

For the HTML dashboard, one of us had taken an HTML class this semester, so we used those lecture notes to structure the page and style it with CSS. We used Plotly.js for the charts, and Python's {} syntax to inject data directly into the HTML. We ran into trouble getting the file to load in-browser, so we pivoted to having the file automatically download instead, testing repeatedly to make sure the statistic cards and charts rendered correctly.

For training, we added a retrain clause so we could reset the saved model and test new hyperparameters without old data interfering.

Detailed Findings

1 / 3
Slide 1
2 / 3
Slide 2
3 / 3
Slide 3
FAST Model

The FAST mode ran 200 episodes at a batch size of 32. The gap between early and late average steps was only 2.48, showing minimal route optimization. Rewards per episode never rose above 0, and the epsilon graph decayed to just under 0.4 — still far from full exploitation, meaning the model was exploring randomly even near the end of training.

FULL Model

The FULL mode used a longer, more thorough training process — scaled down from 1000 to 375 episodes after repeated crashes. Average rewards rose from -679 early on to -31 later, a change of +647. Exploration decreased at a faster rate than in FAST mode, loss was lower thanks to a smaller learning rate and larger batch size, and steps per episode plateaued around episode 100 at under 25 steps on average.

Hyperparameter Experiments

Learning rate 0.0005, batch size 64: at 200 episodes, results resembled FAST mode but showed a larger gap between early and late rewards, and losses staying well under 0.25 — suggesting the model learned successful decisions despite still-negative rewards.

50 episodes, gamma 0.5, batch size 16, buffer size 1000, learning rate 0.005, epsilon decay 0.5: rewards rose from -161 to 108 (+58), with losses near zero due to minimal exploration. The lower gamma pushed the model toward short-term decisions, while the small buffer and fast epsilon decay led to overfitting — the model found decent routes quickly but plateaued rather than continuing to improve.

Future Enhancements

If we did this project again, we would adjust hyperparameters one at a time — learning rate, gamma, epsilon decay, batch size — to improve training stability and isolate each one's impact. We'd also increase training time with more episodes, better computing power, and model checkpointing to see whether performance continues improving or plateaus like the FULL mode did. Finally, we'd consider complicating the neural network itself, such as adding dropout layers or more hidden units, to help the model learn more stable Q-values.

Risks

Deep Q-Learning carries risks beyond those of supervised learning. The Bellman equation and Q-value estimation can overestimate value in continuous environments, and unstable replay buffers or targets can cause overfitting or divergence as the model grows. Balancing exploration and exploitation through the epsilon-greedy strategy is difficult to tune well, and the model remains sensitive to hyperparameter changes, shifts in the training environment, or the introduction of new states — any of which can cause it to "forget" prior learning. There's also the risk of the model exploiting loopholes in the reward structure rather than learning the intended behavior, if that structure isn't specific enough.

AI Acknowledgements

Initial code (Claude): used to summarize project scope, break down concepts like the Bellman equation, and assist with debugging when the model wasn't running continuously.