Introducing AWS DeepRacer Log Analysis

We are pleased to present a new release of AWS DeepRacer Log Analysis tool. Learn what what it lets you do and how you can run it.

One of the most involving aspect of learning with DeepRacer is observing the impact of changes to the training and identifying what can be improved to progress. Without the ability to see that training becomes walking in the dark and can be both frustrating and wasteful.

Log analysis lets you read the training, evaluation or submission logs, load them into data series formats and process them to provide data in form of tables or plots. You can learn more about them in this video:

About the tool

AWS DeepRacer Log Analysis Tool is a set of utilities prepared using in a user friendly way that Jupyter Notebook provides. Jupyter Notebook can be thought of as a technical users’ word processor where a document can contain formatted text that can lead through the presented subject runnable code that can be executed and also altered to see what impact the changes have on the outcomes of execution, and finally readable code run outputs that are stored in the document so that it can be viewed in a read only mode without a need for a full runtime environment to see the results.

The initial version of this tool has been prepared and is maintained by the AWS DeepRacer Science Team as part of the DeepRacer Workshops Repository. I have started modifying it and then leading the Community effort to develop and maintain it.

We have now moved it into a separate repository called DeepRacer Analysis so that it’s easier to start using it.

If you would like to learn more about what changes have been applied to log analysis and what problems we intend to solve this way, have a read of this post on my personal blog.

Benefits of using the log analysis tool

The key aspect of log analysis is learning. It lets you learn on three levels:

• Racing – you are able to review what impact your decisions have on your training and when it is time to apply changes to it
• Machine learning – you can compare outcomes of various combinations of hyperparameters and get more in-depth with information from SageMaker
• Tooling – you use Jupyter Notebook and Pandas, analyse the data in series and not only, visualise it and extend the tools to provide answers that you are looking for; what is more, getting used to the notebooks will make it much easier for you to use notebooks with SageMaker, both for DeepRacer and other tasks.

Features

The tool will let you:

• Download the logs
• Load them into a data series representation
• Visualise progress of the training based on various criteria
• Observe the correlation between rewards and behaviours
• Compare what the rewards would be given if the training recorded had a different reward function
• Compare submissions to races
• View details of training in tables
• Inspect actions taken in different sections of the track

And some more.

Requirements

To be able to run the log analysis tool, you will need:

• Python 3.6 or newer – If you’re on Linux or a Mac you most likely have it already. It might be available as python3 but should also be accessible as python as more distributions remove the deprecated Python 2. For Windows you need to install it yourself. Once installed, go to the command line tool of your system and run “python –version” to check if it’s the right version
• Git client installed – again, on Linux it’s probably already installed, for Windows you can check this guide in the Knowledge Base
• AWS CLI configured to be able to download the logs
• Optionally Docker can be used

Running with Docker

Once you have the dependencies in place, follow these steps using a command line interface of your choice:

• Clone the repository using
  git clone https://github.com/aws-deepracer-community/deepracer-analysis.git
  or
  git clone git@github.com:aws-deepracer-community/deepracer-analysis.git
• Go into the project folder
  cd deepracer-analysis
• Build the docker image:
  bin/build-docker-image.sh
• Start the docker container:
  bin/start.sh
  for Jupyter lab use:
  bin/start.sh lab
• Open the notebook in a browser using:
  bin/open-notebook.sh
  if it fails, run this to get the url with a security token:
  bin/url-to-notebook.sh

Running on Windows or without Docker

Since we do not have Docker scripts prepared for Windows and some users may prefer not to use Docker, the best way to run the analysis is to follow these steps in your Command Line:

• Clone the repository using
  git clone https://github.com/aws-deepracer-community/deepracer-analysis.git
  or
  git clone git@github.com:aws-deepracer-community/deepracer-analysis.git
• Go into the project folder
  cd deepracer-analysis
• Create a virtual env:
  python -m venv venv
• Activate the venv, for Windows:
  venv\Scripts\activate
  For Linux:
  source venv/bin/activate
• Install dependencies:
  pip install  -r requirements.txt
• Start Jupyter Notebook:
  jupyter notebook
  or Jupyter Lab:
  jupyter lab

Further development

The project is far from complete and you should expect changes to appear on a regular basis. Three key goals at the moment are:

• Merging contributions from the community log analysis challenge – the first one has already been submitted and is being worked on to be included
• Reviewing the project structure and usability to make it even easier to work with
• Reviewing the notebooks and organising them in a way that will make them easier to start working with and progress to more advanced analysis

Resources

Most information about the DeepRacer Analysis is currently available in the project’s README.md file. With time additional information will be presented in the Community Knowledge Base.

Troubleshooting

If you face any issues when using the tool, come over to the Community Slack space, to channel #dr-training-log-analysis. You can also submit an issue on GitHub.

Acknowledgments

I would like to thank Lavesh Bhatia for assembling steps to set up the log analysis tool on Windows. Individual contributions like his are a foundation of the community’s growth from a group of individuals at the London Summit to rebranding as the AWS Machine Learning Community. Join us whether you need help or just want to have fun together.