We’re almost halfway through the March, let’s see how the races are unfolding in the second AWS Machine Learning Community League update. This update has been prepared with use of AWS DeepRacer Race Data Project.
AWS DeepRacer
AWS DeepRacer is a 1/18th scale autonomous race car but also much more. It is a complete program that has helped thousands of employees in numerous organizations begin their educational journey into machine learning through fun and rivalry.
Visit AWS DeepRacer page to learn more about how it can help you and your organization begin and progress the journey towards machine learning.
Join the AWS Machine Learning Community to talk to people who have used DeepRacer in their learning experience.
Pro Division
Race info:
- Track named after the winner of 2020 AWS DeepRacer League
- 89.24 meters long
- 107 centimeters wide
- Multiple difficult turns with the most tricky one exiting a chicane past the hairpin in the upper-right corner, branded Yellow Brick Road since it looks like the beginning of Yellow Brick Read from The Wizard of Oz
- Head-to-bot format with four bot cars cruising at 2 m/s without lane changes
- Racers are trying to complete three laps
- Each collision and each driving off the track results in a five seconds penalty
- Top 16 go on into a live race which will be streamed on Twitch on the 8th of April to fight for places in the League Finals and AWS DeepRacer EVOs
Let’s see the stats for the whole group:
All submissions Update time: 2021-03-14 23:00:00 UTC Comparing to 2021-03-07 21:00:00 UTC Racers: 48 (+8) Total submission attempts: 22963 (+16519) Best total time: 01:28.167 (-00:07.433) by JJ Mean total time: 03:47.927541667 (-00:30.577708333) Best lap: 00:29.056 (-00:02.135) by JJ Mean best lap: 01:10.766375 (-00:09.066825) Mean average lap: 01:15.975520833 (-00:10.192529167)
Eight racers joined the race bringing the whole group up to 48 out of 75 eligible participants. I’m beginning to wonder if everyone is aware that they are part of the Pro Division. Well, they cannot race in the Open race now so maybe they just struggle to find time? At least we cannot complain about the present ones’ involvement with 16.5K submissions made during the week.
JJ still leads the group and had managed to reduce his total time by almost 7.5 seconds. The whole group has been busy this week, being able to reduce total time by a whopping 30 seconds on average. One would be tempted to say that they are closing in on James if they didn’t still go at less than half his speed.
A lot of mean improvement comes from people like me who start disastrously slow and then go still disastrously slow, but still quite a bit faster.
Let’s see the top 20(comparing this week’s and last week’s groups, not progress of racers currently at the top):
Top 20 submissions Update time: 2021-03-14 23:00:00 UTC Comparing to 2021-03-07 21:00:00 UTC Racers: 20 Total submission attempts: 19288 (+13342) Best total time: 01:28.167 (-00:07.433) by JJ Mean total time: 01:57.846500 (-00:27.138650) Best lap: 00:29.056 (-00:02.135) by JJ Mean best lap: 00:36.599900 (-00:08.197200) Mean average lap: 00:39.281800 (-00:09.046200)
Mean improvements in this group seem to be confirming my suspicion that we, the slow ones, have brought most time improvements this week. The top 20 are much closer to James with a mean difference of just under 30 seconds but they have made some significant improvements. I’m curious to see where the limit of DeepRacer’s capabilities lies for this track.
Finally, let’s see the standings (in parentheses change compared to last week, “-” means racer did not compete a week ago) :
| Rank | Racer | Total Time | Avg Lap | Best Lap | Resets | Submissions |
|---|---|---|---|---|---|---|
| 1 (↔0) | JJ | 01:28.167 (-00:07.433) | 00:29.389 (-00:02.477) | 00:29.056 (-00:02.135) | 0 (-0) | 1701 (+1280) |
| 2 (–) | RogerRabbit | 01:33.278 (–:–.—) | 00:31.092 (–:–.—) | 00:30.526 (–:–.—) | 0 (–) | 979 (–) |
| 3 (⬆2) | Jochem | 01:37.827 (-00:23.594) | 00:32.609 (-00:07.864) | 00:32.259 (-00:03.271) | 0 (-2) | 763 (+677) |
| 4 (⬆2) | Duckworth | 01:38.774 (-00:22.654) | 00:32.924 (-00:07.552) | 00:32.184 (-00:05.053) | 0 (-2) | 912 (+371) |
| 5 (⬇2) | AJM-Model-1 | 01:44.016 (-00:13.158) | 00:34.672 (-00:04.386) | 00:33.823 (-00:01.789) | 0 (-2) | 1302 (+1267) |
| 6 (⬆1) | PolishThunder | 01:44.028 (-00:22.105) | 00:34.676 (-00:07.368) | 00:32.252 (-00:01.945) | 1 (-3) | 2127 (+1294) |
| 7 (⬇5) | Ernesto | 01:48.410 (-00:01.077) | 00:36.136 (-00:00.359) | 00:33.341 (+00:00.202) | 1 (-0) | 1363 (+1012) |
| 8 (⬇4) | Maikel | 01:48.713 (-00:11.116) | 00:36.237 (-00:03.706) | 00:33.924 (-00:05.530) | 1 (-2) | 600 (+529) |
| 9 (–) | dartjason | 01:54.965 (–:–.—) | 00:38.321 (–:–.—) | 00:37.801 (–:–.—) | 0 (–) | 887 (–) |
| 10 (⬆5) | DBro | 01:57.446 (-00:45.026) | 00:39.148 (-00:15.009) | 00:33.013 (-00:15.457) | 3 (-6) | 256 (+209) |
| 11 (⬆1) | atnmn | 01:59.500 (-00:19.312) | 00:39.833 (-00:06.437) | 00:37.492 (-00:07.086) | 1 (+1) | 914 (+828) |
| 12 (⬇2) | kimwooglae | 02:07.988 (-00:05.877) | 00:42.662 (-00:01.959) | 00:42.399 (+00:00.331) | 3 (-2) | 2248 (+1260) |
| 13 (–) | blueberrypie | 02:08.178 (–:–.—) | 00:42.726 (–:–.—) | 00:42.602 (–:–.—) | 0 (–) | 261 (–) |
| 14 (⬇5) | Deepak-dpk | 02:09.617 (-00:03.288) | 00:43.205 (-00:01.096) | 00:42.595 (-00:00.417) | 0 (-0) | 1395 (+1143) |
| 15 (⬇7) | kimmizian | 02:11.647 (-00:00.732) | 00:43.882 (-00:00.244) | 00:42.596 (-00:00.800) | 0 (-0) | 1712 (+805) |
| 16 (–) | ep-dr-racer1 | 02:13.905 (–:–.—) | 00:44.635 (–:–.—) | 00:41.005 (–:–.—) | 2 (–) | 396 (–) |
| 17 (–) | Penguin-DNPds | 02:14.564 (–:–.—) | 00:44.854 (–:–.—) | 00:39.338 (–:–.—) | 6 (–) | 181 (–) |
| 18 (⬇7) | TonyJ | 02:16.228 (-00:00.000) | 00:45.409 (-00:00.000) | 00:40.240 (-00:00.000) | 2 (-0) | 7 (+3) |
| 19 (–) | rosscomp1 | 02:19.015 (–:–.—) | 00:46.338 (–:–.—) | 00:38.787 (–:–.—) | 4 (–) | 299 (–) |
| 20 (⬇6) | Mentaiko-DevelopersIO | 02:20.664 (-00:09.541) | 00:46.888 (-00:03.180) | 00:36.765 (-00:06.893) | 8 (-1) | 985 (+680) |
JJ’s improvement is pretty good but have a look at what’s happening in places 3-6. We can see total time improvements of 13-23 seconds with Jochem jumping into third just today. Some people just cannot wait to jump into the lead. Out of eight new racers 6 landed in top 20 and four directly in the top 16 zone. Pretty impressive stuff. Well done Roger who jumped into second this week, overtaking the eager four I’ve mentioned before.
It’s worth mentioning that there is only one racer in the top 20 that hasn’t improved his time. What’s up Tony?
Open Division
Open races will always be slightly easier than the Pro ones. Let’s see what’s on in March:
Race info:
- Po-Chun Speedway is a simplified version of the Pro track
- 68.68 meters long
- 107 centimeters wide
- Simpler and shorter than the pro track but still pretty challenging with a variety of straights and turns, some of them pretty tight
- Time Trial format
- Competitors need to complete three laps and total time counts for their classification
- Each off-track results in five seconds penalty.
- Racers fight for qualification to the Pro group which is won by the top 10%
- They will also receive a Pro Welcoming Kit which is a bit of a mystery
Let’s take a look at the overall stats:
All submissions Update time: 2021-03-14 23:00:00 UTC Comparing to 2021-03-07 21:00:00 UTC Racers: 561 (+258) Total submission attempts: 5087 (+3100) Best total time: 01:21.226 (-00:06.245) by GregRacing Mean total time: 05:24.561926916 (-00:03.867102787) Best lap: 00:26.263 (-00:01.464) by GregRacing Mean best lap: 01:42.198030303 (-00:01.264402040) Mean average lap: 01:48.186980392 (-00:01.289039410)
The number of racers has almost doubled, but I want to bring your attention to the mean total time and mean best time which have both been reduced. This is interesting news which have surprised me a bit. What we’ve usually seen in the past is quite a big amount of racers joining and submitting a rather poor first mode, and then not returning to improve it. This data is suggesting that the models submitted have not caused a lowering of the mean values. Either more racers are trying and improving, or there is a group that has put some impressive effort in their models and outpaced the slowness of one-timers.
GregRacing has returned to the first stop, congratulations! VaughanDiesel has been trying really hard and has improved but it wasn’t enough. It looks like the top performers are still gaining advantage over an average Open Division racer with both best total time and best lap time improvements being higher for the leader than for all the racers on average.
Currently we’re looking at 56 racers qualifying to the Pro Division. I’m sure we’re not done yet and that number can even double before the end of month.
Now Top 20 (comparing this week’s and last week’s groups, not progress of racers currently at the top):
Top 20 submissions Update time: 2021-03-14 23:00:00 UTC Comparing to 2021-03-07 21:00:00 UTC Racers: 20 Total submission attempts: 805 (+353) Best total time: 01:21.226 (-00:06.245) by GregRacing Mean total time: 01:56.006400 (-00:09.306450) Best lap: 00:26.263 (-00:01.464) by GregRacing Mean best lap: 00:35.876650 (-00:02.750500) Mean average lap: 00:38.668350 (-00:03.102250)
Similarly, top 20 racers are improving significantly faster than the rest of the group. Here we can see that they appear to be slowly catching up with the leader. An average top 20 racer has cut 9 seconds from the car’s total time week to week which means 7.4% improvement. Not great, not terrible. I think we will see even more next week.
Lastly, let’s see what’s worth noting in the standings.
| Rank | Racer | Total Time | Avg Lap | Best Lap | Resets | Submissions |
|---|---|---|---|---|---|---|
| 1 (⬆2) | GregRacing | 01:21.226 (-00:23.855) | 00:27.075 (-00:07.952) | 00:26.263 (-00:06.868) | 0 (-5) | 13 (+6) |
| 2 (⬇1) | VaughanDiesel | 01:21.557 (-00:05.914) | 00:27.185 (-00:01.972) | 00:26.603 (-00:01.124) | 0 (-0) | 218 (+135) |
| 3 (⬆295) | AIDeepRacer | 01:42.128 (-11:59.535) | 00:34.042 (-03:59.845) | 00:30.137 (-03:49.135) | 2 (-126) | 64 (+63) |
| 4 (⬇2) | satussy777 | 01:43.095 (-00:00.000) | 00:34.365 (-00:00.000) | 00:30.529 (-00:00.000) | 5 (-0) | 47 (+20) |
| 5 (⬆1) | Jerec | 01:44.638 (-00:16.412) | 00:34.879 (-00:05.471) | 00:34.335 (-00:04.760) | 0 (-0) | 19 (+15) |
| 6 (⬇1) | Redfalcon5 | 01:46.018 (-00:11.111) | 00:35.339 (-00:03.704) | 00:34.997 (+00:02.552) | 0 (-3) | 38 (+19) |
| 7 (⬇3) | jmp543 | 01:47.158 (-00:00.000) | 00:35.719 (-00:00.000) | 00:35.609 (-00:00.000) | 0 (-0) | 56 (+10) |
| 8 (⬇1) | TestVinodRacer | 02:02.146 (-00:00.000) | 00:40.715 (-00:00.000) | 00:38.675 (-00:00.000) | 1 (-0) | 43 (+14) |
| 9 (⬇1) | s1thind | 02:02.470 (-00:00.000) | 00:40.823 (-00:00.000) | 00:31.866 (-00:00.000) | 5 (-0) | 27 (+11) |
| 10 (⬆25) | muzi | 02:02.513 (-00:44.318) | 00:40.837 (-00:14.773) | 00:35.028 (-00:16.708) | 3 (-8) | 33 (+22) |
| 11 (–) | DanMeon | 02:03.909 (–:–.—) | 00:41.303 (–:–.—) | 00:40.574 (–:–.—) | 0 (–) | 11 (–) |
| 12 (–) | m-hosokawa-nes | 02:04.586 (–:–.—) | 00:41.528 (–:–.—) | 00:37.468 (–:–.—) | 2 (–) | 6 (–) |
| 13 (⬇4) | hypersky | 02:05.051 (-00:00.000) | 00:41.683 (-00:00.000) | 00:39.660 (-00:00.000) | 1 (-0) | 18 (+2) |
| 14 (⬆41) | jgamblin | 02:05.170 (-01:14.579) | 00:41.723 (-00:24.860) | 00:41 (-00:13.083) | 0 (-12) | 36 (+13) |
| 15 (⬇5) | WaveLabCup-test | 02:06.605 (-00:00.000) | 00:42.201 (-00:00.000) | 00:37.131 (-00:00.000) | 5 (-0) | 4 (-0) |
| 16 (⬇3) | drunken-monkey | 02:06.700 (-00:05.832) | 00:42.233 (-00:01.944) | 00:41.144 (+00:00.745) | 0 (-2) | 16 (+7) |
| 17 (⬇6) | Groovy-DNPds | 02:07.874 (-00:00.000) | 00:42.624 (-00:00.000) | 00:39.550 (-00:00.000) | 1 (-0) | 23 (+7) |
| 18 (⬇6) | Kalky | 02:07.901 (-00:00.000) | 00:42.633 (-00:00.000) | 00:40.704 (-00:00.000) | 1 (-0) | 12 (-0) |
| 19 (–) | js4064 | 02:09.176 (–:–.—) | 00:43.058 (–:–.—) | 00:40.991 (–:–.—) | 1 (–) | 73 (–) |
| 20 (–) | Ultrafast | 02:10.207 (–:–.—) | 00:43.402 (–:–.—) | 00:35.269 (–:–.—) | 5 (–) | 48 (–) |
Check out the number changes for GregRacing. Looks like his strategy was to sacrifice some of the speed in favour of stability. It also might mean that the model is overfitting around the average performance which has allowed to eliminate a few instability issues. We’ll see how that goes later in the month.
Pretty impressive progress for AIDeepRacer. What started with a competition for the slowest performance (fun fact: we’ve had a community challenge for a slowest lap in 2019) has jumped up to third and appears to have an appetite for a higher spot.
Well done muzi and jgamblin, pretty impressive time and standings improvements. A warm welcome to DanMeon, m-hosokawanes, js4064 and Ultrafast who have joined the race this week and got into top 20 already.
Quickly looking at the top of the results I can see that there is a 42 seconds gap between places 1 and 8 and then just 8 seconds difference between places 8 and 20. We’ve only had one racer join the top 8 group this week. Would this mean that the leaders of the race have been set in stone? Not really, but in the next week I would be expecting more movement in places 11-20 than 1-10. Time will show
Thank you for your time. If you’d like to learn some Machine Learning skills while racing for glory, be sure to check out the AWS DeepRacer Page for hints on how to get started. Be sure to join the AWS Machine Learning Community on Slack for more tips and tricks on how to improve your model. See you next week!