SubT Part 5: Results and the Future
This is the final post in our SubT series, in which we’ll cover the results and where SubT is going now that the competition is over.
The SubT competition consisted of three circuit events leading up to the final event. The Tunnel Circuit concluded on Oct 30 2019, Urban Circuit on Feb 27 202, Cave Circuit on Nov 19 2020, and the Final Event on Sep 24 2021. Results from each event can be found at subtchallenge.com/results.html. On the virtual track circuit event winners switched between Coordinated Robotics and BARCS, while Dynamo came from third place in Cave Circuit to claim the Final Event championship. If you’re interested in a more detailed view, then check out the leaderboard on the SubT Portal which contains results for all practice and competition worlds.
The Final Event was the culmination of multiple years of development for Open Robotics. Our goal was to make the Final Event challenging but not impossible. During the final event, each team’s solution was run against eight worlds. Final competition worlds 1, 2, 3, and 7 were constructed from well known tiles while worlds 4,5, 6, and 8 were mostly new. Scores tended to drop for the new worlds, which is not surprising. A few teams had robust solutions that generalized beyond the practice environments and managed to traverse and score points in these previously unknown environments.
Final competition runs started processing on Cloudsim on Aug. 27 2021 and completed on Sep 10 2021. During this period we ran over 200 simulation instances, reaching at one point over 1,000 cloud machines in active use. Of course, no software is perfect and we did encounter a few errors in both Cloudsim and team code. On the Cloudsim side. We encountered roughly 10 instances where our websocket simulation monitoring system failed to connect correctly, and an effort to replace websockets with gRPC is underway. Cloudsim ran on AWS, and 15 times the AWS APIs reported insufficient resources while launching simulations. These errors force an automatic restart for the simulation run.
The largest number of errors we saw were docker image evictions. These evictions occurred when a team’s solution would consume more memory than available on the host machine. Log file inspection indicated that there were likely memory leaks in the solution code. Out of an abundance of caution we re-ran these failures.
Extensive videos were produced during the Final Event. I encourage you to peruse the videos available at SubTV. You’ll find coverage of both the system track and virtual track, with commentary and team interviews. The following is the Final Event Day 1 coverage.
Even though SubT is over, it’s still possible to use SubT and contribute back. We even planted some seeds that will hopefully get you interested in extensions to SubT and Ignition. We recently added is a mine cart that rolls down a track based on the presence of a robot. A subway car with actuated doors now resides in an urban station tile. And lastly, a human model wandering around a warehouse-style tile has been incorporated. All of these additions are available in the Finals Qualification world. Feel free to open pull requests or issues to osrf/subt, or one of the Ignition repositories with your ideas and contributions.
In other good news, Cloudsim remains open for business. You can develop a SubT solution and run it on Cloudsim at app.ignitionrobotics.org/applications. This is now a paid service offered by Open Robotics. However, if you score a point on one of the qualification worlds, you’ll get access to a free version hosted on the SubT Portal. Time is short though, as the free option will expire at the end of July 2022.
Thanks for reading, and I hope you enjoyed this blog post series. Maybe we’ll see you at the upcoming VRX competitions!