Thursday, July 21, 2016

How Team Delft won the APC 2016 (1): System Overview

When we decided to participate in the Amazon Picking Challenge, the first decision was to a) gather the best team possible, b) design the best possible system, to win. We would choose the best technologies and components available to build a reliable and efficient solution, as industry requires.
Team Delft has gathered academic researchers and industry professionals,
but overall passionate roboticists!
To make the baseline design decisions for our solution we analyzed the application requirements, studied the results from the previous edition of APC, and considered the expertise we had in our team. We aimed for a fast and robust system, capable of detecting failed picks and try them again. For example, instead of complex maneuvers to reach occluded products, our robot would simply move blocking objects to other bins.

Our solution was based on the sense-plan-act paradigm. This allows for optimal motions at the price of more precise sensing information. The loop for the Picking challenge proceeds as follows. First the order of the picks is decided to maximize final scoring and guarantee the fastest first pick possible (this proved key to win!). Then, for each item to pick the robot moves to scan the bin with the first target item with the 3D camera to: a) locate it, b) get the collision information to move inside the bin. Then the grasp planner computes the grasping strategy and grasp pose, and a motion plan is generated to approach, and retreat from the bin with the item. Following the motion is executed, including gripper configuration and activation on the way. State is checked to confirm a successful pick. If so, the robot moves to place the item in the tote, using a simple logic to drop the item in pre-defined locations without crashing fragile products. The flow for the Stowing Challenge is similar.

For the control software we used ROS-Industrial. We had extensive experience in the team using ROS-I for industrial applications. The use of ROS proved mandatory. ROS allowed us to quickly adapt, test and integrate extant software components, and develop the new ones that we needed  (e.g. a grasp planner for our hybrid gripper, or the application executive/coordination component).

We describe below our basic design approach and core decisions for each general issue n the APC challenge: robot, perception, grasping and gripper, motion and task planning. In following posts we will cover more details of our robotic system in each specific area.

After preliminary analysis, we concluded that maximum reachability in the workspace was a critical requirement for the competition. We chose a configuration with a SIA20F Motoman robot arm by Yaskawa mounted on a rail. The total 8 degrees of freedom allowed the system to reach all the bins with enough maneuverability to pick the objects from any location inside.

We decided to use two industrial stereo camera's with rgb overlay camera's. One camera for detection of objects in the tote and one on the robot gripper to scan the bins. Detection went in two steps. For object recognition we used a deep learning neural network based on Faster RCNN. Next we performed localization using an implementation of super4PCS to do a global optimization of the pose estimation. After this the estimation is refined using ICP.

Grasping and gripper
We developed a hybrid gripper customized to handle all the 39 products in the competition. It incorporates both suction, with 1dof for more grasping flexibility, and a pinch mechanism for the products difficult to suck. We designed an algorithm that auto generates grasp locations on the objects based on their position, geometry and environment constraints. Object-specific heuristics were also included after testing.

To make the robot motions as fast as possible, yet accurate enough for our sense-plan-act approach, we divided the problem into the motions outside the shelf and inside its bins. Outside there is no need for dynamic path planning: the environment is static and known a priori. We created a database of collision-free trajectories between the relevant fixed locations, e.g. image-capture and approach locations in front of the shelf’s bin and over the tote. To move inside the bins we simplified the dynamic path planning problem by using cartesian trajectories. To implement our solution we developed customized ROS-services on top of the ROS framework for motion planning MoveIt!

Task planning and coordination
To control the execution of our sense-plan-act overarching loop we used a state machine approach implemented with ROS-Smach.
We implemented a dynamic task-planning to decide in which order to pick the items and where to place them. It coded the competition scoring rules in logic rules, averaged with heuristics for the difficulty of picking the different items, also depending on the bin cluttering.

More details about Team Delft robot coming soon!

Twitter: @teamdelft_apc

Sunday, July 3, 2016

Team Delft wins the Amazon Picking Challenge 2016

Team Delft wins the Amazon Picking Challenge 2016

We are happy and proud to announce that Team Delft has won both the Picking and the Stowing competitions in the Amazon Picking Challenge. The competition was really intense (just check the results), and we have a lot of fun at RoboCup2016 with the rest of the APC teams.
Team Delft celebrates victory. So many people... try to find the robot!

The stowing challenge was easier than expected, with high scores, and an outstanding performance of Team Delft robot, that stowed correctly 11 products in less than half the max time allowed!

The pick task finals proved to be nerve wrecking. Both Team Delft and our colleagues in team PFN collected 105 points. According to the competition rules, the tie breaker would be the time to successfully complete the first pick; a ‘photo finish’!  The jury maintained the nervous expectancy till the very last minute in the awards ceremony, when they announced Team Delft . NimbRo Picking ended third.

Several news and tech sites have made echo of it:
For more details about the technology behind our robot success, keep alert to this blog in the upcoming days!

Tuesday, May 24, 2016

Team Delft finalist in the Amazon Picking Challenge 2016

We, Team Delft, are proud to be one of the 16 finalist teams for the Amazon Picking Challenge 2016 [video]. The challenge has two competitions: one for picking products from an Amazon Kiva shelf to fill in a tote, and the other one consisting of the reverse task of picking products from a tote to stow them in a shelf.

These tasks are representative of the current challenges that warehouse automation faces nowadays. The unstructured environment and the diversity of products require new robotic solutions. Smart mechanical designs and advanced artificial intelligence techniques need to be combined to address the challenges in object recognition, grasping, dexterous manipulation, and motion planning. The goal of Amazon Picking challenge is to promote innovative and open solutions for automation in unstructured warehouse environments. It aims at fostering research and collaboration between industry and academia.

Our team is formed by robotics researchers from the TU Delft Robotics Institute, and robotic engineers from the startup Delft Robotics. We are really excited about the challenge, to demonstrate how robotic technologies can solve the present automation challenges in industry.
Team Delft
Since qualifying, we at Team Delft have been developing an industrial grade robotic system for the challenge. It involves a 7 degree-of-freedom Motoman robot mounted on a rail, courtesy of Yaskawa (sponsor). Ensenso cameras from sponsor Imagining Development Systems will feed high quality 3D images to our vision pipeline for object recognition and localization using Deep Learning techniques. The team is fully committed to ROS-Industrial, an initiative TU Delft is leading with Fraunhofer IPA in Europe to promote the use of reusable and reliable open source software for advanced robot functionalities. Components for motion planning, robot control, grasping, and PointCloud processing are being integrated into a fault-tolerant control architecture for Team Delft's robot.

Team Delft is supported by the Factory-in-a-day EU project who is fostering flexible and smart robotic solutions for manufacturing and RoboValley, a unique network consisting of researchers, industry and government agencies who are working on the next generation robotics. 

The new flexible and reusable robotic solutions that we are developing will reduce costs, improve quality and increase productivity. Robotic technologies will also shift human labor from dull and physically demanding tasks to more creative activities, creating new jobs around robots as they were created before around computers or mobile phones. The Amazon Picking Challenge will be held in conjunction with RoboCup 2016 in Leipzig, Germany from June 30 to July 3, 2016.

Follow us at: @teamdelft_apc

For more information about the Amazon Picking Challenge, please visit