Leaked Documents Show How Amazon’s Astro Robot Tracks Everything You Do – incompetently

Amazon’s new robot called Astro is designed to track the behavior of everyone in your home to help it perform its surveillance and helper duties, according to leaked internal development documents and video recordings of Astro software development meetings obtained by Motherboard. The system’s person recognition system is heavily flawed, according to two sources who worked on the project.

The documents, which largely use Astro’s internal codename “Vesta” for the device, give extensive insight into the robot’s design, Amazon’s philosophy, how the device tracks customer behavior as well as flow charts of how it determines who a “stranger” is and whether it should take any sort of “investigation activity” against them.

[…]

The meeting document spells out the process in a much blunter way than Amazon’s cutesy marketing suggests.

“Vesta slowly and intelligently patrols the home when unfamiliar person are around, moving from scan point to scan point (the best location and pose in any given space to look around) looking and listening for unusual activity,” one of the files reads. “Vesta moves to a predetermined scan point and pose to scan any given room, looking past and over obstacles in its way. Vesta completes one complete patrol when it completes scanning all the scan point on the floorplan.”

[…]

“Sentry is required to investigate any unrecognized person detected by it or Audio Event in certain set of conditions are met,” one file reads. “Sentry should first try to identify the person if they are not still unrecognized for as long as 30s [seconds]. When the person is identified as unknown or 30s passed, Sentry should start following the person until Sentry Mode is turned off.”

A flow chart presented during the meeting explains exactly what happens when Astro detects a “presence” and how it is designed for “investigating strangers.” If a user has disabled “stranger investigation,” the robot will ignore a stranger. If it’s set to “Sentry mode” or a patrol mode, it will either approach the stranger or follow them, and begin a series of “investigation activities,” which Amazon describes as “a series of actions Sentry takes to investigate audio or presence while recording.” Generally, if Astro begins an investigation, it will follow the stranger, record audio and video of them, and then automatically upload a recording the user can view later.

[…]

Developers who worked on Astro say the versions of the robot they worked on did not work well.

“Astro is terrible and will almost certainly throw itself down a flight of stairs if presented the opportunity. The person detection is unreliable at best, making the in-home security proposition laughable,” a source who worked on the project said. “The device feels fragile for something with an absurd cost. The mast has broken on several devices, locking itself in the extended or retracted position, and there’s no way to ship it to Amazon when that happens.”

[…]

Another source who worked on the project mentioned privacy and navigation as chief concerns. “As for my personal opinions on the device, it’s a disaster that’s not ready for release,” they said. “They break themselves and will almost certainly fall down stairs in real world users’ homes. In addition it’s also (in my opinion) a privacy nightmare that is an indictment of our society and how we trade privacy for convenience with devices like Vesta.”

The source also corroborated that Astro’s facial recognition abilities perform poorly, which is concerning for a device designed mainly to follow people around and determine if they’re a stranger or not.

[…]

“The goal is to make Vesta an ‘intelligent robot,’ and allow some simple but magical interactions with people,” the social robotics document states. To do this, Astro needs to fully map a user’s home, creating a heat map of “choke points” and highly trafficked areas where the robot is likely to get stuck or “places where it will easily get hit by humans” such as hallways, doorways, and the kitchen.

A map of a user’s home, with “choke points” in red

Astro is supposed to learn over time, meaning that it must track what humans are doing, where they are going, and where they are likely to congregate.

[…]