导航

Dreamview for Teleoperation (Remote Control)

Introduction

When the autonomous car is running without any driver, it is paramount to add the extra layer of safety by providing teleoperation or remote control ability between the autonomous car and the control center staffed with well-trained teleoperators. Passengers in the autonomous car may need to communicate with the teleoperator about urgent situations. Safety operator also needs to have a full picture of the traffic situation the autonomous car is in before one can make the best decision. Therefore we will need at least some audio and video streaming functionalities. When the teleoperator chooses one of the emergency actions such as “pull-over”, the autonomous car also needs to respond with appropriate motion planning. We will also need to monitor network sigal strength as it involves remote data transmission.

Dreamview module tries to provide a web interface that helps to operate and visualize the teleoperation features. It defines some interfaces for data transmission so that users can design and implement their own audio/video streaming and network modules as long as it conforms to the interfaces.

Teleop Service

User Interface

There are two types of views supporting the teleoperation service. The first one, called Car Teleop, is run on the autonomous car, and contains minimal teleop features for the passengers to stay informed. The other one, called Console Teleop, is run in the control center with GUI for the teleoperator to observe the traffic situation and issue commands when needed.

Car Teleop

car_dreamview In addition to monitoring the status of modules working on the self-driving feature, we also monitor the status of network router, video encoder, and audio modules.

The passenger will be able to initiate audio communication with the teleoperator by toggling on the audio feature. Once turned on, it enables both microphone and speaker for conversation on both sides. The passenger can mute oneself if desired.

If the user decides to connect with more than one network modems to make sure data is never lost, we also show the signal strength of each modem in a list, as long as the information is provided in accordance with modem_info.proto.

If there is a pull-over or emergency stop request issued by the teleoperator, there will be a visual/audio signal for the passenger to be aware of the situation. Meanwhile an acknowledgement will also be sent back to the teleoperator automatically.

The router, video encoder etc. also resides in the autonomous car system. However the passenger does not need to have access to them. They should be toggled on either by default or triggered when the teleoperator requests for video streaming.

Console Teleop

console_dreamview Similar to the basic setup in teleop car view, we also monitor the status of additional modules such as network router, video decoder, image viewer and audio.

The teleoperator will be able to initiate audio communication with the remote passenger by toggling on the audio feature. Once turned on, it enables both microphone and speaker for conversation on both sides. The teleoperator can mute oneself if desired.

In order to get a full view of the traffic situation the remote car is in, the teleoperator needs to toggle on one or more viewing windows for video streaming. The request should automatically turn on the video decoder module in the control center, as well as the video video encoder in the remote car. Network router module should be automatically turned on when starting the teleop process.

Again if the user decides to connect with more than one network modems to make sure data is never lost, we also show the signal strength of each modem in a list. The modem information can be provided by the user according to the format modem_info.proto.

Based on one’s assessment of the situation, the teleoperator can issue either a pull-over or emergency stop request if necessary. Two separate visual/audio signals are provided to acknowledge this command has been issued by the teleoperator and then received by the remote car.

To provide a top-down view of the location of the remote car, we also supports expandable map using Baidu Map API.

User Commands

Dreamview module only provides an interface for teleoperation. It needs to work with a few backend processes to complete the desired audio/video streaming and emergency operations. Here is a list of possible hardwares needed and processes users need to implement in the programming language they chose.

  • Network Interface Card

  • Modem

  • Router

  • Video encoder/decoder

  • Image viewer

  • VOIP encoder/decoder

As you would expect for Apollo, most messages are communicated via cyber channels. So the interaction between Dreamview and user-defined backend processes are also expected to be done via cyber messages. An example can be seen in daemon_channels. Similarly, information about the status of modems needs to be sent to Dreamview via cyber messages example.

Send Audio Stream

  • When “Audio” switch is toggled, both the speaker and microphone are turned on/off for the autonomous car as well as the control center.

  • The backend process is not implemented or provided by Dreamview. Users can implement their own application for audio streaming with their choice of programming language and library. Then they can make use of the daemon_cmd.proto and the corresponding function SendAudioStreamCmd in teleop.cc to start/stop the audio streaming process.

Send Mic Stream

  • When the button with “Microphone” icon is clicked, only the local microphone is turned on/off.

  • The backend process is not implemented or provided by Dreamview. Users can implement their own application for audio streaming with their choice of programming language and library. Then they can make use of the daemon_cmd.proto and the corresponding function SendMicStreamCmd in teleop.cc to start/stop the local audio streaming process.

Send Video Stream

  • When “Video” switch is toggled, both the video encoder on the remote car and the decoder on the control station should be turned on/off.

  • The backend process is not implemented or provided by Dreamview. Users can implement their own application for video streaming with their choice of programming language and library. Then they can make use of the daemon_cmd.proto and the corresponding function SendVideoStreamCmd in teleop.cc to start/stop the video streaming process.

Send Emergency Stop Command

  • When “Stop” button is clicked, a pad message with DrivingAction of STOP, defined by Protobuf message PadMessage in pad_msg.proto, will be published by Dreamview.

  • Upon receiving a pad message with DrivingAction of STOP, Planning module will execute emergency stop by abruptly braking and stopping at the current position.

Send Emergency Pull-over Command

  • When “Pull Over” button is clicked, a pad message with DrivingAction of PULL_OVER, defined by Protobuf message PadMessage in pad_msg.proto, will be published by Dreamview.

  • Upon receiving a pad message with DrivingAction of PULL_OVER, Planning module will execute emergency pull-over by safely pulling over as soon as possible.

Send Resume Cruise Command

  • When “Resume Cruise” button is clicked, a pad message with DrivingAction of RESUME_CRUISE, defined by Protobuf message PadMessage in pad_msg.proto, will be published by Dreamview.

  • Upon receiving a pad message with DrivingAction of RESUME_CRUISE, Planning module will resume cruising based on the current routing.

上一页

Customize Your Own HMI

下一页

Vehicle Modes