How to create a user interface for high latency conditions?

Work with an application that controls a remote robot, where there is the likelihood of a significant delay between the press of a button and the actual action. In addition, it is likely that the team was unable to successfully reach the intended recipient in the end (due to unreliability of the network, etc.). In addition, there are variables in the game whose changes are not instantaneous. For example, there is a variable for both a given speed and current speed; changing the set speed does not immediately lead to the fact that the current speed will correspond to this value.

The question is how to make the application reflect both the current states reported by the remote robot and the confirmation to the user that his team was understood by the application, but the system has not yet received a notification from the robot that it has been recognized? (Pop-ups are an absolute absence.)

Some ideas that were discussed:

Disable buttons

When the command button is pressed, start the timer for a reasonable amount of seconds and turn off the button during this time. Do not update the corresponding label directly, but instead wait for a response from the robot. (for example, if you click the "Speed ​​+" button, and on the right is a text label showing the current speed, do not immediately change the label, but rather wait for a response from the robot). When this answer occurs or when the timer expires, turn on the button again.

Pros: there are no additional management widgets on the page. Shortcuts always reflect the current state of the robot.

Cons: if you want to send two speed updates in a row, you have to wait until the first are received and confirmed. Will feel lethargic and unresponsive.

Registration Information

You have a log that users can view, which shows textual representations of all the actions that the user has taken, time stamps and clearly shows the history. Can be color coded based on user preferences.

Pros: The user has immediate feedback that his team has understood how it appears in the journal

Cons: does not solve the problem of what to do with the behavior of the button (especially with the switch).

Does anyone have experience creating user interfaces for environments in which there is significant latency between action and response? I would appreciate any input.

+6
user-interface
source share
4 answers

It looks like you have the following information that should be passed to the user:

  • The current state or value of the robot attribute.

  • The target (i.e., received) that the robot is looking for.

  • The setpoint value sent by the user to the robot.

  • The status of each team (waiting, received, achieved, timeout).

There are also several other considerations:

  • Continuous or discrete feedback. Do you have constant feedback of the current time from the robot? Or is it discrete feedback when the robot sends the current value only after reaching the target? Obviously, continuous is best for the user, as it allows the user to distinguish between a slow robot and a jam, but if you don’t have one, you can find a way to live without it.

  • Synchronous or asynchronous transfer of commands. If commands are sent synchronously (i.e., no new ones are sent until the last of them is received), then the user may need to (a) force out the next command without waiting for a response from the previous one (in case of loss of response) and (b) cancel the command in the queue in case of changing conditions between the creation and sending of the team.

  • A robot with or without conflict resolution. Does the robot have logic to look ahead in the list of received commands and resolve conflicting teams? For example, if its stop and its line of received commands includes a command for a 5 m / s transition followed by a command for a 2 m / s transition, is it smart enough to remove a 5 m / s command? Or will he first try to accelerate to 5 m / s, then 2 m / s, which can lead to overshoot? Will he wait until he reaches 5 m / s before he even “looks” at a command of 2 m / s? Lack of conflict resolution complicates your user interface because users may need to keep track of all the commands sent to understand why the robot behaves as it is.

Integrated information with positioning controls

Suppose the robot has conflict resolution and asynchronous command transfer. Instead of having separate controls for command, target, and current values, I recommend that you integrate them all into one control so that the user can easily compare current, target, and command values ​​and see discrepancies. Perhaps the best way to do this is to present values ​​by position in a window. This position-coding of values ​​is unparalleled in showing the relationship between things. There are two standard GUI controls, switches and sliders, which perform such position coding. However, you must increase them in order to fulfill them for your purposes. Better usability may require custom positioning controls where you schematically represent a robot, and possibly an environment, and allow the user to control it through direct manipulation. However, I assume that you need an easy-to-develop implementation, and a well-designed combination of sliders and switches can bring you closer to this ideal.

Use the switches to set the categorical value and the slider to set the numerical value. The slider may contain a text box that allows the user to fine-tune the value. You increase each of these controls to display the set, target, and current values ​​at the same time. The “normal” indicator (the point for the switch and the slider knob) represents the set value, and separate graphic indicators indicate the current and target values. Highlight the current from the target, making the current more visible. Id design them so that they merge into one pointer when they have the same meaning, to minimize clutter for the normal state of things. If your users are not trained in the system, you may want to include text labels in the pointers ("current" and "target" if the target is different from the current one).

Using these positioning controls allows the user to easily compare current, target, and command values ​​and see discrepancies. Status is implied in the relative positions of indicators. When the target pointer moves to the command position, the user knows that the command has been received. When the current pointer is on the target, the robot has reached the set value. This is especially good for continuous feedback of numerical values, because users can not only see the difference between the current and the target value on the slider, they can estimate how long it will take to reach the goal, seeing how quickly the pointer to the target lights up. For radio buttons, you can include the text "X% Done" with the Target pointer to indicate when the goal will be reached (if this information is available).

For the most responsive user interface, changing the value of a slider or radio button should send an immediate command. The "Apply" button is missing. Users can resubmit the setpoint at any time by clicking on the desired slider position or switch. I think you will find that this is a natural human tendency when faced with clearly non-responsive controls (consider the elevator buttons).

The failure between the target pointer and the command indicator may be too subtle to signal a lack of command reception if the responses are usually slow (within a few seconds, so the user's attention is likely to shift elsewhere). If this is the case, you might want to turn on the deadband alert after a timeout period, which almost certainly indicates that the command was lost during transmission. The modified alert may include a text alert located next to the control and / or graphically highlight the alert for a given purpose. Depending on your criticality, you can use an audio alert, such as an audio alert or animation, to speed up capturing the user's attention. The simulation warning automatically disappears when the target value matches for any reason.

Separate controls for team and current

If the sliders and radio buttons take up too much space for your purposes (or have other problems), you can use separate out-of-position controls for command and current values, which is implied in the design of Disable Buttons. However, in general, this is a more complex design with a lot of problems to solve.

I would like to use field controls such as text fields, check boxes, and drop-down menus, rather than command buttons, so that the set value is clearly displayed. Continuous numeric attributes may include spinning buttons with a text field if they do not clog the queue using incremental commands. As with the above option, changing the value sends an immediate command.

You may be worried about using a timer and turning off. In addition to the problem of creating a system sluggish, this means that you are gray from a command state. This can make reading difficult, and also requires some mental gymnastics, which the user can interpret ("it is not available, so I already selected it"). Interpretation can also be ambiguous, since often disabled means are not applicable (for example, speed control is disabled because the robot has lowered stabilizers for working with a fixed base).

The solution is to use a different graphical feedback than disconnecting. I'd stay away from color coding. Color coding tends to be arbitrary and thus confused (e.g., red queue average or timeout?). This may be one of the (rare) good places to use animation, as the animation is intuitive to represent the current process. A flashing or pulsating border (or other function) for a control with the given values ​​may indicate that the command sent is awaiting a response. A flashing / pulsating border to control the current value indicates that a command has been received and the robot is looking for the target value. If the animation is distracting in this situation (as in most other situations), then consider the dashed border (against the solid) to indicate a pending response or search for a target; the dotted line assumes a preliminary or transitional state.

Target value and status are implied animated. If the boundary of the command value is animated, the value inside is waiting - no response has yet been received. If only the current border is animated, the value inside the command control is the target value. If both borders are animated, then the robot is looking for one (unspecified) target value, and the other is waiting. If you think that the problem is that the target is not indicated in such circumstances, then you may need three controls to recognize the given, target and current. However, if this is a boundary case, it might be better to display the target value when moving from the current value or using the drop-down list.

If the feedback is continuous, you can also update the current value of the numeric attributes in regular increments (approximately every 200-500 ms) so that this animation gives additional feedback from the robot looking for the target. For any attribute, if the robot needs 10-15 seconds or more to achieve common goals, and the robot has conflict resolution, you can also display a progress bar inside or next to the current attribute so that the user can judge when the robot reaches the target value.

To resubmit a command, users can always re-select a value or press Enter while the focus is on the command control. This is a bit weird and inconvenient for some controls (e.g. checkboxes), so Id also considers the weak notification (rather than a popup) that appears next to the control if the command is turned off to respond. The notification includes a button to resend the command.

If your users are not trained in the system, you may want to include redundant text for the animation (for example, “Sending” while waiting for a response and “Search” when moving to the target value).

Log table

The approach to the log is probably best if the commands are sent synchronously and / or the robot does not have enough conflict resolution. Thus, the user can track the order of commands for sending commands or commands received by the robot, to predict the behavior of the robot. However, I would not make this a read-only text field, but rather a table that can be manipulated. While the table is sorted by default by timestamp, there will be separate sorting fields for the attribute, set value, status (waiting for a response, search, reached, timeout). If the feedback is continuous, then the status should indicate progress in achieving the target value (for example, percent or progress bar).

If there is synchronous transfer of commands, users can edit the command values ​​of pending commands or force send, move or delete pending commands. In any case, the commands can be copied and re-pasted to resend any command from any time. Maybe even provide tools to save the selected commands and receive and insert them later — now you have the macro tool.

If the robot tends to be especially moody (frequent loss of communication, slow responses), then you might want this log table to be next to the controls to create commands and view current values. The controls must be configured to make the discrete command understandable to the user. With the intricate computer-robot interface, false commands are expensive, so each command should be well planned and deliberate. This probably means a set of field controls, such as text fields and drop-down lists, for setting the values ​​of various attributes and a button that generates commands for these values. It’s awkward, yes, but it’s an accurate representation of communication with the robot.

Alternatively, if, as a rule, the queues are almost empty, you may want to make this table available in a separate window for experts to troubleshoot robot behavior. Typically, users use one of the other two options above.

+3
source share

I would not go to the magazine: the focus is on widgets. There are several methods for reporting status for different components, here I will talk about a simple one:

  • The button has a status icon next to it, showing its status. Use different colors to indicate latency. Green means ready, when the user presses the button, the icon changes to orange and indicates busy. When the user clicks again, the color changes to "red", which means the queue in the queue. When the queue is empty, the color changes to orange. If the action has been performed, the color will be changed to green.

  • The slider can be used for floating values: use two "slider". The first is better seen and you can drag it. The second and the layer below the first is the "actual reported value", which shows the delay.

  • Text input can also use green / orange status icons. When editing, the color changes to orange. If your queue / network protocol supports canceling editing actions, you can resend a new line each time the user presses a key. If not: change the icon to orange when changing, send a line and wait for the status report. The status report should contain the actual value, and if this actual value is equal to the value in the component, change the icon to green. If the actual is not equal to the value in the component, then resubmit the value in the component.

  • The radio beads / flags must have a dual display. One editable, one unedited. The first is for user input, and the second is for the actual state of the message. Same behavior as slider component.

This requires custom components or widgets. You can distribute original components or recreate them from scratch.

If your robot can also be "controllable":
Create a rectangle that you can drag. The rectangle has a small cross drawn on top of the current value. As you pull, you see the latency of the cross. You can use interpolation and time values ​​to smooth out robot control. The user will notice a lag because the cross "follows" the mouse pointer. (Often used in space shooter simulation games when controlling virtual ships, for example, Allegiance)

+4
source share

Perhaps you can use a variation of the command template. Each user action generates a command that goes into the queue. The queue is displayed to the user in a printout on the screen. Thus, you do not disable the button, but allow the user to press the button several times, but show the user that the command is queued. At the same time, you do not update labels showing the current state of the robot until you get the status from the robot.

In the queue, the team can somehow show their status, maybe text and color. And perhaps you should allow the user to delete the command before it is processed by the robot (if possible).

So, the queue might look like this:

  Command Status Result of Action
     speed + 5 pending speed will increase to 200 (Delete This)
     speed + 5 pending speed will increase to 205 (Delete This)
     speed-5 pending speed will decrease to 200 (Delete This)

etc.

+1
source share

I don’t feel it should be possible to turn off responsiveness. Everything on the Internet should be the right response and should always meet all the requirements, without exception, and we should always transmit the GDPR and compliance.

0
source share

All Articles