I don’t know Maya, so I can only guess that its rotation looks like this: if you rotate left-right, it seems natural. Then, if you turn the object up 180 degrees, then turn left-right again, it still feels natural.
If you are familiar with the concept of using a matrix to perform transformations (for example, rotate, scale, and translate), then the quaternion is one and the same concept, but it only allows rotation, so you can use it to limit your transformation to just rotation. In practice, you can use either a matrix or a quaternion to do the same.
What you need to do is remember the current state of the quaternion for the object, then when the next rotation frame happens, multiply the new rotation by the old quaternion (in that order) to give you the next quaternion of the frame. This ensures that no matter what orientation the object is in, the next rotation of the frame will be applied from the viewer's point of view. This is in contrast to some naive rotation, in which you simply say: "the user scrolls up / down, therefore, changes the rotation of the X axis", which leads to a revolution.
Remember that, like matrices, quaternions must be multiplied in reverse order for the actions to actually apply, so I said to multiply the new operation with the existing quaternion.
To finish with an example. Let's say a user performs 2 actions:
- In frame 1, the user rotates the object 180 degrees around the X axis (up / down rotation).
- In frame 2, the user rotates the object 90 degrees around the Y axis (left / right rotation).
Suppose the object has a quaternion Q. Each frame, you will reset the object to its default coordinates and apply the quaternion Q to rotate it. Now you can initialize it with an identity quaternion, but let's just say that the initial quaternion is called Q0.
- In frame 1, create a new quaternion R1, which is the “180-degree rotation along the X axis” quaternion (you can find the math to calculate such a quaternion). First, multiply the new operation by the existing quaternion: Q1 = R1 * Q0.
- In frame 2, create a new quaternion R2, which is a “90 degree rotation around the Y axis.” First multiply the new operation by the existing quaternion: Q2 = R2 * Q1.
In frame 1, you will use Q1 to display the object, and in frame 2 you will use Q2. You can simply continue to apply any subsequent user actions to the quaternion, and it will always be rotated in the preview frame.