Yes, USB drivers can usually be compiled for architectures other than x86. Of course, this assumes that your board has a host USB port. There are several boards that only have USB device ports, and many SoCs have USB ports and devices.
But successful driver compilation (USB) can only be part of the task.
Some (USB) devices may require additional library packages and other drivers to interact with application programs. For example, a USB digital TV tuner actually requires a lot of packages to work (V4L, ALSA, I2C driver, Userland firmware download).
Explanation
These additional dependencies that you may have to create are not USB related.
Dependencies are related to the type of device.
The Ethernet interface, regardless of whether it is integrated into the SoC or external via USB, is easily configured for full support in the kernel (e.g., protocol stack) and user area (e.g., Busybox has ifconfig , ping and routing applications). The PCI TV tuner will have the same dependencies as the USB tuner. But the embedded environment usually means that you do not have any / most of these already installed / installed multimedia dependencies.