Test emulator

Question

How should we organize unit tests and system tests for a program that emulates hardware (such as vmWare)?

Background:

Over the years, we have controlled the computer since the 1980s and related peripheral equipment and software. The system is critical for our customers and they do not want to replace it. Therefore, we decided to develop emulators for some hardware. The problem is that it is poorly documented on many thousands of pages of typewriter written text. Therefore, we try and develop mistakes.

Problem:

Currently, we do not have unit tests for the emulator, and system tests are very complex. It is difficult to check whether a complex OS works in all aspects by entering a text terminal and simulating data input from external systems. The only way we are testing right now is to add large inlet pressure from external systems (via X.25) and automate some heavy operations regularly. But you miss so often.

Answer 1

I used to work for a company doing something comparable. To test our system (which was actually a dynamic binary translator, not an emulator), we wrote a test structure that will run the same commands initially and translated, and then compare the results. We started to do this with user programs, but as more complex products were developed, we used the same methods to automate testing of emulated equipment. Roughly speaking, you want to write some programs that access this equipment and do something on it, dumping all the output to the terminal or somewhere in the log. Then run these programs on both sides (real and emulated) and compare the output. Depending on how accurate your emulation is, you may need some scripts to ignore certain parts of the output when they distinguish between addresses, hostnames, etc.

Another thing to consider when emulating equipment is state changes: a particular command can give the same output from both sides, but it can change the internal state in different ways. This may be difficult to foresee, but in general you need to identify the internal state that may be affected and unload it along with the output for each command.

Before we bought it, we started working on something even smarter, using kernel tracking tools to monitor the state of the OS / hardware in stages, when we ran our tests and then compared the sequence of steps between full-scale and translated runs. It was never fully developed, but it looked very promising.

Unfortunately, all this was internal and closed source code, so I can’t tell you everything that you can work with and play with, but the idea is very simple - we had thousands of automatic tests similar to this work for each assembly, our translators with very satisfactory results.

Edit: the more I think about this problem, the more violently I have to pounce on it. I do not assume that your project is open source, but if it is me, I would like to participate. Feel free to contact me if possible.