How do I manage test harnesses in Git - should they be in a separate repo?

Question

I have a large (ish) project [90 files, code 650kb], which I now manage in Git. I have several independent test harnesses used to test / test new low-level bits of computation, which are subsequently combined into the main code and its branches (currently via copy-paste!).

What is the recommended best practice for managing test harnesses?

Should they be in a separate repository or should I create an empty branch in the main repo to start it, or just create a branch of "Test Harness" and overwrite the old code?

It is expected that the checked code in the main branch will be clearly "the same" as the checked one.

I am on Windows (msysgit) and I am a lead researcher for using Git in a company.

Answer 1

The usual structure that I saw in most projects is to add the test/ directory hierarchy parallel to src/ and store them there (in the same repo).

Answer 2

90 files and 650 KB of source code are definitely small. It is better to save a test harness / test suite, etc. Together with the source code in the same repository. Check out some of the github repositories (ex: PLY ) and determine how you organize your source code and test suite.

Answer 3

The number and size of your files is within the git ability to save it all as one repo. Even if you pushed them an order or two. Therefore, the real reasons to split them into two repositories or save them in one repo should have been done with ease of use, and not with the technical limits of git.

I like to run tests in the same repo as the ones they test. When I update the code to include new features, I update the unit tests, and it's nice to have synchronization.

When I add code to fix a flaw and add a regression test that has two synchronisms, it’s nice again.

When testing for unit and regression in synchronization with the code, when I check the old revision, I know that all related tests must pass. Any errors that I can attribute to another component of the system (say, changing the OS or tool), which helps me point out such things without interference, to guess which tests might be “expected failures”.

The downside is that if I notice something missing from my unit tests, it's not easy to add it retroactive to where it "should have been." However, I find that there is a smaller flaw, after which there are many errors of “guessing that this might be normal” when you check if the last April code is working with some new subsystem or another.

Your trade-offs may be different. Perhaps your control chain does not provide enough support to add advanced unit tests with new features, so you may have a higher percentage of tests that you want to apply retroactively. You might be better off exporting functionality changes through some readable attribute, and your test suites might just not trigger the expected failures. Perhaps your tests are managed by another group, and then code. Any of them can change the balance.