This "dead code removal" has already been done - in part - by the go tool. The go tool does not include all of the imported packages, only what is needed (or, more precisely, eliminates things that might not be possible).
For example, this application
package main; import _ "fmt"; func main() {}
leads to a minimum executable binary of almost 300 KB (on amd64 windows) compared to the following:
package main; import "fmt"; func main() {fmt.Println()}
Exceptional things include functions, types, and not even exported and exported variables. This is possible because even with reflection, you cannot call a function or "instantiate" types or refer to package variables by simply having their names as a string value. So maybe you should not worry so much about it.
Edit: With the release of Go 1.7, this is even better: read the blog post: Smaller Go 1.7 binaries
So, if you design your types and functions well and you don’t create “giant” registries where you list functions and types (which explicitly generate references to them and therefore make them inactive), the compiled binaries will contain only what actually used from imported packages.
I would not suggest using build tags for this problem. Using them, you will be additionally responsible for maintaining dependencies between packages and files that are otherwise executed by the go tool.
You should not create and separate code from packages to make your executable files smaller. You must develop and split the code into packages based on logic.
I would like to divide things into packages when it is really necessary, and import accordingly. Because this is really what you want: some code intended only for the client, and only one for the server. You may have to think a bit at the design and coding stage, but at least you will see the result (what actually belongs / compiles to the client and server).