You haven’t specified exactly how you created the line items. So I'm going to suggest that you use Perlin noise to create height values on a height map. So, for any X, Y position in hieghtmap, you use the 2D noise function to generate the Z value.
So suppose your position is calculated as follows:
vec3 CalcPosition(in vec2 loc) { float height = MyNoiseFunc2D(loc); return vec3(loc, height); }
This creates a three-dimensional position. But in what space is this position? This is a question.
Most noise functions expect loc be two values for some specific floating point range. How good your noise function is will determine in which range you can transmit values. Now, if your spatial positions in the model space are not guaranteed within the range of noise functions, then you need to convert them to this range, perform calculations and then convert it back to model space.
So now you have a three-dimensional position. Converting the values of X and Y is simple (inverse to converting into a space of noise functions), but what about Z? Here you need to apply some scale to the height. The noise function will return a number in the range [0, 1), so you need to scale this range to the same model space on which the X and Y values will be displayed. This is usually done by choosing the maximum height and scaling the position accordingly. Therefore, our revised calculation position looks something like this:
vec3 CalcPosition(in vec2 modelLoc, const in mat3 modelToNoise, const in mat4 noiseToModel) { vec2 loc = modelToNoise * vec3(modelLoc, 1.0); float height = MyNoiseFunc2D(loc); vec4 modelPos = noiseToModel * vec4(loc, height, 1.0); return modelPos.xyz; }
Two matrices are converted into a space of noise functions, and then converted back. Your actual code may use less complex structures, depending on your use case, but the full affine transform is easy to describe.
Well, now that we have established that you need to keep this in mind: nothing makes sense if you do not know what space it is in. Your normal, your positions do not matter until you determine what space it is in.
This function returns positions in model space. We need to calculate the normals in the model space. To do this, we need 3 positions: the current position of the vertex and two positions that are slightly offset from the current position. The positions that we receive must be in the model space, or our normal one will not.
Therefore, we need to have the following function:
void CalcDeltas(in vec2 modelLoc, const in mat3 modelToNoise, const in mat4 noiseToModel, out vec3 modelXOffset, out vec3 modelYOffset) { vec2 loc = modelToNoise * vec3(modelLoc, 1.0); vec2 xOffsetLoc = loc + vec2(delta, 0.0); vec2 yOffsetLoc = loc + vec2(0.0, delta); float xOffsetHeight = MyNoiseFunc2D(xOffsetLoc); float yOffsetHeight = MyNoiseFunc2D(yOffsetLoc); modelXOffset = (noiseToModel * vec4(xOffsetLoc, xOffsetHeight, 1.0)).xyz; modelYOffset = (noiseToModel * vec4(yOffsetLoc, yOffsetHeight, 1.0)).xyz; }
Obviously, you can combine these two functions into one.
The delta value represents a small offset in the input space of the noise texture. The size of this offset depends on your noise function; it must be large enough to return a height significantly different from the height returned by the actual current position. But it should be small enough so that you do not stretch out from the random parts of the noise distribution.
You need to know your noise function.
Now that you have three positions (current position, x-axis offset, and y-offset) in model space, you can calculate the normal vertex in model space:
vec3 modelXGrad = modelXOffset - modelPosition; vec3 modelYGrad = modelYOffset - modelPosition; vec3 modelNormal = normalize(cross(modelXGrad, modelYGrad));
From here, do the usual things. But never forget to keep track of the spaces of your various vectors.
Oh, and one more thing: this must be done in the vertex shader. There is no reason to do this in the geometric shader, since none of the calculations affects other vertices. Let the parallelism GPU work for you.