sizeof it is an operand of the sizeof or unary & operator or is a string literal used to initialize another array in the declaration, an expression of the type "N-element array of T " is converted ("decays") to an expression of the type "pointer to T " , and the value of the expression is the address of the first element of the array.
If an array declaration is passed
int S[4][4] = {...};
then when you write
rotateArr( S );
expression S is of type "4-element array of a 4-element array int "; since S not an operand of the sizeof or unary & operators, it will be converted to an expression like βpointer to a 4-element array int β or int (*)[4] , and this pointer value is what is actually passed to rotateArr . So your function prototype should be one of the following:
T rotateArr( int (*arr)[4] )
or
T rotateArr( int arr[][4] )
or even
T rotateArr( int arr[4][4] )
In the context of the parameter list, the declaration functions of the form T a[N] and T a[] interpreted as T *a ; all three declare a as a pointer to T
You are probably wondering why I changed the return type from int to T As written, you are trying to return a value like "4-element array from a 4-element int array"; Unfortunately, you cannot do this. C functions cannot return array types, nor can they assign array types. IOW, you cannot write something like:
int a[N], b[N]; ... b = a; // not allowed a = f(); // not allowed either
Functions can return pointers to arrays, but that is not what you want here. D will cease to exist after the function returns, so any returned pointer will be invalid.
If you want to assign the results of a rotated array to another array, you need to pass the target array as a parameter to the function:
void rotateArr( int (*dst)[4], int (*src)[4] ) { ... dst[i][n] = src[n][M - i + 1]; ... }
And call him
int S[4][4] = {...}; int D[4][4]; rotateArr( D, S );