I assume that raw sockets are created at level-3 and therefore the protocol should not be IPPROTO_TCP / IPPROTO_UDP, but it should be IPPROTO_IP. Is this understanding correct?
No. You are right that raw sockets are basically layer 3 packets, but the protocol should not be IPPROTO_IP . The protocol argument in the case of raw sockets indicates what type of packets you want to receive on this socket. Remember that the protocol essentially performs demultiplexing at the transport level, so you need to specify what type of protocol your raw socket is interested in. This is clearly seen in man 7 raw :
All packets or errors corresponding to the protocol number specified for the raw socket are sent to this socket. For a list of allowed protocols, see RFC 1700, Assigned Numbers, and getprotobyname (3).
Since you are interested in receiving IP packets for a TCP connection, you should use IPPROTO_TCP .
But when I create a raw socket with a protocol like IPPROTO_IP (* socketFd = socket (AF_INET, SOCK_RAW, IPPROTO_IP);), creating sockets with an error Protocol is not supported.
Yes, this is expected: the IP protocol is not a layer 4 protocol. As I said, the protocol field is used for demultiplexing at the transport level, so it makes no sense to use IPPROTO_IP .
When I create a raw socket with a protocol like IPPROTO_RAW (* socketFd = socket (AF_INET, SOCK_RAW, IPPROTO_RAW);), my application does not receive any of the packets
This means that IPPROTO_RAW means that you are interested in sending all types of protocol packets (TCP, UDP, or any other protocol). But with IPPROTO_RAW you cannot do the opposite: IPPROTO_RAW means that you can get any protocol in this raw socket that is not supported. This is also shown in man 7 raw :
The IPPROTO_RAW protocol implies the inclusion of IP_HDRINCL and can send any IP protocol specified in the transmitted header. receiving all IP protocols through IPPROTO_RAW is not possible using raw sockets.
In other words, IPPROTO_RAW gives you the ability to send packets that match any protocol, but at the cost of not receiving a response. You could create other specific raw protocol-bound sockets to get the answers as workarounds, but this complicates the design because you have to manage the socket pool and this is definitely not what you want to do here.
When I create a raw socket with a protocol like IPPROTO_TCP (socketFd = socket (AF_INET, SOCK_RAW, IPPROTO_TCP);), my application receives TCP, but the kernel also responds to these packets (and in my case it passes an RST link). I assume this is because the kernel thinks that there is no one listening on the port this packet is destined for.
You cannot prevent the kernel from doing its work. From the raw sockets man page:
When a packet is received, it is sent to any raw sockets that have been associated with its protocol before it is transferred to another handler protocol (for example, kernel protocol modules).
So, you are right that the kernel sends the RST packet because it does not know the active TCP sockets or connections on the specified port. As I said, you cannot stop the kernel from doing your work, but a relatively quick (and possibly ugly) hack is to drop RST packets using iptables:
iptables -A OUTPUT -p tcp --tcp-flags RST RST -j DROP
Yes, not very elegant, but I think that little can be done here.
As pointed out in the comments, you can also create a fictitious TCP socket bound to the same port and address where you just received and dropped the messages. Thus, the kernel will not send RST responses, and you do not need to bother with iptables.
Also remember that since you need to specify IPPROTO_TCP for your raw socket, you must set IP_HDRINCL to the socket using setsockopts(2) so that you can create your own IP header.
Finally, make sure that this process has an effective user ID of 0 or CAP_NET_RAW (in practice: run it as root).