Your system is very unsafe, but I'm not trying to dissuade you or anyone else from playing with such things. You must go on. But itβs very important that you consider what you are creating in order to be a βtoyβ system that should never be viewed or advertised as βsafeβ.
Let me break the security question into two parts.
- How secure is key exchange?
- How secure is the encryption you use when you have a shared key?
Let me answer (2) first, as this will be the simplest. It will be terribly uncertain if you are not smarter than all the people who have worked and studied TLS over the years. TLS prior to version 1.2 (using multiple sites) is vulnerable to Chosen Ciphertext Attacks (CCA) in principle and to a BEAST attack in practice, depending on the choice of encrypted suit. And SSL 2.0 is more severely compromised.
The fact is that very smart people working on these protocols for many years were wrong. There is every reason to believe that I am working on these things myself, I will make huge mistakes. Basic encryption algorithms are in order. They are not broken. But the protocols.
So, if you have not yet studied and completely understood all the details of SSL, why they were there and how they were mistaken in some cases, then any protocol that you develop will almost certainly be terrible.
Now to the question (2). There are two problems with this. (a) Diffie-Hellman is not intended to provide the types of security that you probably need; and (b) I donβt think you implemented DH correctly.
2.a:
Key exchange Diffie-Hellman Key, when done correctly, is safe for key exchange, but it does nothing for authentication. This is why the question βsafeβ is often the wrong question. It is safe for some purposes, but largely unsafe for others, as it is not intended for other purposes.
As Josh3737 pointed out, the client and server do not know that they are talking to the right side. If Sam is the server and Charlie is the Client, there is nothing that prevented Mallory from creating his own server, which masquerades as Sam. So, Katie can exchange the keys with Mallory, thinking that she is talking with Sam. Mallory can pretend to be Charlie when talking to Sam.
Once he's set up that way, Mallory can act as the man in the middle between Sam and Charlie. When Charlie sends the data intended for Sam, Mallory decrypts it using the shared key between C and M, reads it (and possibly changes it), and then re-encrypts it with the shared key between M and S and sends it to S.
To solve the authentication problem, you need some kind of public key infrastructure (PKI), and this is really a pain. The System of Certification Authorities and such that with SSL / TLS is fraught with problems, but it remains the best system there.
2b:
A 512-bit public module along with 512-bit secret keys is not strong enough. DH keys should be larger. I would not go with something less than 2048 bits. You can leave with 1024 bits, you donβt worry that someone will be able to break secrets today in five years.
You did not provide enough information about how your primes were chosen. Not every prime minister will work. You need to use the "safe bar" for your module, otherwise there are shortcuts available for the attacker to calculate the discrete logarithm.