Let me go through phase 2 and phase 3 first
In step 1 of step 2, the iOS device will send a server response signed with the device’s certificate / key (each device comes with a pre-installed certificate / key that is different for each device). These device certificates / keys are issued by Apple.
On the server side, you should check it with Apple Root Cetificate.
In Phase 2, step 1-3, your profile service will send a SCEP request. This SCEP request contains information that allows the device to know which SCEP server it should speak to. This SCEP server is your server. Thus, the device will talk to this SCEP server and will request a new identity certificate from it.
In Phase 3, the response of the 2nd step device will be signed with the certificate / key of this identification certificate. And now you have to verify it with the root certificate of the certification authority. (Another note that the SCEP server in phase 2 is a proxy server for your certificate authority)
And now answering your questions: "MDM profile signature, which certificate to use?"
An MDM profile can be encrypted and / or signed.
If you want to encrypt it, you encrypt it using the authentication certificate associated with this device. So, a device that has a key for this identifier, therefore, can decrypt it.
If you want to sign it, you sign your server key. The device must have a server certificate installed, so it can verify the signature.
BTW. About this question. One thing that is not shown in this diagram, but is usually required - the first step (before the whole registration), as a rule, is to install a server certificate (for future verification of the profile signature). You can potentially skip this step if the server certificate is issued by a known CA (e.g. Verisign or something like that).
Let me know if you have any further questions. It took me a while to fully understand this OTA / MDM application.
Update 1
I don't know why iOS 6 treats your certificate as untrustworthy for signing. I did not work with certificates signed by well-known CAs.
I have only one guess. It is possible that between iOS 5 and iOS 6 they have changed something in relation to key chains. Generally speaking, each application has its own keychain. And all known certificates, I believe, should be stored in Keychain Mobile Safari. Perhaps MDM / Preferences shared this keychain with MobileSafari in iOS 6, and now they don’t share it. In this case, you will need to install this "CA Root CA CA high quality EV" through the profile (to put it in the correct keychain). However, this is a wild hunch.
As for encryption. First of all, you are right, if each device has its own private key, it is more secure. In this case, if someone steals the profile, he will not be able to decrypt it (because only for this device there is a private key). This is especially important if you send profiles that are sensitive (for example, an email account with a username and password).
Very high level of implementation in cryptography:
Any key (with any length) can encrypt data of any length. All encryption algorithms are designed in such a way that you can use the same key to encrypt any amount of data.
Asymmetric algorithms (e.g. RSA) are rarely used to directly encrypt data. In most cases, this algorithm is used to encrypt the key for a symmetric algorithm (as an example of AES), and all subsequent encryption / decryption is performed using AES. There are two reasons for this: performance (AES is faster than RSA) and resources (AES is less resource intensive than RSA).
So, as a result, if you need to encrypt a profile, you use PKCS7 , which internally uses RSA, AES (or other algorithms). Usually you have a library for this (OpenSSL or BouncyCastle). Thus, you do not need to find out all these difficulties.
BTW. If you have questions that are not suitable for SO, you can directly contact me (my contact information in my profile).