OAEP is defined by PKCS # 1, Section 7.1 .
OAEP requires the following parameters:
- hash function;
- "mask generation function", which can be considered as a hash function with unlimited output length;
- a "label" (arbitrary sequence of bytes).
There is only one defined mask generation function, called MGF1, and this function is built on top of the hash function. So your arg3 is the hash function that MGF1 will use. It may be the same hash function than the first (I'm not sure if it can be the same Digest instance in the Bouncy Castle API, I am mathematically speaking here). It could be another hash function.
A label can be used as a kind of difference between instances (for example, you can encrypt data with an explicit “target” encoded in the label). This is convenient in some mathematical proofs, but right now PKCS # 1 recommends using an empty string and doing it. For the purposes described in PKCS # 1, an empty label is no worse than any.
The decryption process must know these parameters in order to work. Usually they encode them in the structure that comes with the encrypted message and says: "it is encrypted using RSA / OAEP"; what is happening in the CMS .
If in doubt, use the same hash function as the first parameter for MGF1, and use an empty label.
source share