I have been looking for the same thing recently. Apparently av_read_frame internally calls a specific read_packet codec function for each codec / format. This in turn allocates memory for the packet and frees it up depending on the maximum limits that the codec / format creates. As long as you have the memory to support the actual stream (in), you should be fine, and the restrictions are specific to codec / format decoders. [You can find the function defined in AVInputFormat for each format as .read_packet].
source share