Big PCIe DMA Linux x86-64

Question

I work with a high-speed serial card for high-speed data transfer from an external source to a Linux box with a PCIe card. The PCIe card comes with some third-party drivers that use dma_alloc_coherent to allocate dma buffers to receive data. However, due to Linux limitations, this approach limits data transfer to 4 MB. I read and tried to use several methods to allocate a large DMA buffer and could not get it to work.

This system has 32 GB of memory and works with Red Hat with kernel version 3.10, and I would like to make 4 GB available for continuous DMA. I know that the preferred method is scatter / gather, but in my situation this is not possible, because there is a hardware chip that translated the serial protocol into DMA out of my control, where the only thing I can control is to add an offset to the incoming addresses (t .e. the zero address visible from the external system can be mapped to the address 0x700000000 on the local bus).

Since this is a one-time lab machine, I believe the fastest / easiest approach would be to use the mem = 28GB boot configuration parameter. This works fine for me, but the next step to access this memory from virtual space is where I am having problems. Here is my code condensed with the relevant components:

In the kernel module:

size_t len = 0x100000000ULL; // 4GB size_t phys = 0x700000000ULL; // 28GB size_t virt = ioremap_nocache( phys, len ); // address not usable via direct reference size_t bus = (size_t)virt_to_bus( (void*)virt ); // this should be the same as phys for x86-64, shouldn't it? // OLD WAY /*size_t len = 0x400000; // 4MB size_t bus; size_t virt = dma_alloc_coherent( devHandle, len, &bus, GFP_ATOMIC ); size_t phys = (size_t)virt_to_phys( (void*)virt );*/ 

In the application:

 // Attempt to make a usable virtual pointer u32 pSize = sysconf(_SC_PAGESIZE); void* mapAddr = mmap(0, len+(phys%pSize), PROT_READ|PROT_WRITE, MAP_SHARED, devHandle, phys-(phys%pSize)); virt = (size_t)mapAddr + (phys%pSize); // do DMA to 0x700000000 bus address printf("Value %x\n", *((u32*)virt)); // this is returning zero 

Another interesting thing: before doing all this, the physical address returned from dma_alloc_coherent is more than the amount of RAM in the system (0x83d000000). I thought that in x86 RAM would always be the lowest addresses, so I expect an address less than 32 GB.

Any help would be appreciated.

Answer 1

Instead of limiting the amount of system memory with mem try using CMA: https://lwn.net/Articles/486301/

Using the CMA kernel command line argument allows you to reserve a certain amount of memory for DMA operations that are guaranteed to be continuous. The kernel will allow processes other than DMA to access this memory, but as soon as the DMA operation needs this memory, processes other than DMA will be crowded out. Therefore, I would advise cma=4G changing your mem parameter, but adding cma=4G to your cmdline. dma_alloc_coherent should automatically pull out of the reserved space, but you can enable CMA debugging in your kernel configuration to make sure.