Doctrine: Why can't I free memory when accessing entities through an association?

Question

I have an Application that is related to ApplicationFile :

 /** * @ORM\OneToMany( * targetEntity="AppBundle\Entity\ApplicationFile", * mappedBy="application", * cascade={"remove"}, * orphanRemoval=true * ) */ private $files; 

The file object has a field in which binary data is stored, and can be up to 2 MB in size. Repeating a large list of applications and their files increases the use of PHP memory. I want to save it.

I tried this:

 $applications = $this->em->getRepository('AppBundle:Application')->findAll(); foreach ($applications as $app) { ... foreach ($app->getFiles() as $file) { ... $this->em->detach($file); } $this->em->detach($app); } 

Detaching an object should cause the object manager to stop caring for this object and delete it, but it does not unexpectedly affect the amount of memory used - it continues to increase.

Instead, I have to manually download the application files (instead of extracting them through the association method), and the memory usage is not increasing. It works:

 $applications = $this->em->getRepository('AppBundle:Application')->findAll(); foreach ($applications as $app) { ... $appFiles = $this ->em ->getRepository('AppBundle:ApplicationFile') ->findBy(array('application' => $application)); foreach ($appFiles as $file) { ... $this->em->detach($file); } $this->em->detach($app); } 

I used xdebug_debug_zval to track links to the $file object. In the first example, there is an additional link somewhere that explains why a memory burst occurs - PHP cannot garbage collect it!

Does anyone know why this is? Where is this sitelink and how to remove it?

EDIT . An explicit call to unset($file) at the end of its loop is not affected. At the moment there are two more references to the object (checked using xdebug_debug_zval ). One of them is contained in $file (which I can disable), but there is somewhere else that I cannot undo. Calling $this->em->clear() at the end of the main loop also has no effect.

EDIT 2: SOLUTION . The answer from @origaminal led me to a solution, so I accepted his answer, rather than providing my own.

In the first method, when I access files through an association on $application , this has the side effect of initializing the previously uninitialized collection of $files on the $application object, which I repeat in the outer loop.

Calling $em->detach($application) and $em->detach($file) only tells Doctrine UOW to stop tracking objects, but it doesn’t affect the $applications array, which I repeat on, now you have a full collection of $files that eat.

I need to disable every $application object after I have finished with it to remove all links to the downloaded $files . To do this, I modified the loops as such:

  $applications = $em->getRepository('AppBundle:Application')->findAll(); $count = count($applications); for ($i = 0; $i < $count; $i++) { foreach ($applications[$i]->getFiles() as $file) { $file->getData(); $em->detach($file); unset($file); } $em->detach($applications[$i]); unset($applications[$i]); // Don't NEED to force GC, but doing so helps for testing. gc_collect_cycles(); } 

Answer 1

If we are talking about your first implementation, you have additional collection links in the PersistentCollection::coll Application::files property - this object is created using Doctrine on Application .

With the disconnect, you simply delete the UoW object references.

There are several ways to fix this, but you need to use a lot of hacks. The most enjoyable way is perhaps to disable the Application object and disable it.

Nevertheless, it is preferable to use more complex methods of batch processing : some of them were indicated in another answer. The current method forces the doctrine to use proxy servers and issues additional database queries to obtain files of the current object.

Edit

The difference between the first and second implementations is that in the second case there are no circular links: Application::files remains with the uninitialized PersistenceCollection (without elements in coll ).

To test this - can you try to remove the file association explicitly?

Answer 2

Cascade

EntityManager::detach should really remove all the links that the Doctrine belongs to. But it does not do the same for related objects automatically.

You need to perform the cascading action by adding the detach cascade parameter for the association:

 /** * @ORM\OneToMany( * targetEntity="AppBundle\Entity\ApplicationFile", * mappedBy="application", * cascade={"remove", "detach"}, * orphanRemoval=true * ) */ private $files; 

Now $em->detach($app) should be enough to remove references to the Application object, as well as its associated ApplicationFile objects.

Find vs Collection

I highly doubt that loading ApplicationFile objects through an association instead of using the repository for findBy() is the source of your problem.

I am sure that when loading through the association, the collection will have a link to these child entities. But when the parent object is dereferenced, the whole tree will be garbage collected if there are no other references to these child objects.

I suspect that the code you are showing is a pseudo-sample code, not the actual code in production. Please read this code carefully to find other links.

Clear

Sometimes it’s worth clearing the entire EntityManager and merging several objects back. You can try $em->clear() or $em->clear('AppBundle\Entity\ApplicationFile') .

Reset is not valid

You say that cleaning the EntityManager is not affected. This means that the links you are looking for are not part of the EntityManager (UnitOfWork) because you just cleared this.

Doctrine, but not Doctrine

Do you use event listeners or subscribers? Any filters? Any custom mapping types? Multiple EntityManager? Is there anything else that can be integrated into the Doctrine or its life cycle, but is not necessarily part of the Doctrine itself?

Especially often when listening to the source of problems, event listeners / subscribers often skip. Therefore, I suggest you start looking there.

Answer 3

The focus is on the PHP garbage collector, which works a little weird. First of all, every time the scripts need memory, it allocates memory from RAM, even if you use unset() , $object = null or other tricks to free memory, the allocated memory will not be returned to the operating system until the script is completed. and the process associated with it was killed.

How to fix this ?

• This is usually done on Linux systems. Create commands that run the necessary script parameters with the limit , offset parameters, and re-run the necessary scripts in small batches more times. That way, the script will use less memory, and each time the memory is freed, when the script ends.
• Get rid of Doctrine , it will start the memory by itself, PDO will be much faster and less expensive.

Answer 4

For this kind of task, when objects in memory can cause these leaks, you should use Doctrine2 Iterators

 $appFiles = $this ->em ->getRepository('AppBundle:ApplicationFile') ->findBy(array('application' => $application)); 

you need to reorganize to return a query object, not an ArrayCollection , and then from this query object you can easily call the iterate() method and clear the memory after each inspection of the object

Edit

You have "hidden links" because the detach operation will not delete an object in memory, it only tells EntityManager not to process it anymore. That is why you should use my solution or unset() object with php function.