Skip to content

Idle workers eventually exhaust memory and crash #45386

Closed
@Bilge

Description

@Bilge

Symfony version(s) affected

5.4.4

Description

An idle worker subscribed to one queue and processing no messages, running via bin/console messenger:consume -vv, will, eventually, simply crash and die with an error similar to the following:

16:22:32 CRITICAL [php] Fatal Error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 32768 bytes) ["exception" => Symfony\Component\ErrorHandler\Error\OutOfMemoryError^ { …}]

That is, a worker process that purportedly does absolutely nothing, quietly consumes more and more memory until it dies.

How to reproduce

Ensure at least one message queue is configured, e.g.

framework:
  messenger:
    reset_on_message: true

   transports:
      email: 'beanstalkd://127.0.0.1:11300?ttr=60&tube_name=email'

    routing:
      Symfony\Component\Mailer\Messenger\SendEmailMessage: email

Run the worker for the default (only) queue:

bin/console messenger:consume -vv

Eventually the process will crash. In my example, I am using memory_limit = 256M, but obviously smaller limits will allow quicker reproduction.

Possible Solution

A worker should not consume resources indefinitely, especially when it isn't even processing any messages! Most memory acquired should subsequently be dereferenced so it can be freed. Judicious use of unset() or null assignments may be necessary in some cases.

Additional Context

  • PHP 8.1/Win
  • Debug mode is on (dev environment).

Before I discovered the Symfony Messenger component, I wrote my own PHP queue worker and never had to consider running out of memory. It processes hundreds of thousands of messages per day and has an uptime of over 6 months with exactly the same PHP memory limits as the Symfony worker which dies within a few hours processing nothing. Clearly something is wrong here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions