Cache of Doctrine SQL Statement leads to 503 error in Apache #44270

Art4 · 2021-11-25T11:07:00Z

Art4
Nov 25, 2021

Hello 👋

(Please note the Update 1 below)

I'm not sure if this is the right place to ask, but I'm a little lost with this mysterious problem...

The problem occurs only on the prod server and I was not able to reproduce it locally.

Lets say I have a Symfony app (v5.3.10) with a page under http://example.org/admin where the admin dashboard lives. After a new deploy of the app I can only request /admin one time successfully. It shows me the admin dashboard without errors.
But if I request this page a second time the server gives me immediately (not after some seconds or a timeout) a 503 error.

The full error response is:

HTTP/2 503 Service Unavailable
content-length: 299
content-type: text/html; charset=iso-8859-1
date: Thu, 25 Nov 2021 09:09:08 GMT
server: Apache
X-Firefox-Spdy: h2

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

Obviously this error comes from Apache, not from Symfony. But interesting is that if I clear the cache and then request the /admin page it will be show successfully one time. On the second request (and every other request) Apache will response with 503 error again. After a cache:clear the page will work again one time.

This problem occurs only on a specific route or controller. So /admin is affected, but not /admin/subpage.
I first thought that this problem has something to do with the AdminController, so I've debugged the controller (yes, on prod server 🙈 ) to simply show some fake data in a simple rendered twig template but the problem is the same. I've also found out that this error occurs somewhere after the controller returns the Response class (by calling $this->render()).

And now I've found 2 other endpoints/controller with the same strange behaviour. It seems that this problem occurs independent from the other controllers. So I can request /admin one time, after then it returns the 503 error. Next I can request /another-route one time, and the next request will return the 503 error and so on.

Another point that I noticed is that this behavior first occurred after I switched from the legacy security system to the new authenticator-based Security system, so maybe there is a connection there too.
see Update 1: This behavior occured first after I switched from doctrine/orm 2.9.5 to 2.10.1.

I've looked into the server logs but there is nothing logged. So for me it seems like a weird bug with the Symfony cache and Symfony Security in combination with Apache.

I'm not sure if someone could help me but I would be glad for every hint what or where I could look next to find the root of this problem.

Thank you.

Update 1 (2021-11-27)

I was able to reproduce the issue on a staging area and looked deeper into the cache problem. I've found out that the problem comes from a specific Doctrine cache. In my case of the /admin endpoint there will be created a cache in var/cache/prod/pools/iWwGDKOazk/R/6/085f6b9716d8e39e78c0de0dcdefffd8. If I remove only this cache file the request will work one time because on the next request the cache file will be there again.

This behavior occured first after I switched from doctrine/orm 2.9.5 to 2.10.1, so maybe there is a connection too.

This is the content of the cache file:

<?php //085f6b9716d8e39e78c0de0dcdefffd8

return [PHP_INT_MAX, static function () { return \Symfony\Component\VarExporter\Internal\Hydrator::hydrate(
    $o = [
        clone (($p = &\Symfony\Component\VarExporter\Internal\Registry::$prototypes)['Doctrine\\ORM\\Query\\ParserResult'] ?? \Symfony\Component\VarExporter\Internal\Registry::p('Doctrine\\ORM\\Query\\ParserResult')),
        clone ($p['Doctrine\\ORM\\Query\\Exec\\SingleSelectExecutor'] ?? \Symfony\Component\VarExporter\Internal\Registry::p('Doctrine\\ORM\\Query\\Exec\\SingleSelectExecutor')),
        clone ($p['Doctrine\\ORM\\Query\\ResultSetMapping'] ?? \Symfony\Component\VarExporter\Internal\Registry::p('Doctrine\\ORM\\Query\\ResultSetMapping')),
    ],
    null,
    [
        'Doctrine\\ORM\\Query\\ParserResult' => [
            '_sqlExecutor' => [
                $o[1],
            ],
            '_resultSetMapping' => [
                $o[2],
            ],
            '_parameterMappings' => [
                [
                    1 => [
                        0,
                    ],
                ],
            ],
        ],
        'Doctrine\\ORM\\Query\\Exec\\AbstractSqlExecutor' => [
            '_sqlStatements' => [
                1 => 'SELECT count(u0_.id) AS sclr_0 FROM users u0_ WHERE u0_.deleted_at IS NULL AND u0_.created_at > ?',
            ],
        ],
        'stdClass' => [
            'scalarMappings' => [
                2 => [
                    'sclr_0' => 1,
                ],
            ],
            'typeMappings' => [
                2 => [
                    'sclr_0' => 'integer',
                ],
            ],
        ],
    ],
    $o[0],
    []
); }];
~

Answered by Art4

Nov 29, 2021

I'm sure that this is a problem with doctrine/orm or doctrine/dbal, so I created a ticket there, see doctrine/orm#9207.

Update: It looks like this problem only occurs with PHP 7.4. After switching to PHP 8, the problem was solved.

View full answer

Art4 · 2021-11-29T15:25:50Z

Art4
Nov 29, 2021
Author

I'm sure that this is a problem with doctrine/orm or doctrine/dbal, so I created a ticket there, see doctrine/orm#9207.

Update: It looks like this problem only occurs with PHP 7.4. After switching to PHP 8, the problem was solved.

0 replies

DigitalTimK · 2022-01-18T10:11:14Z

DigitalTimK
Jan 18, 2022

Thank you for this post. The description sound similar to the problem I am facing. So I don't feel so alone and desperate anymore ;-)

3 replies

Art4 Jan 18, 2022
Author

You're welcome. My "workaround" was an update to PHP 8.

DigitalTimK Jan 18, 2022

I saw this in doctrine/orm#9207, but I cannot change PHP Version on the Production Server.

Art4 Jan 18, 2022
Author

Maybe you could change the framework.cache.pools.doctrine.system_cache_pool config from cache.system to cache.app. This had also worked in my tests, but I don`t know all the effects of this change.

bezin · 2022-07-20T06:58:04Z

bezin
Jul 20, 2022

Hello, I ran into this issue the other day as well with a project and I can confirm that swapping the Cache fixes this issue. My hoster digged into this more deeply and came to the conclusion, that the Ion Cube Loader misinterprets the format of the cache files as its own file format, tries to parse it and fails. This crashes the PHP process and hence Apache returns 503.

I am not gonna dig into this more deeply as it is supposed to be a PHP 7.4 issue only and we will be on PHP 8.0 soon with this project, but maybe this information saves somebody some hours :-)

0 replies

thomas2411 · 2022-08-14T13:44:05Z

thomas2411
Aug 14, 2022

I had the same problem today. Out of nowhere it started 503 Service unavailable but only for some of the Entities in my project, not all of them. Removing query_cache_driver removed the issue. Changing doctrine.system_cache_pool config from cache.system to cache.app fixed the problem too, for now. I have:

PHP 8.1.1
Symfony 6.1.3 and
"doctrine/annotations": "^1.13.3",
"doctrine/doctrine-bundle": "^2.7.0",
"doctrine/doctrine-migrations-bundle": "^3.2.2",
"doctrine/orm": "^2.13.1"

It drives me nuts. I spent with server admin a few hours and what they said it is something from the PHP app. A switched from Redis cache to filesystem and the same issue appears so it is not a Redis problem. In DEV environment, where this cache is not used it works fine. On prod does not.

On logs I cannot see anything. Just when it runs query to problematic Entity it disconnecting and throws 503.

3 replies

Art4 Aug 15, 2022
Author

What's your PHP version?

thomas2411 Aug 15, 2022

8.1.1. This week I will try to update to 8.1.9

thomas2411 Aug 16, 2022

We have update to 8.1.9, the problem still exists. What is interesting is that the same situation is on 3 other servers (but the same hosting company) where there are other apps in Symfony, different version of Symfony, Doctrine etc. And it happened a few days ago. So it might be connected with server software but the hosting company is not willing to tell me what has changed.

thomas2411 · 2022-08-19T08:46:03Z

thomas2411
Aug 19, 2022

After digging deeper I found that as before, the problem is with cache.system I do not have APCu on server so with this information: https://symfony.com/blog/new-in-symfony-3-1-cache-component#symfony-integration we know that cache.system uses APCu by default, if not found fallback to filesystem.

So with this setting:

framework:
    cache:
        app: cache.adapter.redis
        pools:
            doctrine.result_cache_pool:
                adapter: cache.app
            doctrine.system_cache_pool:
                adapter: cache.system

I have 503 error. If I change cache.system to cache.adapter.redis it works. And what is weird if I change cache.system to cache.adapter.filesystem it works too. But if I change cache.system to cache.adapter.apcu I get an error while clearing the cache that apcu is not enabled in php. So what is going on here?

0 replies

thomas2411 · 2022-08-22T10:08:26Z

thomas2411
Aug 22, 2022

As @bezin wrote, the same situation was here. The server admin finally admitted that IonCube might have been updated recently. The remove it from my PHP setup and it works.

4 replies

stof Sep 13, 2022
Collaborator

if IonCube breaks the Symfony filesystem cache adapter, I suggest reporting a bug to IonCube. Their extension should not break working code.

creiner Sep 20, 2022

Did anybody reported the bug to the IonCube helpdesk?
Since it needs registration, I cannot see it.

kevinpapst Oct 22, 2022

Same issue here. Hosting company All-Inkl forces that ioncube stuff on every customer.
I am trying to get their support have a look and raise a ticket, they should be big enough.

justpilot Oct 22, 2022

Same issue here. Hosting company All-Inkl forces that ioncube stuff on every customer. I am trying to get their support have a look and raise a ticket, they should be big enough.

Same here, all-inkl.com customer. I called the support. They are unwilling to change a thing.

justpilot · 2022-09-13T10:38:54Z

justpilot
Sep 13, 2022

I have the same error from now on, without changing my scripts.
When I clear the cache I can call the API Endpoint once, after that it's status 503.

PHP: 8.1.7

This program makes use of the Zend Scripting Language Engine:
Zend Engine v4.1.7, Copyright (c) Zend Technologies
with the ionCube PHP Loader v12.0.2, Copyright (c) 2002-2022, by ionCube Ltd.
with Zend OPcache v8.1.7, Copyright (c), by Zend Technologies

<title>503 Service Unavailable</title>

<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>

Anybody with a solution?

1 reply

justpilot Sep 13, 2022

As suggested above the temporary solution is:

File: doctrine.yaml

cache.system -> cache.adapter.filesystem

[...]

    framework:
        cache:
            pools:
                doctrine.result_cache_pool:
                    adapter: cache.app
                doctrine.system_cache_pool:
                    adapter: cache.adapter.filesystem
                    #adapter: cache.system

justpilot · 2022-10-22T10:49:36Z

justpilot
Oct 22, 2022

Please check if the problem only occurs when dealing with DateTimeImmutable property type.
I changed the type to DateTime Mutable and the problem with the cache is gone.

0 replies

joerndyherrn · 2022-10-24T07:31:03Z

joerndyherrn
Oct 24, 2022

@justpilot seems not to be the problem...

1 reply

justpilot Oct 24, 2022

@justpilot seems not to be the problem...

Thank you, then this was just a coincidence!

Cache of Doctrine SQL Statement leads to 503 error in Apache #44270

Update 1 (2021-11-27)

Replies: 9 suggested answers · 12 replies

Art4 Nov 29, 2021 Author

Art4 Jan 18, 2022 Author

Art4 Jan 18, 2022 Author

Art4 Aug 15, 2022 Author

stof Sep 13, 2022 Collaborator

Insert Link

Replies: 9 suggested answers 12 replies

Art4
Nov 29, 2021
Author

Art4 Jan 18, 2022
Author

Art4 Jan 18, 2022
Author

Art4 Aug 15, 2022
Author

stof Sep 13, 2022
Collaborator