BinaryFileResponse fails on Azure when running under Nginx on Linux (but works correctly on Windows/IIS) #47059

aokrasavin · 2022-07-22T11:36:25Z

aokrasavin
Jul 22, 2022

Symfony version(s) affected

5.4, 6.0 and 6.1 confirmed, presumably others

Description

A fairly simple controller which picks form input, chooses image from the disk and then returns it to webbrowser. The code works absolutely perfectly on local development instance of Laravel (Windows 10) or Azure Windows-based PHP service, but corrupts data when deployed on Azure PHP App Service on Linux (under nginx).

All headers are delivered correctly with code 200
The binary payload for actual image has first 117 bytes stripped out. Eg after the headers, the binary data starts at byte 118, where byte 0 should have been. The rest of the bytes arrive correctly and at the end there are 117 empty bytes (filled with 0x00), so the total size is correct.
The issue exists for any files (not just images), for different types of images, for different security (eg HTTP or HTTPS)
The amount of bytes missing at start seems to be always the same and unrelated to image content (eg no magic byte sequence)

How to reproduce

REPRODUCER

Any recent version of Symfony (confirmed for versions 5.4, 6.0 and 6.1).
Any recent PHP version (7, 8, etc)
The problem manifests itself when run under nginx server on Azure PHP app service on Linux. To be clear - as of few months ago, PHP app stack on Azure is ONLY available on Linux and nginx. If you want to use PHP 8 or later - you MUST switch to this service. Original Windows/IIS based service will not longer be developed and is stuck at PHP 7.
There is anecdotal evidence that the problem will manifest itself on any instance on nginx running under Linux nginx versions 1.20, 1.22 or 1.23

public function test()
{
$path = "......"; // a valid locally accessible path to a binary file (image, etc) of 30K or bigger
return new BinaryFileResponse($path);
}

Possible Solution

The following fix works for me, but I do not fully understand the reason for the problem or possible implications, so would prefer someone with better in-depth knowledge to review prior to submitting pull request. The issue is in the following line:

namespace Symfony\Component\HttpFoundation;
......
class BinaryFileResponse extends Response
.....
    public function sendContent(): static
{
......
   stream_copy_to_stream($file, $out, $read);
......
}

to fix, I implemented the following - eg PHP native function stream_copy_to_stream is replaced with a custom function, which does (as far as I could tell) the same thing.

  function pipe_streams($in, $out)
    {
        $size = 0;
        while (!feof($in)) $size += fwrite($out,fread($in,8192));
        return $size;
    }

    public function sendContent(): static
{
......
   $this->pipe_streams($file, $out);
......
}

As people observed, this simple code does not support Ranges (so you need to disable it in headers). The buffer size is chosen randomly - we could have a lively argument of what it should be, but it does not materially changes the fix.

Additional Context

I attached two versions of the same image returned by the SAME code running under Windows IIS (correct) and Linux (nginx) (corrupted

)

In Laravel, the original code was

    $img = .... <get data object describing image>
    $response = response()->file(Storage::disk('public')->path($img->filename));
    $response->headers->set('Content-Type', $img->mime);
    $response->headers->set("Cache-Control", "max-age=86400");
    return $response;

response()->file() and response->download() produce the same result - this is understandable as they only differ in headers

michaelLoeffelmann · 2022-07-22T12:22:13Z

michaelLoeffelmann
Jul 22, 2022

i can confirm this behavior. i have exactly the same issue.

0 replies

MatTheCat · 2022-07-25T12:26:23Z

MatTheCat
Jul 25, 2022

The code does not look like Symfony 🤔

Anyways did you check if the request to the image includes a Range header?

9 replies

MatTheCat Jul 26, 2022

Please provide an application’s repository containing the minimum code required to reproduce your problem and running with Docker.

aokrasavin Jul 26, 2022
Author

I apologize - I do not know how do it while demonstrating stated manifestation conditions - I narrowed it down to calling a single function in any context - in effect as it currently stands, Symfony's BinaryFileResponse is completely broken under nginx/Linux. Revieweing the code - the implementation is inherently flawed and will break randomly on other platforms. I am not using Symfony directly - it is a second or third level dependency for me. Ultimately, I am happy with my fix - I provided the ticket as a courtesy to help other developers (I spent close to a day figuring out this one and it is frustratingly obscure in its manifestation). As far as my explanation of the fix stays on top of Google searches, I am content. and do not particularly care if it is fixed ot not - so I am signing off of this discussion.
As an observation, the built-in method in question is used incorrectly in current Symfony code. It introduces a thread-racing condition which exposes itself in certain envrionments. With developers being forced to Nginx I expect more people will be encountering it.

MatTheCat Jul 27, 2022

I used BinaryFileResponse with nginx countless time without any issue.

The “fix” you propose is impractical on top on breaking range requests so I don’t advise anybody to apply it.

Also by checking the laravel issue you failed to link I found laravel/framework#42681 (comment) As I don’t know which version you meant by 6.1 it may be a solution.

aokrasavin Jul 27, 2022
Author

The bug consistently manifests itself under nginx running on Linux on the cloud. I could not reproduce it on Windows version. I suspect that the reason for that is a different threading model - Linux is a "true" multithreaded system, while Windows compilations tend to be lazy and only pretend to be one. Because of this, thread-racing condition on Windows are either rare or impossible - this is why thread-racing issues are so frustrating. Since my original ticket on Laravel desk, I purposefully tested it on different versions of Symfony and consistently get the same outcome.

Laravel ticket is here laravel/framework#42685 - if people search for the same header - it comes as the second link.

laravel/framework#42681 is a different issue - it is not applicable here.

aokrasavin Jul 27, 2022
Author

I agree that my simple fix does nto support Range - I had it disabled on my system. This is the reason I did not want to put PULL request in - I wanted people with better understanding on low-level Symfony operations to apply proper fix.

javierreguillo · 2022-07-26T06:43:06Z

javierreguillo
Jul 26, 2022

Hi,

I have the same problem with Symfony 5.4 on Azure App Service product using php 8.0 + nginx
The same project works fine on AWS and on premise... I think it could be some problem with the headers

Could you help me?

15 replies

MatTheCat Aug 3, 2022

This thread seems eerily similar: using readfile fails but emulating it with fread works. Looks like Azure has an issue when you don’t chunk data you send to PHP output buffer 🤔

aokrasavin Aug 3, 2022
Author

Very intetresting - yes - this is the kind of ammunition to argue the case!

klodoma Aug 9, 2022

I am having the same issue see here

I did some testing and it seems that all these php functions are affected:

stream_copy_to_stream
readfile
fpassthru

@aokrasavin: the BUG is coming out from symfony, laravel and wherever these functions are used. It's definitely not a library bug. It is a native Azure Linux Web App bug, a very nasty one imo. I can confirm this on 2 Azure Installations. Considering the little feedback, it seems that PHP + Azure is not really used so much.

MatTheCat Aug 9, 2022

Seems like the response body size has to meet a threshold greater than 8192 bytes for the problem to manifest (that may explain why VenkateshDodda-MSFT does not reproduce). Could you try increasing the chunks size until it appears again?

MatTheCat Sep 2, 2022

@aokrasavin I think you can close this discussion as Microsoft answered.

ryanmwolfe · 2023-05-22T16:53:58Z

ryanmwolfe
May 22, 2023

While I can't drop a lot of details as the code is proprietary, we've seen this issue manifest in an existing codebase when jumping from Ubuntu 18.04 to 20.04 as part of an EOL upgrade.

Symfony version 5.x running php 7.4 under a debian variant container. The container itself did not change in any meaningful way, the host running the container workloads was what was upgraded.

Same situation, pulling a file to be served by a symfony server (no nginx) but we only got 8192 of the file (cutting off the top in our case).

0 replies

Uh oh!

BinaryFileResponse fails on Azure when running under Nginx on Linux (but works correctly on Windows/IIS) #47059

Uh oh!

Uh oh!

Symfony version(s) affected

Description

How to reproduce

Possible Solution

Additional Context

Replies: 4 comments · 24 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aokrasavin Jul 26, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aokrasavin Jul 27, 2022 Author

Uh oh!

aokrasavin Jul 27, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aokrasavin Aug 3, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 4 comments 24 replies

aokrasavin Jul 26, 2022
Author

aokrasavin Jul 27, 2022
Author

aokrasavin Jul 27, 2022
Author

aokrasavin Aug 3, 2022
Author