[Asset] Adding an Asset Pipeline: versioned/digested files in pure PHP #50112

weaverryan · 2023-04-21T19:06:00Z

Q	A
Branch?	6.3
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	Partner of #48371
License	MIT
Doc PR	TODO

Hi!

This will partner with (and remove some of the code from) the importmaps PR #48371 . The goal is to allow you to write modern JavaScript & CSS, without needing a build system. This idea comes from Rails: https://github.com/rails/propshaft - and that heavily inspires this PR.

Example app using this: https://github.com/weaverryan/testing-asset-pipeline

Here's how it works:

A) You activate the pipeline:

framework:
    assets:
        pipeline:
            paths: ['assets/']

B) You put some files into your assets/ directory (which sits at the root of your project - exactly like now with Encore). For example, you might create an assets/app.js, assets/styles/app.css and assets/images/skeletor.jpg.

C) Refer to those assets with the normal asset() function

<link rel="stylesheet" href="{{ asset('styles/app.css') }}">
<script src="{{ asset('app.js') }}" defer></script>

<img src="{{ asset('images/skeletor.jpg') }}">

That's it! The final paths will look like this:

<link rel="stylesheet" href="/assets/styles/app-b93e5de06d9459ec9c39f10d8f9ce5b2.css">
<script src="/assets/app-1fcc5be55ce4e002a3016a5f6e1d0174.js" defer type="module"></script>
<img src="/assets/images/skeletor-3f24cba25ce4e114a3116b5f6f1d2159.jpg">

How does that work?

In the dev environment, a controller (technically a listener) intercepts the requests starting with /assets/, finds the file in the source /assets/ directory, and returns it.
In the prod environment, you run a assets:pipeline:publish command, which copies all of the assets into public/assets/ so that the real files are returned. It also dumps a public/assets/manifest.json so that the source paths (eg. styles/app.css) can be exchanged for their final paths super quickly.

Extras Asset Compilers

There is also an "asset" compiler system to do some minor transformations in the source files. There are 3 built-in compilers:

A) CssAssetUrlCompiler - finds url() inside of CSS files and replaces with the real, final path - e.g. url(../images/skeletor.jpg') becomes url(/assets/images/skeletor-3f24cba25ce4e114a3116b5f6f1d2159.jpg) - logic taken from Rails
B) SourceMappingUrlsCompiler - also taken from Rails - if the CSS file already contains a sourcemap URL, this updates it in the same way as above (Note: actually ADDING sourcemaps is not currently supported)
C) JavaScriptImportPathCompiler - experimental (I wrote the logic): replaces relative imports in JavaScript files import('./other') with their final path - e.g. import('/assets/other.123456abcdef.js').

Path "Namespaces" and Bundle Assets

You can also give each "path: in the pipeline a "namespace" - e.g. an alternative syntax to the config is:

framework:
    assets:
        pipeline:
            paths:
                assets: ''
                other_assets: 'other_namespace'

In this case, if there is an other_assets/foo.css file, then you can use {{ asset('other_namespace/foo.css') }} to get a path to it.

In practice, users won't do this. However, this DOES automatically add the Resources/public/ or public/ directory of every bundle as a "namespaced" path. For example, in EasyAdminBundle, the following code could be used:

<script src="{{ asset('bundles/easyadmin/login.js') }}"></script>

(Note: EA has some fancy versioning, but on a high-level, this is all correct). This would already work today thanks to assets:install. But as soon as this code is used in an app where the pipeline is activated, the pipeline would take over and would output a versioned filename - e.g.

<script src="/assets/bundles/easyadmin/login12345abcde.js"></script>

OPEN QUESTIONS / NOTES

A) Should this be a new component?
Only the "default" asset package uses the pipeline. Extend to all?

TODO:

Write tests
Add a way for bundles to add their paths to the asset paths.
Add more documentation to the Configuration options

Cheers!

stof

this misses tests for the whole Pipeline system.

src/Symfony/Bundle/FrameworkBundle/Command/AssetsPipelinePublishCommand.php

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/Configuration.php

stof · 2023-04-21T19:30:27Z

src/Symfony/Component/Asset/composer.json

-        "php": ">=8.1"
+        "php": ">=8.1",
+        "symfony/filesystem": "^5.4|^6.0",
+        "symfony/mime": "^5.4|^6.0"


those should be optional dependencies, as only the asset pipeline needs them, which is an optional feature of the component (projects using webpack won't use the asset pipeline).

Alternatively, the AssetPipeline might be big enough to become its own component (depending on symfony/asset), with the additional benefit that FrameworkBundle could then have a smart default to enable it when the package is installed. Projects will generally need either the AssetPipeline or WebpackEncoreBundle but not both

Yup - I'd be interested in what others think about this. If we keep this in Asset, but make these optional (which is definitely more correct), then users might get met with extra Run "composer require symfony/mime" to use the pipeline type of messages. Unless we also add an "asset pipeline pack" that contains all of it... but that doesn't make me super excited either.

If I had to lean towards something, it would probably be to split this into a new AssetPipeline component.

I would vote for a new component as well.

if the split is just motivated by the extra deps, I think mime and filesystem are too niche in the new pipeline system to be a strong enough reason:

here is the only place where mime is used:
return self::EXTENSIONS_MAP[$extension] ?? $this->mimeTypes->guessMimeType($filePath);

Given the list in EXTENSIONS_MAP, this means mime is almost never going to be used in practice. Instead of adding the dep to it, I'd rather suggest to throw an exception when an extenstion is unknown, and add a way to configure the mapping.

For filesystem, we're only using isAbsolutePath, so either we borrow some logic from there, our we're fine with the dep. Borrowing might be my preference.

I'd prefer having all this in the Asset component personally.

It is not only about the dependencies. It would also allow FrameworkBundle to enable the pipeline automatically based on the installed packages.

And we could ship a recipe with a config file that adds your assets/ as a pipeline path. That recipe flexibility might be another reason - though I still need to think about how we’ll accomplish the mixture of the pipeline recipe, importmaps and stimulus recipes… as each system can technically be used without the others 😀

weaverryan · 2023-04-21T20:20:50Z

this misses tests for the whole Pipeline system

It's one of my TODO items above :)

94noni

having a look on this :)
nice to get inspired by other ecosystems to improve frontend DX development for "backend" devs ^^

94noni · 2023-04-22T07:46:28Z

src/Symfony/Bundle/FrameworkBundle/Resources/config/assets_pipeline.php

+                service('assets.pipeline'),
+            ])
+
+        ->set('assets.pipeline.dev_server_subscriber', AssetPipelineDevServerSubscriber::class)


you describe in PR desc In the dev environment, a controller (technically a listener)
but I fail to see where this is conditionally registered in dev

See the FrameworkExtension which conditionally removes that definition depending on whether the server is enabled or no (which defaults to kernel.debug)

thx @stof I get it better now

94noni · 2023-04-22T07:47:11Z

src/Symfony/Component/Asset/Pipeline/AssetPipelineDevServerSubscriber.php

+    public static function getSubscribedEvents(): array
+    {
+        return [
+            // priority higher than RouterListener


this seems touchy, how we can assert that this comment stay true overtime ?

changing the priority of RouterListener would be a BC break for the ecosystem anyway, as many projects may have listeners that must run either before or after the router.

Yup, so actually not nearly as touchy as it would seem at first!

94noni · 2023-04-22T07:48:16Z

src/Symfony/Component/Asset/Pipeline/Compiler/CssAssetUrlCompiler.php

+ *
+ * Originally sourced from https://github.com/rails/propshaft/blob/main/lib/propshaft/compilers/css_asset_urls.rb
+ */
+class CssAssetUrlCompiler implements AssetCompilerInterface


nitpick, what about StyleSheetAsset as you named JavaScript and not JS in the class below :)

94noni · 2023-04-22T07:51:32Z

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

+
+    private function getPublicDirectoryName(string $projectDir): string
+    {
+        $defaultPublicDir = 'public';


what about making this variable? still have some projects using /web as "public" directory
can't we define in the kernel a public dir like there is a cache dir etc?

For whatever reason, this IS configurable, but via composer.json. I stole this logic from the assets:install command, which seems to be the only other place where symfony is aware of your public directory.

94noni · 2023-04-22T07:57:53Z

src/Symfony/Bundle/FrameworkBundle/Command/AssetsPipelinePublishCommand.php

+ * @final
+ */
+#[AsCommand(name: 'assets:pipeline:publish', description: 'Publishes all pipeline assets to the final public output directory.')]
+class AssetsPipelinePublishCommand extends Command


what about

Suggested change

class AssetsPipelinePublishCommand extends Command

class AssetsPipelineCompileCommand extends Command

to me, "publish" sound like "deploy" (I'm not native English speaker tho)
here it is just compiling/dumping the content

It is an action you take during deploy though? Rails does call this command “precompile”, however…

javiereguiluz · 2023-04-22T14:00:46Z

@weaverryan how would be the workflow for folks using e.g. SCSS? Before we just used Webpack Encore. Should we keep using that now to compile it into CSS? Then, how do we connect the compiled assets from public/build/ with this new pipeline feature? Thanks.

stof · 2023-04-22T15:09:56Z

@javiereguiluz if the assets are already in the public folder, you don't need the asset pipeline at all. The pipeline does not replace the existing way (which still makes sense when using the webpack stack)

javiereguiluz · 2023-04-22T15:23:23Z

So, if you use anything that it's not pure CSS or JS, you can't use this, right?

weaverryan · 2023-04-22T16:12:27Z

@weaverryan how would be the workflow for folks using e.g. SCSS? Before we just used Webpack Encore. Should we keep using that now to compile it into CSS? Then, how do we connect the compiled assets from public/build/ with this new pipeline feature? Thanks.
So, if you use anything that it's not pure CSS or JS, you can't use this, right?

This will be the NUMBER 1 question & thing to document. If you need to compile scss -> css or use something like Tailwind, the answer is that you can/should now use their standard tools. For example, for Sass, you would install the sass binary - https://sass-lang.com/install. You'd probably have an assets/styles/*.scss type of setup like you have now. When you run sass, you would output these to perhaps assets/styles/*.css or perhaps another directory like assets/built/*.css.

Anyways, once you do that, you'll ultimately end up with plain, normal CSS files. And THOSE are what you'd reference in your app:

<link rel="stylesheet" href="{{ asset('styles/app.css') }}">

Same goes for Tailwind. We'll definitely need to give people specific instructions / recommendations on how to set this all up.

yguedidi · 2023-04-22T18:22:51Z

@weaverryan being a "pipeline", what about providing extra asset compilers for those external tools?

weaverryan · 2023-04-22T23:11:24Z

That’s definitely an option :). Those would use the underlying binaries (e.g. sass), which I’m pretty sure is what you’re thinking too. But yes, certainly an option! Rails does something like this - iirc for tailwind for example - but I haven’t dug into the specifics.

Plus making sure dev server requires the digest unless pre-digeste

weaverryan · 2023-04-24T00:31:43Z

I DO think this should be split into its own component if we want to make the recipe situation simpler. Basically, you could still install symfony/asset without opting into the pipeline - which is what Encore users would do, for example. It if you installed symfony/asset-pipeline, then we could ship a config file that activates the pipeline and likely ships with a starting assets/ directory.

Other than splitting and some windows tests (the code loves /‘s), this is ready for review.

Neirda24 · 2023-04-24T07:55:47Z

This seems neat ! However I'm a bit confused too by the pipeline naming. I was expecting a multi step configurable process where each output is injected as input of the next one. To me it looks a lot like what assets.packages behave at the moment I don't understand the pipeline here.

Kocal · 2023-04-24T08:36:31Z

@weaverryan how would be the workflow for folks using e.g. SCSS? Before we just used Webpack Encore. Should we keep using that now to compile it into CSS? Then, how do we connect the compiled assets from public/build/ with this new pipeline feature? Thanks.
So, if you use anything that it's not pure CSS or JS, you can't use this, right?

This will be the NUMBER 1 question & thing to document. If you need to compile scss -> css or use something like Tailwind, the answer is that you can/should now use their standard tools. For example, for Sass, you would install the sass binary - sass-lang.com/install. You'd probably have an assets/styles/*.scss type of setup like you have now. When you run sass, you would output these to perhaps assets/styles/*.css or perhaps another directory like assets/built/*.css.

Anyways, once you do that, you'll ultimately end up with plain, normal CSS files. And THOSE are what you'd reference in your app:
<link rel="stylesheet" href="{{ asset('styles/app.css') }}">
Same goes for Tailwind. We'll definitely need to give people specific instructions / recommendations on how to set this all up.

How will it works with HMR?

dreadnip · 2023-04-24T08:55:43Z

@weaverryan being a "pipeline", what about providing extra asset compilers for those external tools?

If you go down this route you're just building a new Assetic. It seems more fitting to keep the scope of this component small and focused on vanilla CSS/JS, matching the related PR about importmaps (#48371). Webpack can still be the default option for people who need a compilation step.

weaverryan · 2023-04-24T13:06:50Z

This seems neat ! However I'm a bit confused too by the pipeline naming

I borrowed this name from rails "Propshaft is an asset pipeline library for Rails" - https://github.com/rails/propshaft - but i'm open to other names (though pipeline is pretty catchy). There IS a "compiler" system in there where you can make changes to the source code, but at the moment (and this should perhaps remain this way) that is meant to be lightweight - it's used, e.g., for updating import paths in JS/CSS files.

... Same goes for Tailwind. We'll definitely need to give people specific instructions / recommendations on how to set this all up.
How will it works with HMR?

AFAIK, HMR doesn't work without a built system like Webpack. If someone wanted to get crazy, an HMR system could be built using Mercure - e.g. run a command that watches CSS files for changes, use Mercure to send that change to the frontend, and on the frontend, swap the old CSS file out for the new one. It would be a neat extra for someone to build in an external bundle, I think. You might even be able to do this without Mercure (since it's just a dev thing) by having an endpoint that stays alive forever, watches the file changes in a loop, and return the server events as they happen.

If you go down this route you're just building a new Assetic. It seems more fitting to keep the scope of this component small and focused on vanilla CSS/JS, matching the related PR about importmaps (#48371). Webpack can still be the default option for people who need a compilation step.

This is fair. We shouldn't rush into adding asset compilers for transforming Sass or Tailwind. Let's document using those tools on their own the "normal" way and see how things go.

Neirda24 · 2023-04-24T13:39:49Z

I borrowed this name from rails "Propshaft is an asset pipeline library for Rails" - https://github.com/rails/propshaft - but i'm open to other names (though pipeline is pretty catchy). There IS a "compiler" system in there where you can make changes to the source code, but at the moment (and this should perhaps remain this way) that is meant to be lightweight - it's used, e.g., for updating import paths in JS/CSS files.

⚠️ I would emit a warning about the "Catchy" because I think one of the strong assets of Symfony is the fact that we can understand the purpose of each component by their name. "Pipeline" is often used in CI/CD context and can be very misleading. I don't really see the benfit of having a dedicated component where it seems to integrate very nicely with the Asset component. Shouldn't it be directly in here ?

weaverryan · 2023-04-24T14:18:42Z

I would emit a warning about the "Catchy" because I think one of the strong assets of Symfony is the fact that we can understand the purpose of each component by their name.

Can you suggest some alternative names? I like descriptive names too - what kinds of words would be helpful? If not pipeline, then...

AssetResolver?
AssetCompiler?
AssetExposer?
AssetMapper?
AssetPublisher?

I don't really see the benfit of having a dedicated component where it seems to integrate very nicely with the Asset component. Shouldn't it be directly in here ?

From a practical perspective, if it lives in the same component as Asset, then we probably can't ship a recipe that enables the pipeline by default... because often people composer require symfony/asset to just get the {{ asset() }} function. And so we wouldn't want to activate a new system for them. By splitting, if you install symfony/asset, you get exactly what you have right now. THEN you can install symfony/asset-pipeline (or whatever we name it) to activate that new system in your app.

it seems to integrate very nicely with the Asset component. Shouldn't it be directly in here

It does integrate nicely :). At the same time, there is only 1 small class that that interacts with the Asset component! 95% of the code in the new component is standalone. I just split it into its own component, and it felt very natural.

carsonbot added Status: Needs Review Asset Feature labels Apr 21, 2023

carsonbot added this to the 6.3 milestone Apr 21, 2023

stof reviewed Apr 21, 2023

View reviewed changes

94noni reviewed Apr 22, 2023

View reviewed changes

weaverryan added 17 commits April 23, 2023 13:56

[WIP] Config

261d2a2

[WIP] Initial AssetPipeline class and "dev server" via listener

dd3860e

Adding hook into the asset() system + adding service config

656f4ed

[Asset] Adding assets:pipeline:publish command for production

f7278eb

[Asset] Writing and using manifest.json file

44bdc9f

Plus making sure dev server requires the digest unless pre-digeste

[WIP] First additions of asset compiler system

b33354c

Adding the first asset compiler

a481621

Adding a 2nd compiler for rewriting source map URLs

48ad55e

Adding experimental compiler to rewrite relative import paths

9de5e8b

license headers

01e16ae

Fixing path resolution

0e4e20e

use shared max age

a3e08c4

fixing bad json

f40eea3

phpcs

71d12b2

Fixes thanks to stof!

68a9963

fixing pslam

b0fd7d9

Test + bug fix

20ff8e5

weaverryan added 2 commits April 23, 2023 17:03

changing order so custom paths override bundle paths

e354f8f

fabbot

6cfe54a

weaverryan added 3 commits April 24, 2023 09:16

Fixing test on Windows

540e7c7

Using realpath to ease path comparisons on windows

44d4bb7

removing todos about logging

fa6a648

weaverryan added 4 commits April 24, 2023 09:45

[AssetPipeline] Moving files into their own component

ca92d4c

Adding the necessary component files

d0592b2

[AssetPipeline] Updating configuration

c4942b2

Removing service if the Asset component is not enabled

845e9ad

weaverryan added 14 commits April 24, 2023 10:21

fixes

16682d9

fixing tests

bc157d5

normalizing service names

3215785

various typo fixes

5ad5dba

phpcs

ce9171e

require pipeline in dev for fwbundle

0578969

Moving kernel into FW bundle so it can be tested in isolation

eab8237

fabbot

09dc87e

require fwbundle so we can functional test

193964d

wrong TestCase!

3cb2ee7

updated fwbundle version

b633b8d

small tweaks

084f6c8

dev require symfony/asset

ec60ac2

also need browserkit

82252e3

[Asset] Adding an Asset Pipeline: versioned/digested files in pure PHP #50112

[Asset] Adding an Asset Pipeline: versioned/digested files in pure PHP #50112

weaverryan commented Apr 21, 2023 •

edited

stof left a comment

stof Apr 21, 2023

weaverryan Apr 21, 2023

fabpot Apr 22, 2023

nicolas-grekas Apr 22, 2023 •

edited

stof Apr 22, 2023

weaverryan Apr 22, 2023

weaverryan commented Apr 21, 2023

94noni left a comment

94noni Apr 22, 2023

stof Apr 22, 2023

94noni Apr 24, 2023

94noni Apr 22, 2023

stof Apr 22, 2023

weaverryan Apr 22, 2023

94noni Apr 24, 2023

94noni Apr 22, 2023

94noni Apr 22, 2023

weaverryan Apr 22, 2023

94noni Apr 22, 2023

weaverryan Apr 22, 2023

javiereguiluz commented Apr 22, 2023 •

edited

stof commented Apr 22, 2023

javiereguiluz commented Apr 22, 2023

weaverryan commented Apr 22, 2023

yguedidi commented Apr 22, 2023

weaverryan commented Apr 22, 2023

weaverryan commented Apr 24, 2023

Neirda24 commented Apr 24, 2023

Kocal commented Apr 24, 2023

dreadnip commented Apr 24, 2023

weaverryan commented Apr 24, 2023

Neirda24 commented Apr 24, 2023

weaverryan commented Apr 24, 2023 •

edited

	class AssetsPipelinePublishCommand extends Command
	class AssetsPipelineCompileCommand extends Command

[Asset] Adding an Asset Pipeline: versioned/digested files in pure PHP #50112

Are you sure you want to change the base?

[Asset] Adding an Asset Pipeline: versioned/digested files in pure PHP #50112

Conversation

weaverryan commented Apr 21, 2023 • edited

Extras Asset Compilers

Path "Namespaces" and Bundle Assets

stof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas Apr 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weaverryan commented Apr 21, 2023

94noni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javiereguiluz commented Apr 22, 2023 • edited

stof commented Apr 22, 2023

javiereguiluz commented Apr 22, 2023

weaverryan commented Apr 22, 2023

yguedidi commented Apr 22, 2023

weaverryan commented Apr 22, 2023

weaverryan commented Apr 24, 2023

Neirda24 commented Apr 24, 2023

Kocal commented Apr 24, 2023

dreadnip commented Apr 24, 2023

weaverryan commented Apr 24, 2023

Neirda24 commented Apr 24, 2023

weaverryan commented Apr 24, 2023 • edited

weaverryan commented Apr 21, 2023 •

edited

nicolas-grekas Apr 22, 2023 •

edited

javiereguiluz commented Apr 22, 2023 •

edited

weaverryan commented Apr 24, 2023 •

edited