Replace Webpack

[jmikrut]

We need to move away from Webpack to optimize for DX. A user should be able to select their bundler and pass it to Payload on initialization.

We should decide if we want to support Vite or Turbopack first.

Adapter pattern

The first step to implementing support for additional bundlers is to abstract out everything that Webpack is currently used for into a new adapter-style pattern.

For example, we could create a new config property called bundler that accepts a class or object which provides the following properties:

interface PayloadBundler {
  dev: (config: SanitizedConfig) => Promise<PayloadHandler>, // this would be a typical Express middleware handler
  build: (config: SanitizedConfig) => Promise<void> // used in `payload build`
  serve: (config: SanitizedConfig) => Promise<PayloadHandler> // serve built files in production
}

First quick pass tells me that this might be all that we'd need. Usage would look like this:

export default buildConfig({
  bundler: getViteBundler({
    // Vite used as example here, you'd pass any customization options to Vite here
  }),
  collections: [
    // your collections here
  ]
});

Pre-breaking change implementation

In order to test / implement future bundlers, we should keep Webpack within Payload's existing dependencies until we release a breaking change w/ major version (right now, that'd be 2.0.0).

But we could support the new bundler config property in advance, and then users could opt-in to passing their own bundler, which, if present, Payload would defer to it instead of using its built-in Webpack dependencies. These external bundlers could come with all dependencies they need (Vite, Turbopack, etc.) so that Payload itself does not need to install them unless they are used.

[philipgher]

Totally agree with moving away from webpack. There are nice alternatives like you mentioned.
What were your ideas surrounding the setup of turbopack?

Imo, the Turbo Kitchen Sink example would be nice for Payload.

An example of how the project could be layed out based on the example:

Apps and Packages

• payload: the payload admin app, consisting of:
  • api: an Express server. Exposing the REST and GraphQL endpoints (kind of the current server.ts file).
  • admin: a Vite single page app. Exposing the current admin UI.
  • admin-ui (maybe): React UI library. Including the admin UI components.
• ui: React UI library. Maybe including the admin UI components - as well as the components to be added by user.
• tsconfig: tsconfig.json;s used throughout the monorepo.

To be added by the dev, one or more of:

• blog/storefront/whatvver: a Remix or Next.js app

This seems ideal to me - you get all the TurboRepo goodness when you are willing to structure your app to follow the turbo setup. But if you want a setup more similar to the current one you just run the Payload app/package separately. Turbo can be a little too intimidating/prescriptive of your setup for all people.

[sannajammeh]

I suggest RSPack. 1-1 port of webpack to Rust.

[cnikolov]

didn't knew that existed but sounds like a reasonable option to start with

[wiesson]

I haven't used turbo, but all Vite based frameworks I have worked with are extremely pleasant to use (For example Astro or Sveltekit, …)

[thgh]

I have been looking into Vite support in #2530 🎉

[purenofuss]

Have you been able to reach a resolution on which bundler to support?

[jmikrut]

No, but it's likely that we will establish an adapter-based pattern so that you as the user can identify if you'd prefer to use Vite, Turbopack, etc.

[VadBe]

Follow your heart and use Vite.

[DasMergo]

+1 for turbo pack: https://turbo.build/pack/docs/comparisons/vite
also is being used by many next.js developers

[AlessioGr]

Wouldn't quite trust that comparison. Apparently it wasn't quite accurate to begin with, + the performance improvements in vite 4.3 will have an effect as well

See:
yyx990803/vite-vs-next-turbo-hmr#8
and
https://twitter.com/sapphi_red/status/1644647019661893633?s=46

[DasMergo]

yeah, but you have to check the results on more components and also SSR:
https://www.youtube.com/watch?v=1NaGzNiFwWI&ab_channel=ChrisHay
but in csr Vite performs better to be honest, and yet not 10x faster as they advertised.

[thgh]

With Vite it takes 3 seconds to load the login page at first. When all is cached, it takes 500ms. Okayish numbers but not optimal. See #2530 . Payload currently needs to fully restart to apply changes, if that was reachitected it wouldn't matter which bundler was used.

If it is easy to switch bundlers as @jmikrut suggested, the best bundler will win over time 😛

[VadBe]

So easy to buy this dudes with huge "BLAZINGLY FAST" title 🗡️

[paulpopus]

What are the plans for server-side code not making it into the client bundle for security and performance reasons?

I'm particularly thinking of plugins which can't predict what bundler each project will use, currently in our plugins we just extend webpack and that just about works enough. Maybe if we had an interface? Or if it could be automated

[jmikrut]

We are talking about this as a team right now, and your question is spot-on. It's going to be tough to come up with a proper solution there.

It's actually making us second-guess our adapter pattern for Webpack specifically, because as you mentioned, plugins will not be able to predict what bundler each project would use, and it will splinter the community to have to worry about which plugins support which bundlers.

Right now, the biggest use case for extending the Webpack config is to alias server-only modules, and expose environment variables, which we could "abstract" into the adapter pattern so that specifying aliasas or environment variables could be done on a higher level. But that still doesn't solve for needing to extend Webpack for Tailwind support / MDX / etc.

In any case, I believe that we can solve the majority of Webpack extension needs by automatically removing server-only code. To do that, we'd need to build a loader that does the following:

1. Loop over every object property in the Payload config in AST
2. Remove all server-side properties like access, hooks, endpoints, etc.
3. Then remove any unused dependency imports from each file that the config references

Difficult, but not impossible. And even if we still did this, there is still a need to extend bundler configs.

It might be that if your plugin needs to do something bundler-specific, that it needs to do what it needs for each bundler that Payload supports. It will likely only be two, anyway.

Thoughts?

[paulpopus]

@jmikrut I'll likely need to think more about it or maybe try it out to really see if there's any issues with that.

You guys could deprecate Webpack entirely and eventually down the line drop it or make it 2nd class to another bundler that you put all your efforts in.

Those loader features sound great to me. I've mentioned it in the discord as well but to me it seems like the current way of needing to alias server-side code is one of the biggest pitfalls for beginners to get to grips with and it's also just not a nice devex.

I'm keen to have a better experience when it comes to plugin development so anything that simplifies that is a good thing. Part of that might be to start to provide helper functions if complexity grows via multiple bundlers or more things get the adapter pattern.

Do you foresee any issues with dropping Webpack entirely and just going all in on Vite or Turbo? Besides a breaking change of course.

[wiesson]

You guys could deprecate Webpack entirely and eventually down the line drop it or make it 2nd class to another bundler that you put all your efforts in.

This! ☝️

[gabrielsoldani]

@jmikrut Whichever route you choose, please make something that works like import "server-only" does for React Server Components.

It is simply a module that throws and is mocked in the server-side bundle, and you can import it whenever you want to assert your module is not being bundled in the client.

[clearfram3]

Most projects except for Vercel and co are standardizing on Vite/esBuild. Even Bun and Deno.

[JarrodMFlesch]

So we have landed on a new bundler pattern. By default (for now) we are attaching the webpack bundler in the sanitize function here. You can see how it imports the bundler from src/bundlers/[bundler-name]/bundler. Within those bundlers we are attempting to use the resolve.alias properties to replace this import with a mock module import. We do this in webpack and it works great. When the bundler goes to load the bundler file it should instead load the mock module and then the import tree should short circuit at the sanitize file and all of the files within the bundler file tree should not need to be loaded on the client.

You can see how this works with the current webpack bundler here.

No matter the bundler, the gist of mocking the bundler file looks like this:

import path from 'path';

const mockModulePath = path.resolve(__dirname, 'relative-path-to-mock-module');
const bundlerPath = path.resolve(__dirname, 'relative-path-to-bundler');

const getBaseConfig = () => {
  return {
    // ... rest of config
    resolve: {
      alias: {
        [bundlerPath]: mockModulePath,
      },
    },
  };
};

Currently I cannot seem to get this working for vite OR rspack. If we could get this working this would unblock us greatly and swift progress could be made. If anyone has any experience or insights as to how these bundlers handle alias resolutions, please chime in!

We have reached out to the contributors of rspack and we think there may be a bug with how module resolutions are handled. You can follow along with that here.

For those curious, the new WIP bundler branches are:
feat/vite-bundler (https://github.com/payloadcms/payload/tree/feat/vite-bundler)
feat/rspack-bundler (https://github.com/payloadcms/payload/tree/feat/rspack-bundler)

THE (first) GOAL: be able to use a new bundler and run yarn dev fields. Currently we get errors when running this command, both with vite and rspack. If you have any questions please let me know, I have spent a bit of time on this and will keep attempting to get this working.

THE (second) GOAL: be able to link a project with payload, build payload core and then build the linked project successfully.

Draft PR for Vite bundler can be found here

[thgh]

Will you also consider the possibility that the bundler needs to be restarted in dev mode? (and replace its configuration if it changed)
See #2563

[jmikrut]

Yes 100%, this is on Jarrod's and my radar, and we love the idea!