[PHP-DEV] Module or Class Visibility, Season 2

Mailing Lists php.internals
1

Hi!

It’s been a few days since I wanted to send this email to internals, but real life has been a bit chaotic so I apologize if it comes off as if I didn’t research the archives enough. I glossed over the Module conversation from 10 months ago and the one that recently surfaced and after deeply thinking about Rowan’s and Larry’s comments I wanted to throw this idea into the pit.

Lets preface the conversation with the fact that 1) a module system for PHP has been discussed for several years and 2) if there was an easy and perfect solution it would have been long implemented by now. With that in mind, I think there are mainly two major “camps”: the ones that would support something new similar to Node ESM vs CommonJS and those who won’t. Having dealt with this mess on the NodeJS side, I’m still on the side that would support it because even though it’s been 10 years worth of “mess”, it has greatly empowered progress. But I think PHP is too conservative to indulge this camp, so I’m going to focus on Rowan’s and Larry’s position of “we need something that builds on top of namespace, not replace it”.

If we consider how GitHub, Composer and Docker Hub works, we can pin a very important aspect of “namespaces”: {entity}/{project}. Entity may either be an individual or an organization, but the concept is mostly the same. Although it can be argued that PHP has nothing to do with that, I think that could be a “good-enough” foundation considering the complexity of the subject. Here is what we could do:


~~~
<?php declare(strict_types=1); namespace Acme\ProjectOne { public class Foo {} // same as class Foo {} private class Bar {} // only visible inside Acme\ProjectOne protected class Baz {} // visible inside Acme } namespace Acme\ProjectTwo { new \Acme\ProjectOne\Foo; // Work as always new \Acme\ProjectOne\Bar; // Fatal error: Uncaught Error: Cannot instantiate private class \Acme\ProjectOne\Bar from \Acme\ProjectTwo new \Acme\ProjectOne\Baz; // Works } namespace Corp\Corp { new \Acme\ProjectOne\Foo; // Work as always new \Acme\ProjectOne\Bar; // Fatal error: Uncaught Error: Cannot instantiate private class \Acme\ProjectOne\Bar from \Corp\Corp new \Acme\ProjectOne\Baz; // Fatal error: Uncaught Error: Cannot instantiate protected class \Acme\ProjectOne\Baz from \Corp\Corp } function (\Acme\ProjectOne\Foo $foo) {} // Works as always function (\Acme\ProjectOne\Bar $bar) {} // Open question: allow or disallow it? function (\Acme\ProjectOne\Baz $baz) {} // Open question: allow or disallow it? ``` ~~~ ``` This would allow public, private and protected classes in a way that I believe to be useful for the large ecosystem that surrounds Composer. From my extremely limited understanding of the engine, I think the easy/natural step would be to allow private/protected classes to be **received** outside its namespace because a type declaration does not trigger autoload. However, an important question is whether this is enough groundwork that could lead to optimizations that have been discussed when the topic of module is brought up. For instance, if type-hint outside the module is disallowed, could that make it easier to pack and optimize an entire module if we could instruct PHP how to load all symbols of a namespace all at once? I don't know.
··· Marco Deleu
2

On Tue, May 13, 2025, 18:35 Deleu <deleugyn@gmail.com> wrote:

Hi!

It’s been a few days since I wanted to send this email to internals, but real life has been a bit chaotic so I apologize if it comes off as if I didn’t research the archives enough. I glossed over the Module conversation from 10 months ago and the one that recently surfaced and after deeply thinking about Rowan’s and Larry’s comments I wanted to throw this idea into the pit.

Lets preface the conversation with the fact that 1) a module system for PHP has been discussed for several years and 2) if there was an easy and perfect solution it would have been long implemented by now.

If we consider how GitHub, Composer and Docker Hub works, we can pin a very important aspect of “namespaces”: {entity}/{project}.

Hi, thank you for starting this thread.

I have a similar thing, an idea that was sitting on my mind that I want to put in writing, so I will share it here.

First of all, I want to acknowledge the problems with big projects, big monoliths that can only have one composer.json and many libraries needed, that would eventually clash and make upgrades harder than they need to be.

Can we develop the modules concept without linking ourselves to namespaces or any other new entity?

I think it might be possible, if every symbol that is defined is attached to a module when it is loaded.
And we can use something like:

module(string $name, callable $closure);

The $closure is defined in the current module, but is executed in the new module.
Anything else follows the logic: the functions and methods are executed in the module where they are defined and associated with. Any new symbols defined inherit the current module.

When a module is defined by another parent module, a relation will be created between them and the parent module would get access to the child module symbols, but not also the other way around.

There is no need to change anything on the libraries code but only on composer and autoloading. That is the key point for easy adoption.
Composer-related, each installed package will be loaded in their own module.

What is yet to figure out:

  • can we define internal modules, where the child module symbols are visible only in the parent defining module and not any level up?
  • once a module defined, can the parent - child relation be redefined/changed/removed?

I apologize if this changes the solution discussion, but I didn’t wanted to start a new thread on the same topic.

–
Alex

3

On Tue, May 13, 2025 at 11:31 AM Deleu <deleugyn@gmail.com> wrote:

Hi!

This would allow public, private and protected classes in a way that I believe to be useful for the large ecosystem that surrounds Composer. From my extremely limited understanding of the engine, I think the easy/natural step would be to allow private/protected classes to be received outside its namespace because a type declaration does not trigger autoload.

This has been discussed before - making Namespaces something beyond what they are - a convenience string replace. That is given

namespace foo;
use otherns\boo;

function bar () {}
bar();
boo();

The engine quietly rewrites as

function foo\bar () {}

foo\bar();
otherns\boo();

The engine doesn’t store any notion of namespace directly, it’s just a string replace on the symbol table. And if I recall correctly this is also the reason that \ is the namespace operator - the original proposal was for namespaces to have the :: operator, same as static classes, but late in the implementation it was found that there were disambiguation problems and the choice was made to use the \ operator rather than solve those problems. I assume the problems were judged to be intractable, but I can’t say for sure.

Also, a namespace block could appear anywhere, so if a stubborn programmer really wanted to get access to a private or protected block they could put a namespace block into their own code with the same namespace.

This also doesn’t isolate a module with shared dependencies from changes to those dependencies. This makes composer libraries impossible to use effectively in the absence of a mechanism for coordinating those conflicts from the core. And for worse, the WP core team doesn’t want to provide such. And yes, it really isn’t the PHP dev team’s direct responsibility to step in and directly fix that problem. Indirectly perhaps? Only if it is a benefit to everyone who uses the language.

Neither PHP nor JavaScript deal with directories or other file collections. We have to look to Java and its jar files or golang modules for examples of this. And no, the PSR standard of mapping namespaces to directories has nothing to do with how the engine itself sees packages, namespaces, or the world. In truth it sees none of these because they don’t exist in the runtime in any meaningful way that I’m aware of. Phar perhaps? I know little of that system beyond the fact is the manner in which composer and a few other libs are distributed.

I sense, and I could be wrong here, that there is no appetite for moving beyond the one file at a time model. So the system I proposed last go round was still a one file solution.

Other ideas I’ve seen kicked around - file based privacy. In this schema a file can declare itself private (public is the assumed and Backward compatible default) and then mark exceptions to this as public. But for this to truly be useful I fear some file structure awareness would be needed and again, at the moment, PHP doesn’t do that.

A very long time ago I proposed a require into namespace mechanism. The engine by default attaches all incoming namespaces to the root /. I suggested (because I naively thought require was a function construct, not a statement, and was ignorant of the difference at the time) that “target namespace” could be the second argument of require and if provided all the symbols established by that file become attached to that namespace. This could only work if the autoloader correctly dispatched symbols to the correct namespaces, and I don’t have a clue how it could do that.

The need of the plugin community is code that just plugs into the app and doesn’t need to care about what the other applications are doing. This is currently possible only if the plugin provides all of its own code and if it does use composer libraries, it resorts to monkey-typing to change the names of all the symbols to something prefixed with the plugin name to avoid collisions. This approach is NOT optimal but it is the only one at present.

However, an important question is whether this is enough groundwork that could lead to optimizations that have been discussed when the topic of module is brought up. For instance, if type-hint outside the module is disallowed, could that make it easier to pack and optimize an entire module if we could instruct PHP how to load all symbols of a namespace all at once? I don’t know.

Doing that would likely involve giving the engine some notion of directory, again as Java does with JAR files, and PHP might do in PHAR files but I know embarrassingly little about them. Could the existing PHAR structure be used in some way for a starting point on this? I just don’t know, not without research.

4

On Tue, 13 May 2025, at 16:30, Deleu wrote:

If we consider how GitHub, Composer and Docker Hub works, we can pin a very important aspect of “namespaces”: {entity}/{project}. Entity may either be an individual or an organization, but the concept is mostly the same. Although it can be argued that PHP has nothing to do with that, I think that could be a “good-enough” foundation considering the complexity of the subject.

While a two-level namespace root for a project is common, it’s far from universal. Picking two examples from the first page of “popular packages” on packagist.org, Guzzle’s root namespace is one level ("GuzzleHttp") and the Symfony Console component’s root namespace is three levels ("Symfony\Component\Console").

So I think any module or visibility tied to namespaces would need a way to declare that prefix explicitly, not have the language assume it’s a particular length.

If we just want namespace visibility, we could use Scala’s approach, where the private modifier itself can be qualified:

private[\Symfony\Component\Console] class Foo { … }
private[\Symfony] class Bar { … }

If we want modules to have more existence - module-wide declares, optimisation, etc - then we need some way of declaring “this namespace prefix is a module” - a “module” keyword, or “declare_module” function, or something. Those are the lines that Larry and Arnaud were exploring along a while ago - see https://github.com/Crell/php-rfcs/blob/master/modules/spec-brainstorm.md and https://github.com/arnaud-lb/php-src/pull/10

What Michael Morris is talking about is really a completely different concept - it’s more like “containers”, in the sense of Docker, Kubernetes, etc, where different sections of code can be isolated, and declare classes with conflicting fully-qualified names. I don’t think it’s what most applications and libraries would want “modules” to be; it’s probably best thought of as a completely separate feature.

–
Rowan Tommins
[IMSoP]

5

On Wed, May 14, 2025 at 4:08 AM Rowan Tommins [IMSoP] <imsop.php@rwec.co.uk> wrote:

What Michael Morris is talking about is really a completely different concept - it’s more like “containers”, in the sense of Docker, Kubernetes, etc, where different sections of code can be isolated, and declare classes with conflicting fully-qualified names. I don’t think it’s what most applications and libraries would want “modules” to be; it’s probably best thought of as a completely separate feature.

Well, it’s what Go calls “modules”. It’s confusing because I was being truthful, not snarky, when I said “Ask 10 programmers for the definition of module and expect 12 answers.” I’m self trained, so I expect to get my terms wrong from time to time. But I know enough to identify problems and needs and I’ve tried to be clear on that.

I’m currently reading up on Phar and seeing exactly how suited it would be as a foundation for a module system. I’ve also been reading on how go approaches things, but go has package management baked into the compiler - PHP outsources this to userland. I’m going to guess that’s largely because of lack of staff - PHP has no large backers (leeches like Facebook that use it heavily and could back it yes, but not backers) and Go has Google.

6

On Wed, May 14, 2025, at 15:24, Michael Morris wrote:

On Wed, May 14, 2025 at 4:08 AM Rowan Tommins [IMSoP] <imsop.php@rwec.co.uk> wrote:

What Michael Morris is talking about is really a completely different concept - it’s more like “containers”, in the sense of Docker, Kubernetes, etc, where different sections of code can be isolated, and declare classes with conflicting fully-qualified names. I don’t think it’s what most applications and libraries would want “modules” to be; it’s probably best thought of as a completely separate feature.

Well, it’s what Go calls “modules”. It’s confusing because I was being truthful, not snarky, when I said “Ask 10 programmers for the definition of module and expect 12 answers.” I’m self trained, so I expect to get my terms wrong from time to time. But I know enough to identify problems and needs and I’ve tried to be clear on that.

I’m currently reading up on Phar and seeing exactly how suited it would be as a foundation for a module system. I’ve also been reading on how go approaches things, but go has package management baked into the compiler - PHP outsources this to userland. I’m going to guess that’s largely because of lack of staff - PHP has no large backers (leeches like Facebook that use it heavily and could back it yes, but not backers) and Go has Google.

Hi Michael,

Since it appears that nested classes probably won’t pass by tomorrow (and thus no need to even touch short-syntax classes); I was going to focus on modules next. As I mentioned in that thread, the two are very closely related on a technical level – it would have only taken 2-3 lines of changes to turn it into namespaces-as-modules and another 10 to turn it into proper modules (minus syntax support). However, I would implement it very differently knowing what I know today and with this as a goal (vs. nested classes). I have zero idea why people voted “no” and the people who expressed their reasons didn’t entirely make sense either. So, I suspect it was just down to a poorly worded RFC and/or misunderstanding of how it worked. I’ll have to revisit it again later.

Sorry for the vent; that’s not what this thread is about.

Modules. First of all, I’d be more than happy to help with the implementation if you’re up for some collaboration. Personally, here are my requirements I was going into it with:

  1. Impossible name collisions. If you want to name something Foo\Bar in your module and I want to name something Foo\Bar in mine; we should be free to do so. Implementing this is straightforward.
  2. Simple. I don’t want to rely on a package manager to create modules or even use them. I really like the simplicity of “require_once” from time to time, and I don’t want to see that go away. I have some ideas here, like require_module my-module.php
  3. Easy to optimize. A module should be compiled as a complete unit so opcache (or the engine itself) can make full use of the context. There are a lot of optimizations left on the table right now because the engine dynamically compiles one file at a time.
    I haven’t really even considered syntax too much; but I personally don’t want anything new – or at least, too “out there.” I want it to feel like a natural extension to the language rather than something bolted on.

I suspect there will need to be at least two new user-land elements to this:

  1. a “module loader” that operates similar to the unified class loader Gina proposed.
  2. importing modules and aliasing them (as needed).
    Would you be interested in collaborating further?

— Rob

7

On 14 May 2025 14:24:57 BST, Michael Morris <tendoaki@gmail.com> wrote:

Well, it's what Go calls "modules". It's confusing because I was being
truthful, not snarky, when I said "Ask 10 programmers for the definition of
module and expect 12 answers." I'm self trained, so I expect to get my
terms wrong from time to time. But I know enough to identify problems and
needs and I've tried to be clear on that.

I don't know much about Go, but at a glance it uses a similar model to JavaScript and Python where *classes don't have a universal name*, the names are always local. That's not a different kind of module, it's a fundamentally different *language design*.

If you want to use two different versions of Guzzle in the same application, the first problem you need to solve has nothing to do with require, or autoloading, or Phar files. The first problem you need to solve is that you now have two classes called \GuzzleHttp\Client, and that breaks a bunch of really fundamental assumptions.

For example:
- plugin1 uses Guzzle v5, runs "$client1 = new \GuzzleHttp\Client", and returns it to the main application
- The main application passes $client1 to plugin2
- plugin2 uses Guzzle v4
plugin2 runs "$client2 = new \GuzzleHttp\Client"

$client1 and $client2 are instances of different classes, with the same name! How does "instanceof" behave? What about "get_class"? What if you serialize and unserialize?

I think if you changed the language enough that those questions didn't matter, it would be a language fork on the scale of Python 2 to 3, or even Perl 5 to Raku (originally called "Perl 6"). Every single application and library would have to be rewritten to use the new concept of what a class is. And most of them would get absolutely no benefit, because they *want* to reference the same version of a class everywhere in the application.

That's why I think "containers" are the more useful comparison - you need some way to put not just plugin1 itself, but all the third-party code it calls, into some kind of sandbox, as though it was running in a separate process. If you can control what classes can go into and out of that sandbox, then in any piece of code, you don't end up with conflicting meanings for the same name - just as a Linux container can't open a network port directly on the host.

Regards,
Rowan Tommins
[IMSoP]

8

On Wed, May 14, 2025 at 10:57 AM Rowan Tommins [IMSoP] <imsop.php@rwec.co.uk> wrote:

I don’t know much about Go, but at a glance it uses a similar model to JavaScript and Python where classes don’t have a universal name, the names are always local. That’s not a different kind of module, it’s a fundamentally different language design.

That said, they call them modules. I’m not going to argue with them, what do I know?

If you want to use two different versions of Guzzle in the same application, the first problem you need to solve has nothing to do with require, or autoloading, or Phar files. The first problem you need to solve is that you now have two classes called \GuzzleHttp\Client, and that breaks a bunch of really fundamental assumptions.

Your fundamental assumption is that the different versions are loaded onto the same symbol. Given the problems you outline yourself, why do that?

You see, PHP doesn’t have a mechanism for symbol changing at compile time. Everything loads onto the root. Does it have to be that way? Or can a file compile onto a namespace, effectively prefixing that namespace.

For example:

  • plugin1 uses Guzzle v5, runs “$client1 = new \GuzzleHttp\Client”, and returns it to the main application
  • The main application passes $client1 to plugin2
  • plugin2 uses Guzzle v4
    plugin2 runs “$client2 = new \GuzzleHttp\Client”

If plugin 2 wants to use version 4 for whatever reason, why can’t it load it into \Plugin2\GuzzleHttpClient instead of onto the root??

This is what userland monkey-typers like Strauss do. It works, but there are issues with this solution outlined elsewhere.

That’s why I think “containers” are the more useful comparison - you need some way to put not just plugin1 itself, but all the third-party code it calls, into some kind of sandbox, as though it was running in a separate process. If you can control what classes can go into and out of that sandbox, then in any piece of code, you don’t end up with conflicting meanings for the same name - just as a Linux container can’t open a network port directly on the host.

Container, module, block, package, plugin, domain, division, fraction, lump, branch, sliver, splinter, constituent or whatever the hell else you call it, I don’t care. What I need is a way to manage package version conflicts which arise in the real world when plugins get abandoned or when coordinating having everyone change dependencies at the same time isn’t feasible.

9

On Wed, May 14, 2025, at 16:57, Rowan Tommins [IMSoP] wrote:

On 14 May 2025 14:24:57 BST, Michael Morris <tendoaki@gmail.com> wrote:

Well, it’s what Go calls “modules”. It’s confusing because I was being
truthful, not snarky, when I said “Ask 10 programmers for the definition of
module and expect 12 answers.” I’m self trained, so I expect to get my
terms wrong from time to time. But I know enough to identify problems and
needs and I’ve tried to be clear on that.

I don’t know much about Go, but at a glance it uses a similar model to JavaScript and Python where classes don’t have a universal name, the names are always local. That’s not a different kind of module, it’s a fundamentally different language design.

Go has some weird scoping, for sure. Everything is done by convention instead of syntax. In other words, if you want to export a symbol, you capitalize it; otherwise, it is lower-cased and thus private to the module. Then each directory is a module, and even in the same project, you cannot access another lower-cased symbol from another directory – er, module.

It is strange, and I don’t think it translates to PHP. PHP is generally explicit via syntax over convention.

If you want to use two different versions of Guzzle in the same application, the first problem you need to solve has nothing to do with require, or autoloading, or Phar files. The first problem you need to solve is that you now have two classes called \GuzzleHttp\Client, and that breaks a bunch of really fundamental assumptions.

As written, that simply isn’t possible in PHP because there is only one class allowed with a given name. Names of classes are global. I don’t think this has to be the case, though. Different languages take different approaches to this. For example, JavaScript allows each module to “close over” its dependencies so each module can import its own version of dependencies. Originally, there wasn’t even any deduplication, so you’d have 500 copies of left-pad or whatever. Then there is Go, which doesn’t allow you to have multiple versions of modules. You get exactly one version, which is similar to how PHP currently works with composer by default. However, with some massaging, you can “prefix” your imports so you get only your own version. I believe many WordPress plugins do this, so each plugin can use their own version of things.

I’m fairly certain we can do a similar thing so that each module gets its own unique ‘namespace’ in the class table such that two modules can define the same classes. So ModuleA and ModuleB can have Foo\Bar without conflicting with one another. From the user’s perspective, we can probably hide that technical detail from them but allow aliasing:

use module ModuleA; // import ModuleA’s namespace into our current namespace for this file
use module ModuleB as Baz; // import ModuleB’s namespace into our current namespace for this file, but with a prefix

Foo\Bar; // ModuleA\Foo\Bar
Baz\Foo\Bar; // ModuleB\Foo\Bar

I’m just spitballing syntax here, and I’m not suggesting it actually work like this, but I just want to illustrate that I think there are reasonable ways to allow modules to have conflicting names.

For example:

  • plugin1 uses Guzzle v5, runs “$client1 = new \GuzzleHttp\Client”, and returns it to the main application
  • The main application passes $client1 to plugin2
  • plugin2 uses Guzzle v4
    plugin2 runs “$client2 = new \GuzzleHttp\Client”

$client1 and $client2 are instances of different classes, with the same name! How does “instanceof” behave? What about “get_class”? What if you serialize and unserialize?

I’m of the opinion that the “names” of the module classes be distinct so that humans (and deserializers) know it is from a module. Something like [ModuleA]\Foo\Bar.

I think if you changed the language enough that those questions didn’t matter, it would be a language fork on the scale of Python 2 to 3, or even Perl 5 to Raku (originally called “Perl 6”). Every single application and library would have to be rewritten to use the new concept of what a class is. And most of them would get absolutely no benefit, because they want to reference the same version of a class everywhere in the application.

I suspect the hard part will be defining the module in the first place. IE, the “package.json” or “go.mod” or whatever it gets called. As composer isn’t a part of the PHP project, I don’t want to take it for granted, but I also don’t want to rely on it. That means each module may have to define its own “loader” or somehow define what PHP files encompass the module. As I mentioned earlier, PHP doesn’t usually operate by convention, though the community tends to force it to anyway (PSR-4 autoloading comes to mind immediately); so we’d need something that is explicit but automatable so the community can implement conventions.

That’s going to be the hard part.

That’s why I think “containers” are the more useful comparison - you need some way to put not just plugin1 itself, but all the third-party code it calls, into some kind of sandbox, as though it was running in a separate process. If you can control what classes can go into and out of that sandbox, then in any piece of code, you don’t end up with conflicting meanings for the same name - just as a Linux container can’t open a network port directly on the host.

Exactly.

Regards,
Rowan Tommins
[IMSoP]

— Rob

10

On 14 May 2025 21:50:25 BST, Michael Morris <tendoaki@gmail.com> wrote:

Container, module, block, package, plugin, domain, division, fraction,
lump, branch, sliver, splinter, constituent or whatever the hell else you
call it, I don't care.

I know you think I'm just being pedantic about names, but
what I was trying to get across was the distinction between different features that we could have both of, because they're solving separate problems.

It's basically about where the dividing line is. If you want this hierarchy of dependencies:

           +-- Plugin1 -- AcmeSDK v2 -- Guzzle v5
App --+
           +-- Plugin2 -- AcmeSDK v1 -- Guzzle v4

The requirement is not to hide Guzzle from Plugin1 - maybe it *needs* to create an object from Guzzle and pass it into AcmeSDK.

Instead, the requirement is for Plugin1 to hide both AcmeSDK *and* Guzzle from Plugin2. You don't want 7 different "things" (whatever you want to call them) in that diagram, you want 3 (App, Plugin1-and- recursive-dependencies, Plugin2-and- recursive-dependencies).

The Linux container analogy is something like this:

           +-- container { WordPress -- PHP -- Apache }
Host --+
           +-- container { MediaWiki -- PHP -- Apache }

The goal of containers isn't to hide WordPress from Apache or vice versa, it's to hide the two copies of Apache and PHP from each other. There are plenty of things hidden *inside* Apache (the equivalent of "private classes") but that's a completely separate concept.

I wasn't saying the feature had to be called "containers", just that the analogy might be useful.

Rowan Tommins
[IMSoP]

11

On 14 May 2025 22:27:32 BST, Rob Landers <rob@bottled.codes> wrote:

As written, that simply isn't possible in PHP because there is only one class allowed with a given name. Names of classes are global. I don't think this has to be the case, though. Different languages take different approaches to this. For example, JavaScript allows each module to "close over" its dependencies so each module can import its own version of dependencies.

I would say that JavaScript doesn't just *allow* this, as an added feature, it *requires* it, as a fundamental design decision:

- In JavaScript, Python, etc, when you declare a function or class, you are creating an anonymous object, and assigning it to a local variable. Code reuse requires you to pass that object around.
- In PHP, Java, C#, etc, when you declare a function or class, you are adding a permanent named item to a global list. Code reuse is about knowing the global names of things.

It's worth noting that JavaScript didn't need to add *any* features to make NPM, Bower, etc work; everything they do is based on the fact that declarations are objects which can be passed around at will.

That's why I don't think "JavaScript can do it" is relevant, because the *way* JavaScript does it is impossible in PHP. We're much better off looking at how *PHP* works, and what problems we're actually trying to solve.

And that in turn is why I was reaching for Linux containers as an alternative analogy, to think about the problem without jumping to the wrong solution.

Rowan Tommins
[IMSoP]

12

On Thu, May 15, 2025, at 10:11, Rowan Tommins [IMSoP] wrote:

On 14 May 2025 22:27:32 BST, Rob Landers <rob@bottled.codes> wrote:

As written, that simply isn’t possible in PHP because there is only one class allowed with a given name. Names of classes are global. I don’t think this has to be the case, though. Different languages take different approaches to this. For example, JavaScript allows each module to “close over” its dependencies so each module can import its own version of dependencies.

I would say that JavaScript doesn’t just allow this, as an added feature, it requires it, as a fundamental design decision:

  • In JavaScript, Python, etc, when you declare a function or class, you are creating an anonymous object, and assigning it to a local variable. Code reuse requires you to pass that object around.
  • In PHP, Java, C#, etc, when you declare a function or class, you are adding a permanent named item to a global list. Code reuse is about knowing the global names of things.

It’s worth noting that JavaScript didn’t need to add any features to make NPM, Bower, etc work; everything they do is based on the fact that declarations are objects which can be passed around at will.

That’s why I don’t think “JavaScript can do it” is relevant, because the way JavaScript does it is impossible in PHP. We’re much better off looking at how PHP works, and what problems we’re actually trying to solve.

And that in turn is why I was reaching for Linux containers as an alternative analogy, to think about the problem without jumping to the wrong solution.

Rowan Tommins
[IMSoP]

Hey Rowan,

When working on nested classes, I did spend quite a bit of time tinkering with alternative implementations. One of those implementations was having the ability for classes to have their own class tables (both literally and emulated via name mangling) which would have allowed for classes to have private classes that could share names with external classes. This turned out to be an utter disaster of an idea for many reasons. Namely, PHP doesn’t really have any native support for shadowing names. Sure, there is aliasing via use statements, but that only works for classes outside the current namespace. As long as we can guarantee that a module acts as a special namespace (under the hood), the only potential for collisions will be in the module itself.

All that is to say that I don’t think comparing PHP to JavaScript is appropriate when considering modules. JavaScript doesn’t have types, so I can pass you an EpicStringV2 when you’re expecting an EpicStringV1, and as long as my EpicStringV2 has the right prototypical behavior and data, it will work just fine. PHP is typed, and fairly strongly typed. There is effectively no way to have multiple versions of the same type running around a codebase and pass type checks. Changing this would be effectively impossible and probably unsound from a type-theory perspective.

— Rob

13

On Thu, May 15, 2025, at 10:35, Rob Landers wrote:

On Thu, May 15, 2025, at 10:11, Rowan Tommins [IMSoP] wrote:

On 14 May 2025 22:27:32 BST, Rob Landers <rob@bottled.codes> wrote:

As written, that simply isn’t possible in PHP because there is only one class allowed with a given name. Names of classes are global. I don’t think this has to be the case, though. Different languages take different approaches to this. For example, JavaScript allows each module to “close over” its dependencies so each module can import its own version of dependencies.

I would say that JavaScript doesn’t just allow this, as an added feature, it requires it, as a fundamental design decision:

  • In JavaScript, Python, etc, when you declare a function or class, you are creating an anonymous object, and assigning it to a local variable. Code reuse requires you to pass that object around.
  • In PHP, Java, C#, etc, when you declare a function or class, you are adding a permanent named item to a global list. Code reuse is about knowing the global names of things.

It’s worth noting that JavaScript didn’t need to add any features to make NPM, Bower, etc work; everything they do is based on the fact that declarations are objects which can be passed around at will.

That’s why I don’t think “JavaScript can do it” is relevant, because the way JavaScript does it is impossible in PHP. We’re much better off looking at how PHP works, and what problems we’re actually trying to solve.

And that in turn is why I was reaching for Linux containers as an alternative analogy, to think about the problem without jumping to the wrong solution.

Rowan Tommins
[IMSoP]

Hey Rowan,

When working on nested classes, I did spend quite a bit of time tinkering with alternative implementations. One of those implementations was having the ability for classes to have their own class tables (both literally and emulated via name mangling) which would have allowed for classes to have private classes that could share names with external classes. This turned out to be an utter disaster of an idea for many reasons. Namely, PHP doesn’t really have any native support for shadowing names. Sure, there is aliasing via use statements, but that only works for classes outside the current namespace. As long as we can guarantee that a module acts as a special namespace (under the hood), the only potential for collisions will be in the module itself.

All that is to say that I don’t think comparing PHP to JavaScript is appropriate when considering modules. JavaScript doesn’t have types, so I can pass you an EpicStringV2 when you’re expecting an EpicStringV1, and as long as my EpicStringV2 has the right prototypical behavior and data, it will work just fine. PHP is typed, and fairly strongly typed. There is effectively no way to have multiple versions of the same type running around a codebase and pass type checks. Changing this would be effectively impossible and probably unsound from a type-theory perspective.

— Rob

Haha, I just reread your email and realized we’re basically saying the same thing (I think?).

— Rob

14

The Problem: Interoperability.

That’s really it. Scenario
Alice provides whatchamacallit A that depends on other whatchamacallit D to work.
Bob provides whatchamacallit B that also depends on D.
Charles is using A and B.
D gets updated with a new incompatible API to its prior version.
Alice publishes an update which includes a security fix.
Bob retired.
Charles, who can’t program, can’t update to Alice’s latest code. His site eventually gets pwned.

That’s the problem. Packages with dependencies are not interoperable at this time. They must be self contained. This is why WordPress doesn’t support Composer at all.

Drupal, Laravel et al bypass this problem by forcing all their whachamacallits to stay on the same version. This has limited their market penetration compared to WordPress because, despite being significantly superior codebases in all respects, they aren’t user friendly to someone who doesn’t code at all.

The Solution (10,000 overview)
Composer could be made to allow interoperable packages, but it will need support at the language level to do so. Specifically, it needs to know who wants what. It can then make decisions based on that information.

Composer’s primary link to the language is the autoload closure it provides. That closure currently takes one argument - the fully qualified name of the symbol to be loaded - currently almost always classes as for various reasons function autoloading isn’t a thing. Can it not take a second argument to modify its behavior? The current behavior is to flat require the file if it is found in accordance to whichever schema is in use. Perhaps we don’t want that anymore - perhaps we want to return the file path to use. This allows the engine to make decisions about how exactly to include the file, including the possibility of monkey typing it as can be done in userland, though when done in userland this effectively generates a new package.

(5,000 ft. overview)
Suppose we have a whatchamacallit that declares its namespace as a new root independent of / . If a file inclusion happens in this namespace, this namespace prepends everything in the included file. So if I do a file include in the \MyPlugin namespace and that file declares its namespace as Twig, it will become \MyPlugin\Twig.

That works, but direct file include is no longer the PHP norm though. Autoloading is. So we need to tell the Autoloader that we want a file path returned - do NOT require the file yourself in your namespace. This could be as simple as a boolean flag of true sent to the autoloader. BUT it isn’t - the autoloader (usually composer) needs to know the identity of this requestor because by configuration in the package json (the details of which are wildly out of scope) it might change which file path it returns.

When the engine gets the path it does the include and the prepending business on the fly that Strauss and similar packages already do in userland.

(2,500 ft overview)

The above I think would more or less work, but it would lead to massive code duplication as Whatchamacallit A and B now have their own D’s at \A\D and \B\D (assuming namespaces match whatchamacallit names).

Here’s what I think would prevent that:

A asks the autoloader for D. The autoloader returns a file path and the engine mounts to \D
B asks for D. The autoloader returns a different file path so the engine mounts to B\D and rewrites the D file with the new namespace the same way Stauss would have done.

This works except for the problem of who had the older version, A or B? and what order are A and B going to be asking - cause depending on the application’s architecture this order is not guaranteed.

To solve this the autoloader can tell the engine it is safe to mount the file on root using an array return of [path, true] and mount on the whatchamacallit’s namespace if [path, false]. So

A asks for D. Autoloader returns [path, false]. Engine maps to \A\D and monkey types D as needed.
B asks for D. Autoloader returns [path, true]. Engine maps to \D

Non whatchamacallit code at namespace C asks for D. It will get the same version B is using and the autoloader shouldn’t be queried unless C makes this ask before B.

When C asks the autoloader gets (string RequestedSymbol, null) so it can either do the require itself or return a string, either will work (and it has to be this way for backwards compat).
When B asks the autoloader gets ( Requested, ‘B’ ) and it should return [path, true]

I hope the above is followable. It’s more of a morning brainstorm than a spec.

15

On 20 May 2025 15:04:49 BST, Michael Morris <tendoaki@gmail.com> wrote:

The Problem: Interoperability.

That's really it. Scenario
Alice provides whatchamacallit A that depends on other whatchamacallit D to
work.
Bob provides whatchamacallit B that also depends on D.
Charles is using A and B.
D gets updated with a new incompatible API to its prior version.
Alice publishes an update which includes a security fix.
Bob retired.
Charles, who can't program, can't update to Alice's latest code. His site
eventually gets pwned.

Let me correct something here. The whole reason I was bringing in the distinction between "module" and "container" is that B and C are one kind of thing, but D is a *different* kind of thing.

D is something like Guzzle. There is zero motivation for Guzzle to be rewritten in a way that forces its dependencies to be isolated. It depends on packages like "psr/http-client" whose *entire purpose* is to define interfaces that multiple packages agree on.

A, meanwhile, isn't a thing at all; it's just any old PHP code - in your example, the whole spaghetti of WordPress core.

B and C are the only "whatchamacallits" - they are, in your example, WordPress plugins. They are the thing you want a boundary around, the black box you want conflicting names to be hidden by.

Suppose we have a whatchamacallit that declares its namespace as a new root
independent of / . If a file inclusion happens in this namespace, this
namespace prepends everything in the included file. So if I do a file
include in the \MyPlugin namespace and that file declares its namespace as
Twig, it will become \MyPlugin\Twig.

What does it mean, exactly, for a file inclusion to "happen in a namespace"? Bear in mind, most of the files we want to load, whether explicitly or via an autoloader, are not requests from A (the WordPress plugin) directly to D (Guzzle); they are references between files inside D, or in further dependencies that A has no idea about at all.

What PHP needs to track, somehow, is that a whole bunch of code is "inside" something, or "coloured by" something, in a way that is completely recursive.

That works, but direct file include is no longer the PHP norm though.
Autoloading is. So we need to tell the Autoloader that we want a file path
returned - do NOT require the file yourself in your namespace.

This for me is a non-starter: the existing packages which you want to make use of have little or no motivation to adapt to this new system.

Again, think about Linux containers: applications don't get a message saying "you're running in a container, please use different file I/O conventions"; they *think* they are accessing the root filesystem, and the host *silently* rewrites the access to be somewhere in the middle of a larger tree.

I think the way it would need to work would be some global state inside the compiler, so that regardless of how the code ended up being loaded, an extra transform was included in the compilation pipeline to attempt to rewrite all definitions, and all references to those definitions.

(I say "attempt", because even with all this built into the compiler, PHP's highly dynamic nature means there would be code patterns that the rewriter would not see; the whole thing would come with a bunch of caveats.)

The above I think would more or less work, but it would lead to massive
code duplication as Whatchamacallit A and B now have their own D's at \A\D
and \B\D (assuming namespaces match whatchamacallit names).

I don't think this is a problem that can or should be solved.

Imagine A and B both use the same version of E, but different versions of D; and E references D. If we try to de-duplicate, we load one copy of E, but when called from A it needs to reference \A\D and when called from B it needs to reference \B\D. Clearly, that's not going to work, so we're forced to define a separate \A\E and \B\E.

Note that this is completely different from any de-duplication of files on disk that a package manager might perform. It's a bit like the same C source file being compiled into two different object files with different #defines in effect.

I'm still not convinced that all this complexity actually leaves you better off than building a Composer plugin that automatically applies the rewriting to a whole directory at source code level.

Rowan Tommins
[IMSoP]

16

On Tue, May 20, 2025 at 6:18 PM Rowan Tommins [IMSoP] <imsop.php@rwec.co.uk> wrote:

On 20 May 2025 15:04:49 BST, Michael Morris <tendoaki@gmail.com> wrote:

The Problem: Interoperability.

That’s really it. Scenario
Alice provides whatchamacallit A that depends on other whatchamacallit D to
work.
Bob provides whatchamacallit B that also depends on D.
Charles is using A and B.
D gets updated with a new incompatible API to its prior version.
Alice publishes an update which includes a security fix.
Bob retired.
Charles, who can’t program, can’t update to Alice’s latest code. His site
eventually gets pwned.

Let me correct something here. The whole reason I was bringing in the distinction between “module” and “container” is that B and C are one kind of thing, but D is a different kind of thing.

D is something like Guzzle. There is zero motivation for Guzzle to be rewritten in a way that forces its dependencies to be isolated. It depends on packages like “psr/http-client” whose entire purpose is to define interfaces that multiple packages agree on.

A, meanwhile, isn’t a thing at all; it’s just any old PHP code

I’ll stop you there. You are deliberately misrepresenting what I wrote and even a cursory glance at it makes that clear. You are not trying to be constructive in any way, you’re trolling.

17

I think this is why Rowan keeps telling you to call or compare this with “Containers” and not modules. When I opened this thread, my interest was in bundling multiple files all at once so that the PHP engine can make assumptions and optimizations about it and expand namespace to also allow class visibility. To me, and I believe to a vast majority of PHP users, interoperability is not a problem. We don’t need, and, depending on how we position it, don’t want multiple versions of the same package on a single application.

···

Marco Deleu

18

On Wed, May 21, 2025 at 7:23 AM Deleu <deleugyn@gmail.com> wrote:

On Tue, May 20, 2025 at 11:08 AM Michael Morris <tendoaki@gmail.com> wrote:

The Problem: Interoperability.

That’s really it.

I think this is why Rowan keeps telling you to call or compare this with “Containers” and not modules.

Which is why I switched to calling them a “whachamacallit” to set the issue aside and focus on concepts, not terms. He insisted on trolling despite that.

When I opened this thread, my interest was in bundling multiple files all at once so that the PHP engine can make assumptions and optimizations about it and expand namespace to also allow class visibility. To me, and I believe to a vast majority of PHP users, interoperability is not a problem. We don’t need, and, depending on how we position it, don’t want multiple versions of the same package on a single application.

Not all of us have the pleasure of living in ivory towers like you do. In the real world code isn’t perfect and problems need to be solved.

No one wants multiple versions of the same package in a single application. How stupid do you think I am?

We don’t always get what we want - the need for multiple package versions does arise in the real world which is why other languages such as golang and JavaScript can allow it.

19

On 21 May 2025 10:20:19 BST, Michael Morris <tendoaki@gmail.com> wrote:

I'll stop you there. You are deliberately misrepresenting what I wrote and
even a cursory glance at it makes that clear. You are not trying to be
constructive in any way, you're trolling.

I'm sorry you got that impression. I can assure you that I am not trolling, and my email was an entirely genuine attempt to engage with the problem that you were trying to describe.

My understanding of the example is that there are two WordPress plugins, which want independent sets of Composer dependencies. There might be 20 different Composer packages used by each plugin, but those packages don't need any special relationship with *each other*, they just need a special relationship with *the WordPress plugin*.

So if we can come up with a solution where only the WordPress plugins need to be changed, and you can use whatever dependencies you want without waiting for them to be changed to a new way of working, is that not a good thing?

I've tried several times to explain why I think Linux containers are a good analogy; I'm not sure if you didn't understand, or just didn't agree, so I don't know what else I can say.

Rowan Tommins
[IMSoP]

20

On 21 May 2025 13:26:27 BST, "Rowan Tommins [IMSoP]" <imsop.php@rwec.co.uk> wrote:

My understanding of the example is that there are two WordPress plugins, which want independent sets of Composer dependencies. There might be 20 different Composer packages used by each plugin, but those packages don't need any special relationship with *each other*, they just need a special relationship with *the WordPress plugin*.

Looking closely, I see I did make one honest mistake: in your example, the WordPress plugins are A and B, not B and C. So my sentence should have read "A and B are one kind of thing, but D is a different kind of thing".

Rowan Tommins
[IMSoP]