Magento 2: Composer and Components

This entry is part 3 of 3 in the series Magento 2 and Composer. Earlier posts include Magento 2: Composer, Marketplace, and Satis, and Magento 2: Composer Plugins. This is the most recent post in the series.

One question I keep getting from new Magento 2 developers, (and we’re all new Magento 2 developers) is, “How should I organize my project files?”. Since Magento has heavily restructured code organization around Composer, it’s not always clear how files should be organized during development, organized for distribution, and how (if at all) to move between these two modes.

While this article won’t definitively answer those questions, we will dive into some of the formalization behind Magento 2 components, as well as how Magento 2’s Composer integration works. With this information in hand, you should be able to come up with a project structure that works for you and your team.

Magento 2 Components

Magento 1 had an informal, inconsistent idea of components. The best way to think about Magento components is

A group of source files, in various formats, with a single purpose in the system

If that’s a little vague, see our previous comments about informal and inconsistent. Speaking more concretely, the four component types in Magento are

  • Modules
  • Themes
  • Language Packs
  • Code Libraries

In Magento 1, modules were pretty well defined and self contained

A folder of files with an etc/config.xml, with developers making Magento aware of the module via an app/etc/modules/Package_Namespace.xml file

Themes were a little less defined and a little less self contained

A collection of files under app/design/[area]/[package]/[name], with developers making Magento aware of the theme via a setting in core_config_data. Unless it’s the admin theme in which case you need a module. Also, modules are responsible for adding the layout XML files to themes. Also, while we’re here, modules aren’t all that self contained either because if they want to use phtml templates then the templates need to be in a theme folder

Things start to get really vague with language packs

A collection of key/value csv files located inapp/locale/[language]/Packagename_Modulename.csv. Also we’re just going to drop email templates in here because reasons

And Magento 1 barely had the concept of a generic code library.

Um, yeah, maybe just drop them in lib and add that path to PHP’s autoloader? And look in the code pool folders too? And maybe just add a top level js folder for javascript libraries? Unless they go in skin?

Oh right! Skins! Magento 1 also had a (now dropped) concept of skins. Skins were best defined as

Any CSS or javascript file that doesn’t belong in a theme.

Where CSS or javascript file that doesn’t belong in a theme was defined as

Any CSS or javascript file that doesn’t belong in a skin

While, in practice, norms developed over time and development wasn’t as chaotic as I’m describing, Magento 1’s lack of formalization around components did make the system harder to work with, particularly if you were trying to redistribute Magento 1 code for reuse.

Magento 2 formalizes the idea of components, and this formalization means the core Magento system, and other external systems (i.e. Composer) can deal with these components in a sane and reasonable way.

Magento 2 Components

In Magento 2, a component is

A group of files, under a top level directory (with sub-folders allowed), with aregistration.php file defining the type of component.

That’s it. There’s nothing about how the components work, interact with the system, or interact with other components. Those things aren’t the concern of the component system.

As of this writing, there are four component types in Magento 2

  • Modules
  • Themes
  • Libraries
  • Language Packs

Let’s take a look at this in action. Consider the Magento_AdminNotification module. This is a collection of files, under a top level directory (AdminNotification), with aregistration.php file. If we take a look at registration.php

#File: app/code/Magento/AdminNotification/registration.php
<?php
/**
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::MODULE,
    'Magento_AdminNotification',
    __DIR__
);

We can see this file registers a component via the static\Magento\Framework\Component\ComponentRegistrar::register method. This component is a module (\Magento\Framework\Component\ComponentRegistrar::MODULE), its identifier isMagento_AdminNotification, and you can find its files in the __DIR__ folder, (i.e. the same directory registration.php is in via PHP’s magic __DIR__ constant).

Next, consider the Luma theme. Again, a collection of files, under a top level directory (luma), with a registration.php file.

#File: app/design/frontend/Magento/luma/registration.php
<?php
/**
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::THEME,
    'frontend/Magento/luma',
    __DIR__
);

Here the component type is a theme (\Magento\Framework\Component\ComponentRegistrar::THEME), and its name isfrontend/Magento/luma.

Even though the Magento GitHub project has these files in familiar locations, (app/code,app/design, etc.), thanks to Magento 2’s new component system, these directories can be located anywhere, so long as as the module, theme, library, or language pack correctly defines its registration.php file.

How Magento Loads Components

At this point, the systems minded among you are probably wondering how Magento 2 loads and identifies components. It’s one thing to say so long as the module, theme, library, or language pack correctly defines its registration.php file, but the system still needs to load these files, and that means there are rules.

In order to get Magento to recognize your module, theme, code library, or language pack (i.e. your component), you need Magento to read your component’s registration.phpfile. There are two ways to get Magento to read your registration.php file

  1. Place your component in one of several predefined folders
  2. Distribute your module via Composer, and use Composer’s autoloader features

Of the two methods, the second is the preferred and recommended way of distributing Magento 2 modules. For development, the first offers a convenient way to get started on a component, or checkout/clone a version control repository to a specific location. The first also offers a non-composer way for extension developers to distribute their components.

Predefined Folders

At the time of this writing, Magento 2 will scan the following folders/files for components (the patterns below are for the glob function).

app/code/*/*/cli_commands.php
app/code/*/*/registration.php
app/design/*/*/*/registration.php
app/i18n/*/*/registration.php
lib/internal/*/*/registration.php
lib/internal/*/*/*/registration.php

This is what allows you to place modules in app/code/Packagename/Modulename, or themes in app/design/[area]/[package]/[name], etc. Magento will explicitly look forregistration.php files to load at these locations. Also of interest are theapp/code/*/*/cli_commands.php files — this appears to be a way for a module to register command line classes without using di.xml.

It’s not clear if these folders were added as a stop-gap measure while Magento 2 gets everyone moved over to Composer distribution, or if they’ll stick around for the long term. If you’re curious, Magento does this registration check in the following file

#File: app/etc/NonComposerComponentRegistration.php
<?php
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

$pathList[] = dirname(__DIR__) . '/code/*/*/cli_commands.php';
$pathList[] = dirname(__DIR__) . '/code/*/*/registration.php';
$pathList[] = dirname(__DIR__) . '/design/*/*/*/registration.php';
$pathList[] = dirname(__DIR__) . '/i18n/*/*/registration.php';
$pathList[] = dirname(dirname(__DIR__)) . '/lib/internal/*/*/registration.php';
$pathList[] = dirname(dirname(__DIR__)) . '/lib/internal/*/*/*/registration.php';
foreach ($pathList as $path) {
    // Sorting is disabled intentionally for performance improvement
    $files = glob($path, GLOB_NOSORT);
    if ($files === false) {
        throw new \RuntimeException('glob() returned error while searching in \'' . $path . '\'');
    }
    foreach ($files as $file) {
        include $file;
    }
}    

If you’re researching how a future version of Magento 2 handles scanning for components, this would be a good place to start.

Composer Distribution

The other way to have Magento notice your component is to distribute your component via Composer. We’re going to assume you have a basic familiarity with Composer, but for the purposes of this article, all you really need to know is

Composer allows you to ask for a package of PHP files, and have that package downloaded to the vendor/ folder

If you’re interested in learning more about Composer, the previous articles in this series, my Laravel, Composer, and the State of Autoloading, and the Composer manual are a good place to start.

So, assuming you have your Magento component in GitHub (or a different source repository Composer can point at), and your component has a registration.php file, the only question left is How do we get Magento to look at our registration.php file.

Rather than have Magento scan all of vendor/ for registration.php files, (an approach that could quickly get “O^N out of hand” as the number packages grows), Magento uses Composer’s file autoloader feature to load each individual component’sregistration.php file.

If that didn’t make sense, an example should clear things up. Assuming you’ve installed Magento via the Composer meta-package, or installed it via the archive available via magento.com (which is based on the meta-package), take a look at the catalog module’scomposer.json file.

#File: vendor/magento/module-catalog/composer.json
{
    "name": "magento/module-catalog",
    //...
    "autoload": {
        "files": [
            "registration.php"
        ],
        "psr-4": {
            "Magento\\Catalog\\": ""
        }
    }
    //...
}

The autoload section is where you configure the PHP class autoloader for a Composer package. This is covered in great detail in my Laravel, Composer, and the State of Autoloading series. The section we’re interested in today is here

#File: vendor/magento/module-catalog/composer.json
"files": [
    "registration.php"
],

The files autoloader section was originally intended as a stop gap measure for older PHP packages that had not moved to a PSR-0 (and later, PSR-4) autoloader system. Composer’s autoloader (not during install or update, but when your application is running) will automatically include any files listed in files (with the specific package as the base directory), and package developers can do whatever they need to do to setup their pre-PSR autoloaders.

Over the years, many frameworks have taken the simplicity and flexibility of the filesautoloader and turned it to different purposes. Magento 2 is no exception. The above autoload configuration ensures Composer will always load the file at

#File: vendor/magento/module-catalog/registration.php
<?php
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::MODULE,
    'Magento_Catalog',
    __DIR__
);    

This, as we’ve already learned, will register the component. The same holds true for third party components (i.e. yours!) — make sure you’ve created a registration.php file with the correct registration code for your component type, and then include an identical filesautoloader.

Here’s an example of each component type from Magento’s core.

Module

#File: vendor/magento/module-weee/registration.php
<?php
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::MODULE,
    'Magento_Weee',
    __DIR__
);

Theme

#File: vendor/magento/theme-frontend-luma/registration.php
<?php
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::THEME,
    'frontend/Magento/luma',
    __DIR__
);

Library

#File: vendor/magento/framework/registration.php
<?php
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::LIBRARY,
    'magento/framework',
    __DIR__
);

Language Pack

#File: vendor/magento/language-de_de/registration.php
<?php
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::LANGUAGE,
    'magento_de_de',
    __DIR__
);

Invalid Assumptions

There’s one last important thing to take away from this, even if you’re not responsible for packaging your company’s Magento work. It’s no longer safe to make assumptions aboutwhere a folder is located in located in the Magento hierarchy. If you’re trying to find a specific file in Magento, it’s more important than ever to learn your way around theMagento\Framework\Module\Dir helper class.

Daily Work

So, now that we have a better understanding of what a component is, and how Magento loads components into the system, that still leaves us with our original question. Where should the code for our in progress Magento projects go? How should we store our projects in source control?

Unfortunately — there’s no clear answer, and a lot will depend on the sort of project you’re working on. Are you an extension developer? A theme developer? A system integrator/store builder or someone integrating with a Magento system? Do you want your working source repository to be the same repository Composer reads from? What tooling is your team is familiar with? While there certainly are approaches that are “better” for each scenario, from a programmer’s point of view Magento 2’s still too new to know for sure.

For what its worth, I’ve been creating symlinks to my source repositories so thatNonComposerComponentRegistration.php finds my components, using a build process to create the final Composer accessible repository, and temporarily patching over any issues Magento has with symlinks.

Part of being a Magento 2 developer will be figuring this out for your own team, even if you’re just a team of one.

Magento 2: Composer Plugins

This entry is part 2 of 3 in the series Magento 2 and Composer. Earlier posts include Magento 2: Composer, Marketplace, and Satis. Later posts include Magento 2: Composer and Components.

In our last article, we talked a bit about Magento 2’s use of Composer, and touched on the Composer meta-package installation method. One high level take away was, when you use Composer’s create-project method to start a new Magento project, you are

  1. Fetching the latest version of a specific Composer package (magento/project-community-edition) hosted at repo.magento.com
  2. Running composer install for that project’s composer.json file

The curious among you may have noticed something strange about this. If we use the --no-install option to skip step #2 above (i.e. only fetch magento/project-community-edition),

$ composer create-project --no-install --repository-url=https://repo.magento.com/ magento/project-community-edition

we’ll see a pretty sparse project folder

$ ls project-community-edition
README.md    composer.json    update    

The README.md file is a basic Welcome to Magento affair, and the composer.json file contains the actual Composer packages you’ll need to install Magento. That’s whymagento/project-community-edition is called a “meta” package — it’s not he actual package, and most software engineers aren’t english or philosophy majors and enjoy stretching the definition of “meta”.

The update folder contains a snapshot of the Magento Component Manager application (i.e Market Place updater), which is a separate project from the Magento core. Why this feature is a separate application, and why Magento distributes it like this is a story for another time.

All in all, relatively straight forward. However, after running composer install (or running create-project without the --no-install flag), we end up with the following

$ ls project-community-edition/
CHANGELOG.md                        dev
CONTRIBUTING.md                     index.php
CONTRIBUTOR_LICENSE_AGREEMENT.html  lib
COPYING.txt                         nginx.conf.sample
Gruntfile.js                        package.json
LICENSE.txt                         php.ini.sample
LICENSE_AFL.txt                     phpserver
README.md                           pub
app                                 setup
bin                                 update
composer.json                       var
composer.lock                       vendor

That’s a huge difference. A plethora of files and folders. Many of these files are necessary for Magento to run, others are informational, and still others are sample configurations. However, if you’re new to the Composer eco-system, you may be wondering

Where the heck did all these extra files and folders come from? I thought Composer would only update files in /vendor.

Today we’re going to explain how these files get here, and why you’ll need to be hyper aware of this as a Magento 2 developer. To start, we’ll need to dive into some less known features of Composer

Composer: Plugins and Scripts

Composer bills itself as a Dependency Manager for PHP. While this is true, and dependency management is an important part of a PHP project, Composer is really a foundational framework for PHP development, and serves the same role that linkers do in the C/C++ world.

Yes yes, I know, from a computer science point of view linkers and Composer couldn’t be further apart. However, the end result of a linker is, the C programmer stops needing to worry about how they incorporate code from other libraries into their program. In a similar way, Composer does the same thing for PHP — if a project conforms to what Composer expects in terms of directory structure and autoloading, and a PHP developer conforms to what Composer expects from a PHP program (i.e., includes the Composer autoloader), the developer stops needing to worry about how they should include other people’s code in their own systems.

When considered from this point of view — that Composer is, itself, just another programmatic framework that your code sits on top of — it makes more sense that Composer would have a plugin system for changing, altering, and extending its behavior. There are two main systems programmers have for altering the behavior of Composer. These systems are scripts, and plugins.

Scripts and plugins share a base set of concepts, but have a few key distinctions. Scripts provide a way, in the project composer.json file, to take additional programmatic action when composer triggers certain events. These events are listed in the Composer manual, and include things like pre-and-post composer install running.

Plugins, on the other hand, provide the same mechanism for individual packages that are part of a larger project. In addition to listening for Composer events, plugins also have the ability to modify composer’s installation behavior.

Put another way, you configure scripts in your main composer.json file, you (or third parties) configure plugins in composer.json files that live in the vendor/ folder.

While understanding both systems is important for a well rounded Composer developer, today we’re going to focus on the plugin system.

Plugin Example

Rather than try to describe things from scratch, we’ve created a simple Composer pluginthat should demonstrate the plugin lifecycle, and help you understand what Magento 2 is doing with Composer plugins.

If you take a look at the plugin class

#File: src/Plugin.php
//...
public static function getSubscribedEvents()
{
    return array(
        'post-install-cmd' => 'installOrUpdate',
        'post-update-cmd' => 'installOrUpdate',            
    );
}    
//...    
public function installOrUpdate($event)
{
    file_put_contents('/tmp/composer.log', __METHOD__ . "\n",FILE_APPEND);
    file_put_contents('/tmp/composer.log', get_class($event) . "\n",FILE_APPEND);            
}

you can see that this plugin listens for the post-install-cmd and post-update-cmdevents. You tell a plugin which events it should listen to by defining agetSubscribedEvent method that returns an array in the above format. Keys are the event, and values are the method, (on the plugin class), that Composer calls as an observer.

In our case, both the post install and post update events call the installOrUpdate method, and this method logs some simple information to the /tmp/composer.log file in our temp directory.

The plugin class also has an activate method.

#File: src/Plugin.php
public function activate(Composer $composer, IOInterface $io)
{
    file_put_contents('/tmp/composer.log', __METHOD__ . "\n",FILE_APPEND);
}

Composer calls the activate method when it detects the plugin every time Composer runs. The activate method is where you instantiate any other objects your plugin will need. In our case, we’ve added a line to log when the method is called.

All in all, a mostly useless plugin, but one that’s useful to diagnose how plugins work.

Adding a Plugin to Your Project

Adding a plugin to your project is the same as adding any other Composer package to your project. Create a new folder

$ mkdir test-plugin
$ cd test-plugin

and then create the following composer.json file in that folder.

//File: composer.json
{
    "repositories":[
        {
            "type":"vcs",
            "url":"git@github.com:astorm/composer-plugin-example.git"
        }
    ],
    "require":{
        "pulsestorm/composer-plugin-example":"0.0.1"
    }
}

In the require section we’ve added our plugin (pulsestorm/composer-plugin-example) and the desired version (0.0.1). The plugin’s name comes from the plugin’scomposer.json file file. The version, 0.0.1, comes from the tagged releases.

Since I didn’t create a packagist.org listing for this package, we need the repositoriessection. This tells Composer to use the git (vcs/version control system) repository at the provided URL as a repository.

This Composer file in place, there’s one last thing we’ll want to do before we run composer install. In a separate terminal window, run the following.

$ touch /tmp/composer.log
$ tail -f /tmp/composer.log

This creates our composer.log file, and then tails it. Tailing a file means showing the last few lines of output. When we run tail with the -f option, we’re telling tail to show us the last line of the file whenever the file is changed. This is a decades old technique for monitoring log files in the *nix world.

Composer Plugin Lifecycle

OK! We’re ready to install our simple project. Run

$ composer install

and composer will install the plugin.

$ composer install
Loading composer repositories with package information
Updating dependencies (including require-dev)                       
  - Installing pulsestorm/composer-plugin-example (0.0.1)
    Loading from cache

Writing lock file
Generating autoload files

More interesting to us though is the output in our /tmp/composer.log file.

Pulsestorm\Composer\Example\Plugin::activate
Pulsestorm\Composer\Example\Plugin::installOrUpdate
Composer\Script\Event

Here, we see Composer called the activate method, and then (per our events) called theinstallOrUpdate method. If we were to run update

$ composer update

We’d see the same lines

Pulsestorm\Composer\Example\Plugin::activate
Pulsestorm\Composer\Example\Plugin::installOrUpdate
Composer\Script\Event    

because we’re also listening for the update event.

A Composer plugin developer can, (via the Composer\Script\Event object Composer passed to our handler or the Composer\Composer object Composer passes to the activemethod), examine and change Composer’s state at run time, implementing all sorts of extra functionality whenever a Composer project updates.

Covering that functionality in full is beyond the scope of this article, but with Composer being open source, there’s nothing stopping you from diving right in.

What Makes a Package a Plugin?

As we mentioned earlier, a Composer plugin is just a standard Composer package. However, it’s a standard Composer package with a special composer.json file. Let’s take a look at our plugin’s composer.json file.

//File: composer.json
{
    //...
    "type": "composer-plugin",
    //...        
    "require": {
        "composer-plugin-api": "^1.0"
    },
    "autoload":{
        "psr-4":{
            "Pulsestorm\\Composer\\Example\\":"src/"
        }
    },    
    "extra":{     
        "class":"Pulsestorm\\Composer\\Example\\Plugin"
    }
    //...
}

The first configuration a plugin package needs is the following

//File: composer.json

"type": "composer-plugin"

This tells Composer that this is a plugin package.

The second configuration a plugin package needs is

//File: composer.json

"require": {
    "composer-plugin-api": "^1.0"
},    

This looks like a standard Composer require — but it’s not. When Composer encounters a package named composer-plugin-api, this indicated which Plugin API version your plugin targets.

Finally, in the extra section, the following configuration

//File: composer.json

"extra":{     
    "class":"Pulsestorm\\Composer\\Example\\Plugin"
}

points to our plugin class (Pulsestorm\Composer\Example\Plugin). Since Composer will need to instantiate this class, that means you’ll need something in your autoload section that ensures PHP will load the class definition file. In our case, we used a standardPSR-4 autoloader

//File: composer.json

"autoload":{
    "psr-4":{
        "Pulsestorm\\Composer\\Example\\":"src/"
    }
}

That’s all you’ll need for a Composer plugin!

Finding Magento 2’s Composer Plugins

Now that we have a better understanding of Composer plugins, we can come back to our Magento problem. As a reminder, we’re trying to figure out how the stock create-project files

$ ls project-community-edition
README.md    composer.json    update 

become a full fledged, many files outside of vendor, Magento installation

$ ls project-community-edition/
CHANGELOG.md                        dev
CONTRIBUTING.md                     index.php
CONTRIBUTOR_LICENSE_AGREEMENT.html  lib
COPYING.txt                         nginx.conf.sample
Gruntfile.js                        package.json
LICENSE.txt                         php.ini.sample
LICENSE_AFL.txt                     phpserver
README.md                           pub
app                                 setup
bin                                 update
composer.json                       var
composer.lock                       vendor

If it’s not obvious by now, these additional files are placed here by a plugin in one of the Composer packages that make up Magento 2.

Unfortunately, Composer doesn’t provide an easy way to check your project for any installed plugins. You’ll need to use some good old fashioned unix command line searching to figure out which Magento packages have plugins.

In plain english, we’ll want to

  1. Create a list of all our project’s composer.json files
  2. Search those files for the all important "type": "composer-plugin", text

In unix english, that’s

$ find vendor/ -name composer.json | xargs grep 'composer-plugin'
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v1/composer.json:    "type": "composer-plugin",
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v1/composer.json:        "composer-plugin-api": "1.0.0"
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v2/composer.json:    "type": "composer-plugin",
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v2/composer.json:        "composer-plugin-api": "1.0.0"
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v3/composer.json:    "type": "composer-plugin",
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v3/composer.json:        "composer-plugin-api": "1.0.0"
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v4/composer.json:    "type": "composer-plugin",
vendor//composer/composer/tests/Composer/Test/Plugin/Fixtures/plugin-v4/composer.json:        "composer-plugin-api": "1.0.0"
vendor//magento/magento-composer-installer/composer.json:    "type":"composer-plugin",
vendor//magento/magento-composer-installer/composer.json:        "composer-plugin-api": "^1.0"

We can safely ignore the results in composer/composer/tests — these are tests in the main Composer package. The result we are interested in is

vendor//magento/magento-composer-installer/composer.json

It looks like the magento/magento-composer-installer package is actually a Composer plugin. If we take a look at the contents of this composer.json file

#File: vendor//magento/magento-composer-installer/composer.json 
{
    //...

    "type":"composer-plugin",

    //...

    "extra":{
        //...
        "class":"MagentoHackathon\\Composer\\Magento\\Plugin"
    }
}

We see the composer-plugin type-tag our command line searching found, as well as the required extra configuration that configures theMagentoHackathon\Composer\Magento\Plugin class as a plugin.

Without getting into the specific technical details, this is the plugin that installs those extra files at the root level, above the vendor folder. In short, theMagentoHackathon\Composer\Magento\Plugin will

  1. Listen for composer install and composer update events
  2. Look at the extra->map section(s) for any composer.json file in the just installed or updated composer vendor packages
  3. Use that information to copy file from the installed package, to the root level project folder

If that didn’t make sense, let’s walk through it. First, let’s find any composer.json files with a "map" section.

$ find vendor/ -name composer.json | xargs ack '"map"'
vendor/magento/magento2-base/composer.json
75:        "map": [

vendor/magento/magento2-base/dev/tests/api-functional/_files/Magento/TestModuleIntegrationFromConfig/composer.json
12:    "map": [

vendor/magento/magento2-base/dev/tests/api-functional/_files/Magento/TestModuleJoinDirectives/composer.json
12:    "map": [

Again, we can safely ignore the files in the tests folder — this leaves us (at the time of this writing) with a single result in the magento/magento2-base package. If we look at a snippet of this file

//File: vendor/magento/magento2-base/composer.json
{
    "name": "magento/magento2-base",
    //...
    "extra": {
        //...
        "map": [
            [
                "lib/internal/Cm",
                "lib/internal/Cm"
            ],
            [
                "lib/internal/LinLibertineFont",
                "lib/internal/LinLibertineFont"
            ],
            [
                "lib/internal/Credis",
                "lib/internal/Credis"
            ],
            //...
            [
                "LICENSE_AFL.txt",
                "LICENSE_AFL.txt"
            ],
            [
                "vendor/.htaccess",
                "vendor/.htaccess"
            ]
        ]
    }
}

When the MagentoHackathon\Composer\Magento\Plugin finds the above map section, it will start running PHP code that’s roughly equivalent to

cp -r vendor/magento/magento2-base/lib/internal/Cm lib/internal/Cm
cp -r vendor/magento/magento2-base/lib/internal/LinLibertineFont lib/internal/LinLibertineFont

cp -r vendor/magento/magento2-base/lib/internal/Credis lib/internal/Credis

//...
cp vendor/magento/magento2-base/LICENSE_AFL.txt LICENSE_AFL.txt

cp vendor/magento/magento2-base/vendor/.htaccess vendor/.htaccess"

This is how the non-vendor files Magento needs to operate get from Magento core vendorpackages into the root folder of your project.

History of magento/magento-composer-installer

Before we wrap up, it’s worth noting that the magento/magento-composer-installerComposer plugin is a fork of the original Magento 1 Composer installer plugin built at a Magento hackathon, promoted by Firegento, and maintained by Daniel Fahlke. The original goals of this plugin were to build a system that allowed developers to use Composer to fetch Magento 1 plugins into vendor, and then install them into a Magento 1 system via a number of different strategies. This was necessary since Magento 1 never officially adopted Composer.

The Magento core team has repurposed the project as an automatic installer which, on one hand, shows the power and usefulness of open source. On the other hand, if you’re not familiar with the project history and you start exploring the plugin’s implementation in

vendor/magento/magento-composer-installer//src/MagentoHackathon/Composer/Magento/Plugin.php

you may be left scratching your head.

However, if you keep the project’s original goals in mind, the source should make a little more sense.

Consequences

Between this article and last week’s, you should have a pretty good understanding of where all Magento’s files come from in a Composer “meta-package” installation. Understanding this is critical for Magento 2 developers and consultants looking to use and develop day-to-day on Magento’s systems. For example, it may be temping to drop some functions into

app/functions.php

as a shortcut way to get your code into a Magento system, but once you understand that any future composer update will wipe these changes away, the true cost of such short cuts become apparent. Don’t edit the core is as true as it ever was, but with Magento 2 it’s not always clear what is, and what isn’t, a core Magento file.

Next time we’ll be diving into Magento 2’s component system — the system that makes Composer distribution possible.

Magento 2: Composer, Marketplace, and Satis

This entry is part 1 of 3 in the series Magento 2 and Composer. Later posts include Magento 2: Composer Plugins, and Magento 2: Composer and Components.

As Magento 2 approaches its first half-birthday, one thing is clear: Magento 2 is leaning heavily on PHP Composer for its developer workflow, and for the merchant facing Marketplace.

If you’re a developer working with Magento, you may be familiar withrepo.magento.com. This is Magento 2’s composer repository, and up until Magento Imagine 2016, its main purpose was to provide modern PHP developers with a way to install Magento 2 using PHP composer.

At Imagine 2016 Magento unveiled their Magento Connect replacement, Magento Marketplace. Behind the scenes, Marketplace is running on PHP Composer, andrepo.magento.com is the source repository for purchased extensions.

In this article, we’re going to show you how you can mirror your Magento specific composer packages on a local server. While not necessary, this is a useful precaution to take if you want to avoid any unscheduled maintenance bringing down your deployment and development pipelines. Along the way, we’ll also discuss Marketplace’s new composer based architecture, and end up touching on many lesser known composer features.

Understanding Composer

If you’re interested in an in-depth look at how composer works, my Laravel, Composer, and the State of Autoloading article is still a great starting point. If you don’t have time for that right now, here’s the basics.

Default Composer Behavior

As a developer, if you want to install a PHP package, you say

Hey, composer, get me the source files for the foo/bar package

i.e.

$ composer require foo/bar

When you do this, behind the scenes, composer says

Hey, Packagist, where can I find the foo/bar package?

In turn, Packagist answers

The package you want is at [THIS Git/SVN/Mercurial URL] and I found an archive at[THIS URL]

Packagist is a composer repository. One thing that catches a few composer newcomers by surprise is packagist, and other composer repositories, don’t actually host any packages. They just point to the location of a package on another server.

Other Repositories

Packagist support is baked into composer. However, it’s possible to point composer atother Composer repositories with a composer.json configuration that looks something like this

//File: composer.json    
{
    //...
    "repositories": [
        {
            "type": "composer",
            "url": "https://repo.magento.com/"
        }
    ],
    //...        
}

So, in reality, this

Hey, Packagist, where can I find the foo/bar package

Has a preamble of

“Hmm”, composer said to itself “Do I have any custom repositories configured? If so, I should ask them first for the location of the foo/bar package”

Magento’s Composer Repository

Magento Inc. hosts a composer repository at the URL https://repo.magento.com/. Magento developers were introduced to this repository via the composer meta-package installer. This installation method fetches and installs Magento’s source code via Composer. Notice the --repository-url flag in the command

$ composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition <installation directory name>

This flag ensures the magento/project-community-edition package is downloaded from repo.magento.com and not packagist.

The meta-package method downloads and installs Magento components (modules, themes, language packs, and libraries) into the vendor/magento/ folder. Thanks to Magento’sregistration.php file and PSR-4 support, there’s no longer a strictly defined place where Magento modules, themes, language packs and libraries need to be located. These components can now be separated out into individual composer packages.

One thing that caught a lot of Magento developers off guard about this meta-package was the requirement for set of composer auth.json credentials, issued by Magento’s main website. It didn’t quite make sense that Magento would create this authenticated method for installing Magento CE when the source code was already publicly available via GitHub.

The key to understanding Magento’s need for credentials is Magento Marketplace.

Magento Marketplace

Magento Marketplace is Magento 2’s replacement for the old Magento Connect. Magento Connect only hosted free Magento modules — commercial module listings pointed off towards independent software vendor’s websites where you could purchase the extension or service directly from the independent vendor.

Magento Marketplace changes that. Marketplace has free extensions available, but it also provides a one-stop shop for purchasing your Magento extensions. Once purchased, you can download the extension package from the My Account section of Magento’s website.

In addition to this download, the extension will also be available via the Magento 2 Component Manager in Magento’s backend, at System -> Web Setup Wizard. The Component Manager is a GUI for installing composer packages from repo.magento.com. It turns out that repo.magento.com is a session-ed packagist repository.

When you log in to repo.magento.com using the aforementioned HTTP Auth credentials, Magento’s composer repository returns a custom list of packages for you to install. This will include

  1. The base Magento 2 CE packages
  2. Any Magento 2 EE versions your account has access to
  3. Any package you’ve purchased

This is what enables you to fetch purchased Magento Marketplace packages via Component Manager. You can also simply add new packages to your composer.json file and then update your system via the command line (i.e. composer update)

Mirroring repo.magento.com

While Magento’s adoption of composer is welcomed, it does add an additional wrinkle to deploying Magento 2 projects. The repo.magento.com repository is a new, single point of failure that, when down, could block a composer based deployment from being updated, or prevent your development team from getting started on a new project. Additionally, unlike the other (still not great) single points of failure in a composer project (packagist.org, github.com, etc.), repo.magento.com isn’t yet a battle tested system.

Because of this, it makes sense to create a local mirror of repo.magento.com. In addition to protecting you from unplanned maintenance/downtime at repo.magento.com, hosting your Magento packages on your local network (or even local development machine) should speed up Magento deployment tremendously.

Mirroring with Satis

The composer project has a second, sibling project called satis. Satis was created to allow developers to create their own local mirrors of packagist.org content. It turns out a stock composer repository requires zero dynamic processing — a repository is just a collection of static json files, and (optionally) mirrored archive packages.

To use satis, you’ll need to clone the GitHub repository to your local development machine with one of the following commands

git clone https://github.com/composer/satis
git clone git@github.com:composer/satis.git

Once cloned, you can run the satis command with php bin/satis

$ php bin/satis 
Satis version 1.0.0-dev

Usage:
 command [options] [arguments]

Options:
 --help (-h)           Display this help message
 --quiet (-q)          Do not output any message
 --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
 --version (-V)        Display this application version
 --ansi                Force ANSI output
 --no-ansi             Disable ANSI output
 --no-interaction (-n) Do not ask any interactive question

Available commands:
 add     Add repository URL to satis JSON file
 build   Builds a composer repository out of a json file
 help    Displays help for a command
 init    Initialize Satis configuration file
 list    Lists commands
 purge   Purge packages

Satis needs two things to build a repository mirror.

  1. A JSON configuration file
  2. An output folder

Satis will output packages.json files and (optionally) package archives to the output folder you specify. You’ll be able to upload these files to any simple web server, and have a working packagist repository.

The JSON configuration file is where you tell satis which repositories you’d like to mirror, which packages in the repositories, as well as any other satis configuration needed.

The syntax for building a satis mirror is

$ php bin/satis build config.json build-folder

The file we’ve named config.json above is often named satis.json by convention.

Satis Configuration

Next up, we’re going to create a satis configuration file and review its options. While the composer manual has a section on satis, the file format isn’t as well documented as it could be. The following is one possible configuration, please get in touch or comment below if you see something egregiously wrong here — I’m learning with the rest of you here!

#File: satis.json
{
    "name": "Pulse Storm mirror of repo.magento.com",
    "homepage": "http://composer.pulsestorm.dev",
    "repositories": [
        { 
            "type": "composer", 
            "url": "https://repo.magento.com" 
        }
    ],
    "require-dependencies": true,
    "require-dev-dependencies": true, 
    "require-all": true,   
    "archive": {
        "directory": "dist",
        "format": "zip",
        "prefix-url": "http://composer.pulsestorm.dev",
        "skip-dev": false
    }    
}

The name property should be a simple description of your repository. It’s used in a static HTML file generated for the repository, so don’t use a name that would embarrass your mother or your supervisor.

The homepage property is the URL you’re planning on hosting your satis repository at. This can be a URL on the Internet, or a URL local to your network/dev machine. This URL will be used in a static HTML file generated for the repository, so make sure its accurate.

The repositories property is the first vital configuration field. This should be an array of the composer repositories you’d like to mirror with satis. In our case, this is the composerrepository at repo.magento.com. Other types of repositories you might see are VCS orpear.

The require-dependencies and require-dev-dependencies flags make sure composer will require any dependencies a specific package may have. Not strictly necessary in our case (see require-all below) but it never hurts to be explicit.

The require-all property tells satis we want to grab every package on the repository. Since our goal is to create a complete local mirror, we want this here. If we were interested in creating a mirror of select packages, we’d use a require property and list out the packages.

The final top level archive property tells satis that, in addition to creating apackages.json file for us that points to packages, we’d like satis to grab and/or build a project archive for us as well.

In the archive object configuration, the directory property tells satis where the archived files should be copied to (i.e. build-folder/dist), the format property tells satis which archive format we should use, the prefix-url should be, again, the URL for your repository, and setting the skip-dev option to false ensures we get every file in a package.

The prefix-url key is important here, as it’s the URL satis will use in the generatedpackages.json file to point to our local mirrored archive.

If you’re building for repo.magento.com, you’ll want to keep the archive format set tozip. There are some bugs and inconsistent behavior with Magento 2’s repository and composer that were only just recently fixed for zip archives, and still exist for tar archives.

Building the Mirror

Once you’ve created your satis.json, you can create the mirror with the following

$ php bin/satis build satis.json public -vvv

If your repo.magento.com HTTP credentials aren’t stored in ~/.composer/auth.json, satis will prompt you for them. The -vvv flag is optional, but will ensure satis is verbose in its output. This can help point to problems in your configuration, or with your network connection.

When the command finishes running, you’ll want to upload the files in public to the web root of whatever URL you configured in satis.json (composer.pulsestorm.dev in our case).

When satis is done running, you can take a look at the repository files using the unix findcommand

$ find public -type f
public/dist/auctane-api-2.0.6.zip
public/dist/belvg-module-facebookfree-1.0.1.zip
public/dist/magento-composer-1.0.2.zip
public/dist/magento-data-migration-tool-2.0.0.zip
//...
public/dist/magento-updater-10.0.0.zip
public/include/all$e89e3a381f6a71df66912bf26c12b89db1200cd8.json
public/index.html
public/packages.json

You should see archives of all the community edition files, any EE files your account license grants you access to, and any extension files you’ve purchased. Upload the entire contents of the public folder to your web server, and you’ll have a local composer repository up and running.

With our mirror created, we’re ready to install Magento 2.

Installing Magento 2 with our Mirror

Normally, when installing Magento via composer, you use the following command

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition <installation directory name>

We’re going to change this to

composer create-project --no-install --repository-url=http://composer.pulsestorm.dev/ magento/project-community-edition <installation directory name>

i.e. — the same command, but with our repository, and the no-install flag. To understand why we’re doing this, we need to talk about what the create-projectcommand does.

Composer: Create Project

The create-project command is a shortcut command. When you use create-project, composer will

  1. Fetch the latest version of the specified package (magento/project-community-edition) and extract it to the <installation directory name>folder
  2. Run composer install from the <installation directory name> folder, installing all dependencies

If you’ve done any work in the modern PHP world, you’re probably familiar with this command from the various framework’s one-line installers.

This presents a problem for our mirror. If you unzip any of the magento/project-community-edition packages archived in our mirror.

$ ls public/dist/magento-project-community-edition-2.*
public/dist/magento-project-community-edition-2.0.0.zip
public/dist/magento-project-community-edition-2.0.1.zip
public/dist/magento-project-community-edition-2.0.2.zip
public/dist/magento-project-community-edition-2.0.3.zip
public/dist/magento-project-community-edition-2.0.4.zip

You’ll see that its composer.json file points to repo.magento.com. The --repository-url option only applies to the package that create-project grabs. Otherwise, composer will use whatever it finds in the project’s base composer.json file.

This is easy enough to work around — all we need to do is use the no-install flag — this way create-project will only download and extract the project, it won’t run composer install. This will give us a chance to edit the composer.json file before running install.

Installing Magento 2

OK! We’re ready. Step 1, lets clear our composer cache

$ composer clear-cache

This is not strictly necessary, but useful when you’re first setting up a mirror, and may have an invalid package reference stashed in cache somewhere.

Also, if your mirror’s not located on an https server, you may need to set composer’s global secure-http flag to false.

$ composer global config secure-http false

Recent versions of composer will refuse to run over non-encrypted HTTP.

Next, let’s run our create project command

$ composer create-project -vvv --no-install --repository-url=http://composer.pulsestorm.dev/ magento/project-community-edition
//...
Downloading http://composer.pulsestorm.dev/packages.json
Writing /Users/username/.composer/cache/repo/http---composer.pulsestorm.dev/packages.json into cache
Reading /Users/username/.composer/cache/repo/http---composer.pulsestorm.dev/include-all$e89e3a381f6a71df66912bf26c12b89db1200cd8.json from cache
Installing magento/project-community-edition (2.0.4)
  - Installing magento/project-community-edition (2.0.4)
Downloading http://composer.pulsestorm.dev/dist/magento-project-community-edition-2.0.4.zip
    Downloading: 100%         
Writing /Users/username/.composer/cache/files/magento/project-community-edition/648f32cf7a59f92769940d85435cf16d7385fa5f.zip into cache from /path/to/magento/project-community-edition//de123c8bffdb844a4093d235a37b66d3.zip
    Extracting archive
Executing command (CWD): unzip '/path/to/magento/project-community-edition//de123c8bffdb844a4093d235a37b66d3.zip' -d '/path/to/magento/vendor/composer/847ae1bb' && chmod -R u+w '/path/to/magento/vendor/composer/847ae1bb'

//...

Again, the -vvvs are optional, but viewing composer’s verbose output can help us ensure that no package was/is downloaded from repo.magento.com. When composer’s done running, change into the project-community-edition folder, and take a look atcomposer.json

$ cd project-community-edition
$ ls
README.md    composer.json    update
$ cat composer.json
{
    //...
    "repositories": [
        {
            "type": "composer",
            "url": "https://repo.magento.com/"
        }
    ],
    //...        
}

As mentioned, there’s repo.magento.com, getting in the way of our local mirror. Let’s edit that file to point at our repository.

#File: composer.json
//...
    "repositories": [
        {
            "type": "composer",
            "url": "http://composer.pulsestorm.dev/"
        }
    ],    
//...

With the above in place, run

$ composer install -vvv

and composer will grab all Magento’s packages from your local mirror — no access torepo.magento.com required.

Wrap Up

All of this, of course, is only a start to keeping and maintaining a local composer mirror. EE users will want to check their enterprise license agreement to make sure doing this falls within acceptable use of the EE source code, and regardless of which version of Magento you’re running you’ll want to make sure your mirror isn’t located anywhere online, as you may become an inadvertent distribution point for commercial extensions you don’t have the right to distribute.

You’ll also want to figure out a way to get satis running on a regular basis — otherwise you may miss important updates to the Magento repository. The very bold may want to expand their mirroring to packagist itself, but that’s a larger problem filled with all sorts of blind alleys and large hard drives.