Drupal Techniques: Resources for Migrations

Written on February 25, 2020

Category: Drupal Techniques

Author: David Rodríguez, @davidjguru

Picture from Unsplash, by @inmasantiago
Picture from Unsplash, user Inma Lesielle @inmasantiago

I am working on notes for a draft that will be a book about migration processes made with Drupal and its Migrate API. It is expected to be released in June 2020 and the work of collecting, experimenting and articulating the content is being quite extensive. As there are still some months left for the launch, in order not to lose the mental sanity and to be able to give partial sense to these tasks, I have thought to publish here some small posts derived from the working notes. This way I will be able to give something useful to the complementary notes and if the COVID-19 attacks me before seeing the book come out, at least I will have shared something before (I guess).

Well, what do I want to talk about in this post? I would like to make a list of Drupal modules related to migration processes, available as contrib modules and that can be used to provide functionality to a migration. This article will be only a lightweight set of basic resources (I swear).

Table of Contents

1- Introduction

2- Basic Resources - Core Modules

3- Other Basic Resources - Contrib Modules

4- Extra Resources - Contrib Modules for Plugins

5- Migration Runners - Contrib Modules Drush-Related

6- Authors you should know

7- :wq!

This article is part of a series of posts about Drupal Migrations

1- Thinking about Drupal Migrations (I): Resources

2- Thinking about Drupal Migrations (II): Examples

1- Introduction

It’s not very easy to talk about migrations in general and, of course, it is not easy in the context of Drupal either. To perform migrations it is necessary to have a good knowledge of the technology, data models (in origin and in destination), experience in ETL processes and a certain know-how about how to implement Drupal Plugins (In migrations there is an extensive use of Drupal-Way Plugins).

In any case, since the topic is extensive and my time is now short, I thought of this article as a summary catalogue (for quick consumption) of tools and basic resources for working with migrations.

2- Basic Resources - Core Modules

3- Other Basic Resources - Contrib Modules

  • Migrate Plus: migrate_plus. Migrate Plus is an essential contrib module wich extends the features and capabilities of the Migrate core module with a lot of plugins and extensions.

  • Migrate Tools: migrate_tools. Another essential resource: provides a lot of Drush commands for running and managing Migrations.

  • Migrate Status: migrate_status. This little contrib module allows get a feedback about a migration process. Do you need to know if a migration is running? this module gives you a service that you can call in order to check the migration.

  • Migrate Files: migrate_files. It’s such an interesting set of process plugins that you will want to move files and images with it.

  • Migrate Commerce: commerce_migrate. General-purpose framework that extends to the main Migrate module from Drupal Core, for moving data in a Drupal Commerce scenario.

4- Extra Resources - Contrib Modules for Plugins

In the Drupal migration processes, we’ll use diverse resources in order to processing the ETL migration plan. One of these basic resources (as I mentioned in the introduction) are the Drupal Plugins, of which you need to have good knowledge and some practice. In a migration scenario, plugins help us processing information from the E:Source (Source Plugins), making T:Processing(ProcessPlugins) in order to save data at L:Destination (Destination Plugins). The assembly of these three parts (usually) results in a correct migration process.

Many modules of the core already bring their own Plugins to facilitate migration processes (as the user module). So let’s review some migration plugins packaged in contributed modules.

Source Plugins Migrate Source Plugin

  • Migrate Source CSV: migrate_source_csv. Contrib Module for migrating data to Drupal 8 from a classical and simple CSV file.

  • Migrate Source SQL: custom_sql_migrate_source_plugin. As a peculiarity of the Plugins used for databases, this module allows to integrate in the .yml file describing a migration, directly SQL queries that will be executed against the source database.

  • Migrate Source YAML: migrate_source_yaml. It’s just a simple tool for migrating content from YAML files.

Processing Plugins Migrate Process Plugins

  • Migrate Process Geofield: geofield. The contrib Geofield module comes with a custom process plugin for migrations. See Process Plugin in Geofield.

  • Migrate Process XML: migrate_process_xml. Provides process plugins for xpath and xvalue.

  • Migrate HTML to Paragraphs: migrate_html_to_paragraphs. Helps to transform HTML from a migration Source in a Paragraph item (managed by a Destination Plugin).

Destination Plugins: Migrate Destination Plugins & Examples

What kind of Drupal entities will be created in the migrating process? content entities? configuration entities? Take a look.

  • Migrate Destination CSV: d8migrate. It’s a light custom module created by @jonathanfranks.

  • Migrate Destination Config: Class Config.php. Offers a plugin for config migration.

  • Migrate Destination Block: Class EntityBlock.php. Just like and example about the resources that every element can offers in a migration scene, in case of moving Block Entities (are Config Entities) see the PHP classes included in its own module for migrating (Source, Process and Destination).

Drupal 8 Migrate Entity Block

6- Authors you should know

  • Mauricio Dinarte: Mauricio is a developer, consultant, trainer and owner of his own business https://agaric.coop, wrote what is probably the mandatory reading guide for all people who want to learn how to migrate on Drupal: 31 Days of Drupal Migrations, a set of 31 articles published in https://understanddrupal.com/migrations with the most important aspects…examples, exercises, descriptions…to understand the whole internal world of migrations inside Drupal.

An essential training material. In addition, his company’s website, under the tag “migrate” also hosts many very good articles about migration topics: https://agaric.coop/tags/migrate.

Some examples from Mauricio Dinarte:

  1. Introduction to paragraphs migrations in Drupal:
  2. Using migration groups to share configuration among Drupal migrations:
  3. What is the difference between migration tags and migration groups in Drupal?

His profile in Drupal.org: https://www.drupal.org/u/dinarcon.

  • Tess Flynn: I heard about Tess Flynn reading articles by Mauricio Dinarte. That’s how I met this expert developer, speaker and communicator of the Drupal community. On her website https://deninet.com I found content of a different nature, but above all, a series of very interesting articles about migrations under the tag “drupal-migration”: https://deninet.com/tag/drupal-migration.

    Along the way I also discovered that it has several contrib modules related to Migrations and Processing Plugins.

Some examples from Tess Flynn:

  1. Migrate Process URL: Provides Process Plugin to migrate link fields.
  2. Migrate Process Vardump: Helping to debugging migrations.
  3. Many Process Plugins:

Her profile in Drupal.org: https://www.drupal.org/u/socketwench.

Some examples from Danny Sipos:

  1. Your first Drupal 8 Migration:
  2. Dynamic migrations using “templates” in Drupal 8:
  3. Quickly generate the headers for the CSV migrate source plugin using Drush:

His profile in Drupal.org: https://www.drupal.org/u/upchuk.


Written on February 25, 2020