davidjguru.github.io

Drupal Tips: Avoid infinite loop in user login

2023-01-03T00:00:00+00:00


Picture from Unsplash, user Tine Ivanič, @tine999

Last week I had to perform an intervention in a Drupal installation due to an unexpected bug. It was a problem that I hadn’t seen in Drupal for a long time… But the key really is that I was the one who created the issue so I think I’m ready to explain how it happened (my own code generated it) and how to fix it… I am very pleased to join you today to share with you my latest ridiculous mistake: the infinite login redirection in Drupal (and how to solve it)!…

Well, actually and to be fair, the title should be something like “what did I do to cause an infinite loop” or maybe “How I generated an endless loop of redirections in Drupal”, but it didn’t work well as a title (marketing issues) so better that way.

Acknowledgments

This post was composed from my current position as Senior Drupal Developer at FFW Agency, one of the biggest companies oriented to Open Source in the world and specifically focused in Drupal.

Checking the node / entity access

My little problem started from a very simple model. In fact my starting idea was quite simple: to check the visibility of certain types of nodes for some users based on some taxonomy terms, that is, depending on the taxonomy term (group 1, group 2, group 3, group 4) applied to a user (group 2), this user could only see nodes labeled with the same taxonomy term (group 2), and for this, better centralize the activity using well known Drupal hooks: hook_node_access for 8.x or hook_entity_access for 9.x and above. I was looking for something that can be summarized in the following image:


Managing node access by matching taxonomy terms

For those unfamiliar with its use, hook_entity_access works like an updated version of hook_node_access, more generalized for entities and with some restrictions on the parameters of CRUD operations to be processed (create, delete, update, view) respect to hook_node_access. But in fact both share a similar scheme of work:

The hooks will receive some item-to-check related params: entity/node, operation and user account.
The hooks will return an access result based in AccessResultInterface: allowed, forbidden, neutral…

Here is a comparison between these two access hooks:


Entity Access and Node Access comparison

So I started using the version hook_node_access():

/**
 * Implements hook_node_access().
 */
function my_custom_module_access_node_access(NodeInterface $node, $op, AccountInterface $account) {

And by the way, I’m only concerned about specific types using the selected vocabulary as field:

$specific_content_types = [
  'content_type_one',
  'content_type_two',
];

Then I’m getting the whole entity for the current user:

$current_user_entity = \Drupal::entityTypeManager()->getStorage('user')->load($account->id());

I would like to think about permissions, checking all scenarios where we will be neutral in access:

If is another content type then is not our business.
We’re only interested in management of views of nodes.
Node has not associated value in taxonomy term field.
Being a registered user the account has not value in taxonomy term field.
Current User has permissions to bypass access restrictions.

I could generate some control like this:

#1  if ((!in_array($node->bundle(), $specific_content_types_content_types)) || 
#2     ($op != 'view') || 
#3     ($node->get('field_groups')->isEmpty()) ||
#4     (($account->isAuthenticated()) && ($current_user_entity->get('field_vocabulary')->isEmpty())) ||
#5     ($account->hasPermission('bypass content access'))) {

        return AccessResult::neutral();
    }

But what about other scenarios? I’m interested in forbidden and allowed cases. For instance, when the node could have loaded taxonomy terms in the key field (or maybe not) but user is not authenticated:

if ($account->isAnonymous()) {
  return AccessResult::forbidden();

And finally, the main key scenario and its derivatives: user and node have loaded values in the taxonomy term field and we have to look for a possible match between them, looping all the possible array of values. Here the user is authenticated and as a first step, we have to get the existing values in taxonomy term field:

$terms_node = $node->get('field_vocabulary')->getValue();
$terms_user = $current_user_entity->get('field_vocabulary')->getValue();

And as sub-case, we have to check if user is just registered but there is no value associated with the taxonomy field. In a hurry and without thinking too much, I chose this way of extracting the value of the field, being an entity reference field:

if ($account->isAuthenticated() && ($current_user_entity->get('field_vocabulary')->getValue()[0]['target_id'] == 0)) {
  return AccessResult::neutral();
}

You know, if any of the taxonomy terms from the node items is equal to any of the user account then we’ll have granted access to the node:

// Third: existing terms in node and user we'll match between terms.
foreach ($terms_node as $node_target) {
  foreach ($terms_user as $user_target) {
    if ($node_target['target_id'] == $user_target['target_id']) {
      return AccessResult::allowed();
    }
  }
}

And finally, while you’re a non-allowed user for the node we will redirect you to the home node:

// 101 is the ID of the node marked as home page.  
$url = Url::fromRoute('entity.node.canonical', ['node' => 101]);
$redirect = new RedirectResponse($url->toString());
$redirect->send();

These ideas, these implementations that seem to have a certain logic… did not take long to cause serious problems after the deployment of the website and testing… after a while, several colleagues started to raise the alarm.

The Issue

As I said in a previous paragraph, these changes caused bugs immediately after deployment in testing environments. The most obvious one was that after the user login, with the user already registered, visiting the nodes resulted in an infinite loop of redirection that prevented access to the website:


Infinite loop of redirection causing blocking access

At first glance, this problem can be recognised with a trained Drupal troubleshooting eye: the user is already logged in, an access is attempted and denied (forbidden) and a redirect is triggered, which will contain the login screen. But the user is already logged in, so it redirects endlessly until the login is closed.

So you have to stop, review and debug the code until you find the bugs… so please enable your xdebug configuration, F5 (in my VSCode installation) and go to run a debugging… in my case I found two main problems:

The way I’m getting the taxonomy terms values.
The way I’m executing redirection.

Let’s check the issues.

1. The way I’m getting the taxonomy terms values

The first issue was how I was getting the taxonomy term values from the different sources in order to look for a match. You know, I’m talking about the values from the Entity Reference Field (Taxonomy Term in the Drupal standard field is Entity Reference). At some point in my code I’m just checking if any value was set in the taxonomy term field:

($current_user_entity->get('field_groups')->isEmpty())

And this expression was avoiding the pass when the field wasn’t filled, passing the action to the next else clause ordering the redirection:

$url = Url::fromRoute('entity.node.canonical', ['node' => $return_page]);
$redirect = new RedirectResponse($url->toString());
$redirect->send();

We said that the field was not set, but this is not really true, due to the value being zero, and not empty. From this point of view, may can be more useful check values in a different way:

($current_user_entity->get('field_groups')->getvalue()[0]['target_id'] == 0)

But while that’s true the response is an array of values -being the taxonomy term field a multivalue field-, if the field was not filled then the response can be an unique value that you can check in a more single way using less expressions and reducing your code:

($current_user_entity->get('field_groups')->target_id == 0)

And remember you can obtain some more taxonomy terms related info if you need by using something like this:

$terms = $node->get('field_vocabulary')->referencedEntities();
foreach ($terms as $term) {
  $name = $term->getName();
  $tid = $term->id();
  $some_field_values = $term->get('specific_field_of_the_vocabulary_term')->getValue();
}

Now we have a more controlled processing of the value for taxonomy terms, checking it and giving permissions on access.

2. The way I’m executing redirection

The second issue (well, is not really an issue but something at risk of becoming technical debt). Now I’m talking about the redirection.Remember that when you want to redirect the user’s navigation, you can use various approaches. Let’s see…

Calling for the 403 system page

system.403:
  path: '/system/403'
  defaults:
    _controller: '\Drupal\system\Controller\Http4xxController:on403'
    _title: 'Access denied'
  requirements:
    _access: 'TRUE'

And so you can use a redirection based in the internal system route:

$url = Url::fromRoute('system.403');
$response = new RedirectResponse($url->toString());

Loading the related node ID for redirection

$url = Url::fromRoute('entity.node.canonical', ['node' => 101]);
$redirect = new RedirectResponse($url->toString());
$redirect->send();

What was my initial approach? I thought I’d go and ask the system what page it had loaded for redirects. Something like:

$site_config = \Drupal::config('system.site');
if (is_null($site_config->get('page')['403'])) {
  // If a 403 error page is not configured then we will redirect to /home.
  $return_page = (int) filter_var($site_config->get('page')['front'], FILTER_SANITIZE_NUMBER_INT);
}
else {
  // If 403 page exists we will redirect to error page.
  $return_page = (int) filter_var($site_config->get('page')['403'], FILTER_SANITIZE_NUMBER_INT);
}

And finally I’ll load the return page for redirection:

$url = Url::fromRoute('entity.node.canonical', ['node' => $return_page]);
$redirect = new RedirectResponse($url->toString());
$redirect->send();

What was the problem here? Well firstly, we have to avoid sending responses directly (this can be problematic due to conflicts with two responses, one from you and another from the Drupal system by itself), Then I have to think about the context: while I’m in a Drupal multisite installation, I’m not pretty sure if every single site has been registered with a specific error page, so maybe a source of errors. I need some more neutral, more… “delegated”… So for me the solution was to use Symfony AccessDeniedHttpException.

  // Fourth: As you're a non-allowed user for this node we'll redirect you.
  throw new AccessDeniedHttpException();
  }
}

This will pass the default value “403” to its parent’ constructor and so, the HttpException will be called and it will generate an autonomous response. This can be a more clean solution ;-)

:wq!

Recommended song: Qué he sacado con quererte AKA Violeta Parra - Guadalupe Plata

Tooling: Python2 error in SASS building frontend

2022-09-16T00:00:00+00:00


Picture from Unsplash, user Markus Winkler, @markuswinkler

Now in my current context I’m working on a multisite Drupal installation with several internal sites (more than a hundred sites) and among other experiences, I’m facing new issues that I had not experienced before. This little post is about that: a very specific error that can be common and how to solve it.

Well, now I’m working in a different stack from what I’m used to: I’m using Docksal, a software virtualization tool for containers and Acquia BLT (or Acquia Build and Launch Tool), a resource for automatization of tasks, deploys and testing. The combination of both allows to build Docker container networks in your local environment and to have pre-configured different tasks for the daily management of Drupal-based websites.

I recommend you to try them, although for containers I prefer DDEV -that’s true- but I think that experiencing a containerization tool (Docksal, DDEV, Docker4Drupal, Gitpod…) and a maintenance and development task management tool (Acquia BLT, Robo / Drobo, Deployer is a MUST for any Drupal developer.

Acknowledgments

This post was composed from my current position as Senior Drupal Developer at FFW Agency, one of the biggest companies oriented to Open Source in the world and specifically focused in Drupal.

The Issue

When I want to build up the frontend of a site, I execute a combined Docksal + BLT instruction, something like using the classical @alias/naming of a site:

 fin blt frontend --site=one.of.my.sites.com

At some point, I’m getting this error message:

error /var/www/docroot/themes/custom/xxx_yyy_base_theme/node_modules/node-sass: Command failed.

And then in the same trace:

gyp verb check python checking for Python executable "python2" in the PATH
gyp verb `which` failed Error: not found: python2

And then the process is stopped and it is not possible to build the frontend of the required site… You can see the complete trace in the next screenshot:


Python2 error while we’re building up the frontend

So what I do is first of all, check the version numbers of each element involved in the Issue, using command line for the first items and then reviewing package.json file and getting some versions, or by using yarn with subcommands just like:

$ yarn list --pattern "node-sass"

And so I get all the values I am interested in, since I will have to google the status of each one, looking for issues, known bugs or deprecated code ;-) :

node -v: v16.16.0
npm -v: 8.11.0
node-sass:  4.14.1
node-gyp: 3.8.0

Fixin’

Ok, let’s try to get a good understanding of what is happening throughout this process since we execute the frontend startup commands. As I said, when I’m running my current configuration in CLI, mixin’ docksal commands + Acquia tooling, I’m trying to run the build up for theming by usings something like this:

$ fin blt frontend --site=one.of.my.sites.com

Essentially I’m launching related scripting, diverse stuff as setup.sh or build.sh. And within these scripts (normally) you’re only running basic yarn specific and custom sub-commands, just like: yarn run build:sass. Ok, and what does it means? traveling to every package.json file, you can see the common scripts block:

"scripts": {
  ...
  "clean": "rm -rf node_modules pattern-lab",
  "build:sass": "node-sass src/_patterns -o src/_patterns --include-path breakpoint-sass/stylesheets",
  ...
},

So finally, when you (or your scripts) launch yarn run build:sass finally you’re using the command:

node-sass src/_patterns -o src/_patterns --include-path ./node_modules/breakpoint-sass/stylesheets

Just in order to transpiling the CSS files in theming from SCSS syntax to processed CSS, marking source, the destiny folder (-o /folder/) and other resources to integrate in the process (--include-path) as usual.

Ok, What’s the problem? Well, the first key here: node-sass. I’ve discovered the dependency is marked as deprecated from more than two years ago (nothing interesting from my side, just the magic of the hyperlink and a little of Google)… It causes problems and errors because nodejs compatibility with the current version of node-sass installed. You can see the current status of node-sass here or in tickets from some Drupal distributions like Varbase. This issue has been detected by me at least in a pair of themes of my current multisite structure, think about theme_one and theme_two. So, we’ll need a new resource instead the deprecated node-sass. Ok, but, what’s the solution?

You can use dart-sass instead. The movement requires changes at some levels, just like those (follow the next steps):

Connect to your selected docker container and move to your theme folder:
```
$ fin bash
docker@cli:/var/www$ cd docroot/themes/custom/
```

Remove deprecated dependencies using yarn and delete node-sass from requirements.

docker@cli:/var/www/docroot/themes/custom/theme_one$ yarn remove node-sass
docker@cli:/var/www/docroot/themes/custom/theme_two$ yarn remove node-sass

Install new dependencies.

docker@cli:/var/www/docroot/themes/custom/theme_one$ yarn add --dev sass sass-loader
docker@cli:/var/www/docroot/themes/custom/theme_two$ yarn add --dev sass sass-loader

Remove node_modules folder in every theme directory and regenerate it using the new dependencies (dart-sass as just “sass” resource).

docker@cli:/var/www/docroot/themes/custom/theme_one$ rm -rf node_modules
docker@cli:/var/www/docroot/themes/custom/theme_one$ yarn install

Review and execute small adjustments and changes in commands, sub-commands and flags or params used from internal scripts (e.g build.sh or setup.sh in some themes but executing scripting orders compiled in package.json files from every theme). So we have to ensure that after the executed changes, the commands are modified also in order to execute the same steps when building frontend.

For instance, while in node-sass you had flags like -o in order to set the output folder, now you don’t need the item in dart-sass. Or param --include-path is changed to --load-pathin the new library.

So in your scripting block you had:

"build:sass": "node-sass src/_patterns -o src/_patterns --include-path ./node_modules/breakpoint-sass/stylesheets",
"build:pl-sass": "node-sass --importer node_modules/node-sass-glob-importer/dist/cli.js src/sass/ -o src/css --include-path ./node_modules/breakpoint-sass/stylesheets",

And now we have:

"build:sass": "sass src/_patterns:src/_patterns --load-path ./node_modules/breakpoint-sass/stylesheets",
"build:pl-sass": "sass --importer node_modules/node-sass-glob-importer/dist/cli.js src/sass/  src/css --load-path ./node_modules/breakpoint-sass/stylesheets",

Pay attention in your code cause you can get some warnings after switching to dart-sass. For instance, after switching from node-sass to dart-sass, there are some deprecated expressions that may cause you receive some warning messages. See:


Deprecations warnings from dart-sass

Before:

span {
  box-sizing: border-box;
  max-width: 40px;
  padding: $space / 2;

After:

span {
  box-sizing: border-box;
  max-width: 40px;
  padding: math.div($space, 2);

Once you have these adjustments made to your .scss files, you are ready to have a completely clean and updated frontend startup. That’s it.

Warning

It was only repaired in some themes of a large set of themes and subthemes in a multisite installation… I mean, if it is your scenario, may be necessary to make similar changes to other existing themes which, at this stage, you have not yet tested. Think that if your internal multisite structure looks like the one I’m using now, then each theme / subtheme has its own build up process, with its own resources and dependencies (package.json, yarn.lock, assets, etc.).

:wq!

Recommended song: In Spain we call it Soledad - Rigoberta Bandini

Drupal Tips: Changing session lifetime for users

2022-03-30T00:00:00+00:00


Picture from Unsplash, user Malvestida, @malvestida

The Session time management is an important aspect in any type of application: it allows to determine under which processes the user login will be performed, how the user session will be managed and how the user logout will take place. In the case of Drupal, as in other technologies, it is essential to know the most important aspects of sessions and also, if the architecture is “decoupled”, we will have to take them into account to connect sessions opened from the frontend but managed in the backend. This only increases its complexity and importance.

That is why we are going to share some concepts and ideas. Unfortunately, as I don’t have all the necessary time and it would be a more extensive subject, I’m going to focus only on Drupal session management as a monolith. The “decoupled” variant is more extensive and complex and involves more intermediate tools, such as token management, SAML, etc. Perhaps for a more advanced article in www.therussianlullaby.com in the future. For now, just a sketch here.

Acknowledgments

This post was composed from my current position as Senior Drupal Developer at Digitalist Sweden, one of the biggest companies oriented to Open Source in Europe and specifically focused in Drupal.

Goals and basic concepts

A Drupal installation uses the built-in PHP session mechanism, which is based essentially on a cookie. The cookie name is SESS and is followed by a long string of characters, and its value is another long string of characters, as you can see in the next caption from a new Drupal local deploy based in DDEV tooling:


Cookie values for sessions in a Drupal installation

PHP can inspect this cookie and makes it available to Drupal’s session implementation, using the database table sessions for storage, with the value of the cookie as the key session identification (sid). Then the session entry is also saving records using the current user ID. In the Drupal bootstrap sequence the session is loaded with the related user object, and is used to manage access to resources and other stuff, using the user’s roles and other specific attributes. Here are the mechanics.

The stored Data in the PHP global array $_SESSION will be transferred and saved to the current session’s database entry in the Drupal backend at the end of a single page request. Such a process can be observed to store important data such as items in a shopping cart and/or other related stuff.

By the way (and as it is very logical): You can work using other cookies that can be saved and retrieved by code. of course. In addition, it is also important to know that cookies with an expiry time of 0 are also often know as “session cookies”, cause these are deleted when you close your browser. This will be part of our goals in this case: to get session cookies.

For this case, we have only a couple of basic goals:

Identify the time session management in backend (Drupal).
Make sure the user is logged out when close tab / window / session.

For which we will first need to understand some basic concepts:

gc_maxlifetime
cookie_lifetime
gc_probability

What do they mean? Let’s see…

GC_MAXLIFETIME

This value is for the server: It is a setting for Session Garbage Collection. Initially If the user’s last visit happened before 200,000s then this session is eligible for garbage collection. Since it is GC, the session value may be discarded and not compulsory. If a GC action happens after the session was made eligible for the GC, it will be deleted.

GARBAGE COLLECTOR (GC values)

The PHP’s Garbage Collector is an internal system that allows PHP to keep the memory clean. You can enable or disable the associated functions using some parameters from code or from internal config files. How it works? Well, we already know that a variable in PHP is stored in a container called “zval” (Zend Value), from C language. This is a data structure implemented in C and it represents any PHP value (number, string, array, etc.). Here we’re storing also the type of variable, its value and some pieces of info:

If the variable is part of the reference set.
If the variable is referenced from another places.

So, periodically, some algorithms are reviewing the variables and resources, deleting the unused registers. For Garbage Collection, we have some params available in php.ini file: php.net/session.configuration.php

In general terms, we’re configuring this in the php.ini file:

session.gc_maxlifetime = 3600
session.gc_probability = 1
session.gc_divisor = 1000

But in the Drupal context, we can change these values from a Drupal installation, inside the file default.services.yml in web/sites/default/ or in core.services.yml in web/core/:

parameters:
  session.storage.options:
    gc_probability: 1
    gc_divisor: 100
    gc_maxlifetime: 200000
    cookie_lifetime: 2000000

As you can see expanded in the original block: from default.services.yml:

session.storage.options:
    # Default ini options for sessions.
    #
    # Some distributions of Linux (most notably Debian) ship their PHP
    # installations with garbage collection (gc) disabled. Since Drupal depends
    # on PHP's garbage collection for clearing sessions, ensure that garbage
    # collection occurs by using the most common settings.
    # @default 1
    gc_probability: 1
    # @default 100
    gc_divisor: 100

The gc_probability and gc_divisor are two interrelated values that defines the probability that the gc (garbage collection) process is started on every session initialization. The probability is calculated doing: gc_probability/gc_divisor (1/100 for us) and it means there is a 1% chance that the GC process starts on each request.

COOKIE_LIFETIME

This value is for the browser: This is the absolute maximum time till which a browser can keep this cookie active. A 0 value here will means immediate or when the browser will be closed.


Cookie values for creation and expiration dates

By default is set with 2,000,000 seconds, equals to almost 24 days. See the previous caption with creation and expiration date, so for a cookie created on March 27th, we can see an expiration date of April 19th, which is just a 23-day life cycle, the marked 2,000,000 seconds. You can think about change this data in service.yml files, but It’s true that these kind of files is out of control for Git so your changes will not be shared with your team, and it won’t be recreated in common environments (stage, preprod, prod, live, etc). You’ll need a more “structural” and persistent solution, so we will need to create a new custom module in next steps.

The two main values related to sessions are coming from a static way in a services.yml file, generated in a Drupal installation from Composer. When I launch the build of a new Drupal site, I’m getting a set of values marked by-default. This file is setting the session lifetime with 200,000 seconds => 55.55 hours => 2.31 days. And the cookie lifetime by default with 2,000,000 seconds => 555.55 hours => 23,15 days.

They are the values setted in the default.services.yml file as you can see for gc_lifetime or cookie_lifetime values in your Drupal installation:

    # Set session lifetime (in seconds), i.e. the grace period for session
    # data. Sessions are deleted by the session garbage collector after one
    # session lifetime has elapsed since the user's last visit. When a session
    # is deleted, authenticated users are logged out, and the contents of the
    # user's session is discarded.
    # @default 200000
    gc_maxlifetime: 200000
    #
    # Set session cookie lifetime (in seconds), i.e. the time from the session
    # is created to the cookie expires, i.e. when the browser is expected to
    # discard the cookie. The value 0 means "until the browser is closed".
    # @default 2000000
    cookie_lifetime: 2000000

As we have already said, these values are in the default configuration of the Drupal installation. Besides this is part of the Container in Drupal, so can only be overwriting from a custom Drupal module using a pair of classes. We need something like:

gc_maxlifetime: 3600 (only one hour of lifetime for the session).
cookie_lifetime: 0 (deleted when user close the browser tab).

Overwriting default values by switching service class

Well, it seems that the default configuration used by the service responsible for managing sessions does not work for me and I need modifications from the container… What can I do?

The bad news is that the above values are statically part of the service container after a Drupal installation, and cannot be modified “dynamically”. And these previous values are in the by-default configuration of the Drupal installation.

The good news is that we can add others values by switching with another class to replace the service provision.This is part of the Container in Drupal, so it can only be changed from a custom Drupal module using a pair of classes. We can do something like add the next values:

gc_maxlifetime: 3600 (only one hour of lifetime for the session, and then make the session available for the garbage collector).
cookie_lifetime: 0 (deleted when user closes the browser tab).

And here we can “decorating” existing service for overriding an existing service, as in Symfony (see “Read More” section with links) or we can provide another Service class and do the switch. To modify existing services, we have to implement a class extending ServiceProviderBase, a new Service class and the alter() method. In short, all we need is only change the class responsible for the sessions management, used in service session_configuration as described in core.services.yml:

 session_configuration:
    class: Drupal\Core\Session\SessionConfiguration
    arguments: ['%session.storage.options%']

From a ServiceProvider class, by linking the service to our new custom class:

public function alter(ContainerBuilder $container) {
    $definition = $container->getDefinition('session_configuration');
    $definition->setClass('Drupal\custom_session_manager\StSessionConfiguration');
  }

Here you can see and download the custom module implemented as example and uploaded to my Gitlab repository. Check out the code and test the feature! (not for production - live environments, please). .

So now:

// Allows the cookie to be destroyed when closing browser.
$options['cookie_lifetime'] = 0;            
// Allows the session to be aligned with the frontend session.     
$options['gc_maxlifetime'] = 3600; 

And just enabling this new custom module, you will have a key change here: now the max-age of the session cookie will be set for expiring by session, see:


Max-Age for cookie set to Session

Now when you close your browser, the session cookie will be deleted, the user logout will be executed and on the next URL load, the user session will still be closed.

Session data & Garbage Collector & Cron

You can load the session time with the values you want. But this only let available the session data for the Garbage Collector of Drupal/PHP. It means that from 1 hour the session can be opened until the Garbage Collector was executed. Who’s executing the Garbage Collector? essentially, the cron in system. The “Automated Cron” core module can be enabled and set to 1 hours. It means that really, the inactive sessions (more than 1 hour without activity) will be deleted every 1 hour now, just when cron runs and launches the Garbage Collector.

But perhaps, we need a little more “granularity.” that is, to launch the cron only for these specific tasks, because launching it completely and for everything very frequently may not be the good idea we were thinking of. We need to be able to filter tasks associated with the cron in a more atomic way. For this we can rely on certain Drupal contributed modules that can provide us with functionality.

Useful Modules

Useful Drupal contrib modules we can use for more control over the session timing, roles and garbage collector from cron executions:

Session Limit: drupal.org/session_limit. You can limit the number of sessions available for a role / user in different environments.
Automated Logout: drupal.org/autologout. You can configure sessions settings values in order to force the logout in users.
Ultimate Cron: drupal.org/ultimate_cron. It Allows refinement in cron timing and execution’s tasks.

Futhermore, we can create some specific cron-tab tasks on the server by prompt, if we need more control over the session data. Cron is a time-based job scheduling daemon in Linux Operating Systems. You can learn more about how to configure tasks in background using cron in the next section.

:wq!

Recommended song: I Feel So Good - Joanna Connor

Decrease noise, gain focus: Reduce your stress level

2022-02-01T00:00:00+00:00


Picture from Unsplash, user Nikko Macaspac, @nikkotations

Pandemic times have increased our stress levels, without a doubt. A very strange time that gradually stabilizes (it does not disappear, but becomes constant, inescapable and already installed in our daily lives). During this time, we have added new layers of concerns to our lives. We have said Goodbye to many loved ones. We have become (somewhat) more aware of the dangers of our consumerist way of life, seeing the risks of global warming and how this is intertwined, creating new serious problems that travel fast in our interconnected “world-system”…

And furthermore, it’s just as true that before these days, we were living in a roller coaster of tasks, worries and rush - a great deal of rush -. A whole life inserted in the guidelines of the so-called “late capitalism”, a turbo-version agitated between the sources of a meritocracy that does not exist, the crazy consumerism and the thousands of daily injustices and arbitrariness. It is in this context that we have recently (and very closely) seen the true fragility of life and the real engines that govern it (and none of them really belong to us).

Strange times in which the old aspiration “to dream life, to change history” is beginning to reappear. But to begin to think we must remove the scabs of nostalgia and eliminate the noise and gain focus. Only then will we be able to reduce our stress and be able to think more clearly: anxiety, fatigue and stress are our enemies that prevent us from dreaming well.
How can we reduce our anxiety and stress level? There are some interesting ideas from some diverse contexts that can be useful to improve our inner state as a previous phase to be able to better reflect on what life (and what world) we want to have.

First: Simplify your daily life

To begin with, we must understand one of the basic tensions of our daily life: the relationship between what is important and what is accessory, what is key and what is secondary… in order to be able to order well how we consume our time.

Some oriental philosophies call our occidental mind “The monkey mind”: our brain is passing quickly from one stimulus to another and then, connecting to another new input. This has been seen as a role model since ancient times, but what were neon lights decades ago are now mostly redirected to the Internet environment: Social Media, links after links, colours, ads… Instagram has a notification to let you know that you’re already up to date on your entire timeline (Ok, I’m sure you know it) but did you know Instagram also has an option to estimate the time you have consumed using the app? …so we can ask ourselves a few questions:

How much time are you spending chasing digital stimulus?
Are digital social media robbing time in your life?

Think about the latest big thing released by Netflix: the film “Don’t look up” (Adam McKay, 2021)… the asteroid could fall on us at any time (as a rock or in the form of a pandemic). Do you want it to reach you while you are looking at your smartphone? would you rather have done something else?


Picture from Don’t look up, Hyperobject Industries, Bluegrass Films

Second: Purify, purify, purify always (debugging)

In addition, Let’s say in software development there are some principles oriented to get better results in any scenario, working iteratively and incrementally, perfecting your code. You have to think that your code can always be improved, but you should also think about reaching a minimum validated form, something functional that brings value and starts to solve your problem. Think about how methodologies such as “Lean” and its derivatives approach the processes and the achievement / scope of goals…Can we learn something?

Sometimes one way to manage our stress and anxiety is to reduce our goals to the achievement of a “minimum viable product”, thought to get ahead and leave aside an obsessive perfection that can be problematic. In times of pandemic we have become acutely aware that our time is limited, and we are learning to give it as much value as possible - but towards the things that matter to us - Look for example at the case of “the great resignation”: What are we doing with our time or our life? What do we need, basically? With the old “protestant” legends about work and the meaning of work out of the way, what’s left? maybe it’s time to think about our tasks and please, review the scope of your goals.

Is this task very complex?
Can they be reduced?
Is it really interesting?
Does this have a minimal version?

Perhaps it is time to review our daily agenda, our commitments and objectives, and as Danton said in the middle of the French Revolution: “Audacity, more audacity, always audacity”. So you know: Purge, debug, clean. Permanent movement, permanent purification, permanent revolution… it may seem interesting, no?


Picture of Jay Baruchel in the film The Trotsky, 2009

Third: Learn how to say ‘No’

Finally, there is an example I would like to share with you: José Saramago (Portuguese writer and Nobel Prize of Literature in 1998) once said in an interview that the word ‘No’ was one of his favorite words, and this made a strong impression on me. I was much younger than I am now, and lived more impressed by the culture of appearance, looks and relationships. Cultivating “no” was like growing a plant that could only lead to isolation, but I was wrong: it’s a sweet treat for personal dignity.


Portrait of José Saramago

Just how the author of “Blindness” and “The Cave”, try saying ‘No’ also as an internal way to review the connection of the thousand daily situations with your values and schemes…as someone once said: the best policy is the policy of principles.
Ask yourself:

How many things do I do a day that I don’t really feel like doing?
How many tasks would I rather not do?
How many comments and attitudes are obvious even though I know they are wrong?

Train your behavior. Start with small “no’s”. Weak, whispered, progressive, musical. Rehearse until you find the tone, the voice. Your voice. Then you can prepare explanations, give context, refute, open up your position, locate the opposite, establish your dividing lines. With time and training the “yes” will gain even more beauty and will be much more refined, more poetic, stronger. They will come to mean much more. We will learn to identify who are the strongest and who are the weakest in a conflict situation. We will be reaching the Satori.

Conclusion

To conclude, I would like to think that these lines have been useful to someone, starting with myself by putting together the ideas in my head after two years of continuous changes and various adaptations… Perhaps this article is not so much about reducing stress (what I had initially set out to do) as it is about stopping for a second to think about how we live, just before the flow of time sweeps us away.

In any case, I think that there are useful things in the former three blocks:

Ordering “real - digital” relationship
Reviewing and adjusting the scope of our goals
Learning to say ‘No’

You can live more comfortably and maybe, reduce your stress level. Or at least, you will gain some time to dwell on the only pending account that nostalgia (backward thinking) and worry (forward thinking) prevent us from facing: dreaming life, changing history.

:wq!

Recommended song: Tarantella del Gargano - Daniele Sepe

Drupal Tips: Altering order of execution in resources

2022-01-15T00:00:00+00:00


Picture from Unsplash, user Andre Taissin, @andretaissin

Changes, changes, changes… throughout a project, we have to implement many changes and alterations to the Drupal code, without touching it directly -of course-. The event system is very powerful but there are still hooks at all levels, so sometimes we must use Hooks that will alter things already modified by other Hooks of the same nature. Also we may need to execute our changes before the changes made by another module (contrib or custom), and for that we must understand well how the order of execution of resources works…

In this brief publication for this sketchbook, I’ve put together some quick techniques for making modifications to the run order.** What can we do to add new submit functions to a form? (incorporating new actions to that submit), How can we make our submit actions precede existing ones? What is the relationship between the “weight” assigned to a module and its position in the execution order? in an intuitive way I want to make an approach to these questions and along the way, share some useful resources, such as functions and available contributed modules that we can use for our changes.
Read this little post, it can be useful if you need execute this kind of changes.

Acknowledgments

This post was composed from my current position as Senior Drupal Developer at Digitalist Sweden, one of the biggest companies oriented to Open Source in Europe and specifically focused in Drupal.

TL;DR -> This is a little post about how to do changes in the order of execution of some Drupal resources by altering the position or weight of modules, hooks and other functions like a submit for forms.

Table of Contents.

1- Altering order in submit functions
2- Altering the priority order of modules
3- How to change the registry of modules using a hook
4- How to change the registry of modules using a contrib module
5- :wq!

1- Altering order in submit functions

Sometimes, it’s very important for us to alter the order of execution in certain types of submit functions. By instance, we’re using a previously existing form with its existing submit fuction, but we want alter the process by using our own hook. And inside this hook we can define a new submit fuction for the form: We need 1) Add a new submit function and 2) Alter the order and have our submit executed first.

Let’s see a more specific use case.

Use Case

We have custom changes in one entity (e.g a new field in the menu_link_content entity type) related with a node and loaded when we’re creating a new node by UI in Drupal:

In this case, just when asking for a menu entry wen we’re creating a new node in Drupal we’re always using the MenuLinkContent entity. Our change is as follow: We have a new field (a custom check for visibility) in this kind of Entity and we need to load/update the new field just when the node + menu_link_content are created and others modules are working on it (specifically the menu_ui module from Drupal Core). So we have to pass values for our new field, due to the original hook is not responsible for loading our new field (and it doesn’t know anything about it). So the main idea is using a form_alter Hook and then add a new submit callback function just when the node form is saved.

First I need to create my new submit function in a new custom module. For this step I’ll use the scheme of the pre-existing function menu_ui_form_node_submit within the menu_ui.module file. The key step here is: while we’re saving a node, the menu_link_content entity was already created and existing in database register.

/**
 * Form submission handler for menu item field on the node form.
 *
 * @see my_custom_module_form_node_form_alter()
 */
function my_custom_module_form_node_form_submit($form, FormStateInterface $form_state) {
    // Get the current node information and search the entity in database.
    $node = $form_state->getFormObject()->getEntity();
    $query = \Drupal::entityQuery('menu_link_content')
      ->condition('link__uri', 'entity:node/' . $node->id());
    $entity_id = $query->execute();
    
    // Load the existing entity menu_link_content.
    $entity = \Drupal::entityTypeManager()->getStorage('menu_link_content')
    ->load(array_key_first($entity_id));
    
    $entity->check_visibility->value = $form_state->getValues()['menu']['profile']['check_visibility'];
    $entity->save();
    }

Then in my new custom module I’ll create a new form_alter_hook. My goal is open the catalog of actions associated to the required form (node_form) and alter the order in submit callbacks. In order to do this, I’m filtering the type of actions that I’m interested to and then unsetting / setting the items.

/**
 * Implements hook_form_BASE_FORM_ID_alter() for \Drupal\node\NodeForm.
 *
 * Adds stuff to the node form.
 *
 * @see my_custom_module_form_node_form_submit()
 */
function my_custom_module_form_node_form_alter(&$form, FormStateInterface $form_state) {  
[...]

  foreach (array_keys($form['actions']) as $action) {
    if ($action != 'preview' && isset($form['actions'][$action]['#type']) && $form['actions'][$action]['#type'] === 'submit') {
        // Delete the former last item in the array of submit callbacks.
        array_pop($form['actions'][$action]['#submit']);
        // Now load the new order including the new custom submit action.
        $form['actions'][$action]['#submit'][] = 'my_custom_module_form_node_form_submit';
        // Reload the former submit function from the original core module menu_ui.
        $form['actions'][$action]['#submit'][] = 'menu_ui_form_node_form_submit';
    }
  }

[...] 
}

So, just when I’m saving a new node I’m reviewing the value of my new custom field “check_visibility”, getting from the form_state of the node form and saving the value in the menu_link_content entity. Then the next submit from the core module “menu_ui” goes on to run.

2- Altering the priority order of modules

In Drupal, the order of execution of a hook is related to the weight assigned to the module that contains it. Each module (custom, contrib, core) has assigned a “weight” value that can be located in the “system” table of the database and in the related configuration file “core.extension.yml” in config/sync/:

  typed_data: 0
  update: 0
  user: 0
  varnish_purger: 0
  views_ui: 0
  workflows: 0
  pathauto: 1
  content_translation: 10
  views: 10
  paragraphs: 11

You can set a low weight to get your module to execute after or before others. If no weight is defined modules get a default weight of 0 (the default value for core modules in config).

To find out the order in which the modules are executed you can get info using the configFactory service, by doing:

$extension_config = \Drupal::configFactory()->getEditable('core.extension');
$module_weight = $extension_config->get("module.MODULE_NAME"); 

Now we’re going to set a weight value for our custom module, inside a .install file, from the hook_install(). We can sue an existing function called ‘module_set_weight()’ available within the Drupal API present in module.inc, with functions oriented to interacting with Drupal modules. See this API here.
doing something like this:

/**
 * Implements hook_install().
 */
function my_custom_module_install() {
  [...]
  // Ensures the hooks contained in your module will be executed after the originals.
  // e.g. the hook menu_ui_form_node_form_alter()
  // Note: Weight for uninstalled modules can't be changed. 
  module_set_weight('my_custom_module', 100);
  [...]
}

Don’t forget update config, passing the active configuration (from database) to the passive configuration (configuration files) in order to have all the config values updated in your Drupal installation:

$ drush cex -y

(And don’t forget commit the new changes to the remote repository, of course).

3- How to change the registry of modules using a Hook

As you know, there are a lot de Hooks in Drupal, that is, predefined functions that cause alterations in in the normal operation of the installation, modifying aspects of the Drupal core or other modules. Ok you can check here all the available Hooks in Drupal 9.x. Ok but is there any Hook that can be useful for us? Yep, the hook_module_implements_alter that implements just the steps we’ve seen before: pull out of the array the indicated hook for a selected module and then save the same items at the end of the array (just like in the case of the submit functionts in the first section).

Exactly, this hook is executing:

function hook_module_implements_alter(&$implementations, $hook) {
  if ($hook == 'form_alter') {
    $group = $implementations['my_custom_module'];
    unset($implementations['my_custom_module']);
    $implementations['my_custom_module'] = $group;
  }
}

This is very useful to specify specifically which hook we want to alter at what time and for which module, locating the module by its name within the array of existing implementations.

4- How to change the registry of modules using a contrib module

There is also a contributed module that acts as a wrapper for this type of functionality but also provides several drush subcommands to execute and manage its features from the command line: The Modules Weight contrib module can change the order by GUI and by drush commands. Let’s see.

First, a fast view in my config/sync/core.extension.yml file, with some core modules:

Now I’m gonna installing and enabling the module in my DDEV deploy for testing:

$ ddev composer require drupal/modules_weight
  
  Using version ^1.9 for drupal/modules_weight
  [...]

$ ddev drush en -y modules_weight
$ ddev drush cr

And now from GUI present in /admin/config/system/modules-weight, I will change the order of the admin_toolbar family modules, as you can see:

And after saving the page, Now I can see the changes in the same config/sync/core.extension.yml file (after drush cex):

I have been testing the set of drush sub-commands provided by the module, but they don’t seem to work on my installed version of Drush (11.0.0 in Drupal 9.3.2) :-(

5- :wq!

Recommended song: Lazy Bones - Wooden Shjips

Drupal Techniques: How to upgrade Drupal

2021-09-29T00:00:00+00:00


Picture from Unsplash, user Raul Varzar, @calypso999

As you may have heard, the End Of Life for Drupal 8 is near. This is closing fast, very fast now. In fact the date for Drupal 8 EOL is November 2nd 2021, aligning this transition with the same EOL for one of the most important dependencies and key components: Symfony 3. Yep, the last minor version available for Symfony 3 (3.4) has marked like end for bug fixes and security fixes november 2021. As a rule, supported versions of Drupal must use supported version of Symfony, so this is the end unless you perform and upgrade in your Drupal platform…

A few days ago I had to work on a upgrade to Drupal 9 and along the way, I’ve taken some useful notes about the process. After publishing a quick guide for internal consumption I’ve decided to write a more extended version with all the issues. Since enough articles with FUD have already been published about the necessary Drupal upgrade, I won’t dwell on the risks or in potential problems: There are less than eight weeks left until Drupal 8 community support development stops and there is little time if you want to focus on Drupal 9 upgrades with your team. Read this article, review its steps, make your own checklist and get started!

Acknowledgments

This post was composed from my current position as Senior Drupal Developer at Digitalist Sweden, one of the biggest companies oriented to Open Source in Europe and specifically focused in Drupal.


Image taken from drupal.org/how-to-prepare-for-drupal-9

In addition, there are a couple of premises or prerequisites that we must satisfy, just like a pair or assumptions:

Assumption 1: We’re working in a Drupal 8 installation now or in a Drupal major version > 7 (Drupal 9 to Drupal 10 maybe). This is not an article about Drupal Migrations, so I’m not covering a Drupal 7 to Drupal 9 migration and upgrading (a more extensive, complex and exhaustive process due to the huge structural difference in the jump between Drupal 7 and Drupal 8), so this post is for Drupal > 7 installations, sorry.

Assumption 2: Your Drupal installation is under the control of Composer as dependency manager. I’m quite aware that there are still projects that don’t use Composer as a main dependency manager and also -as in Allen Ginsberg’s poem: Howl poem- “I saw the best minds of my generation destroyed by madness”. But you need composerizate your project (that is, put it under Composer management). This article is for Drupal 8 installations using Composer.

TL;DR -> This is a post about how to execute (steps, tips, risks…) the upgrading process from major versions of Drupal (D8 to D9, D9 to D10, etc.).

If you want to learn how to execute Drupal upgrading reading resources from external sources, I have a Gist in Github with my favourite content, here.
If you came here looking for information about Drupal updates for minor versions (Drupal 9.2.6 -> 9.2.7 -> 9.2.8), better visit this snippet I have in Gitlab with related information.
If you want to have a quick overview, download the image below and zoom in to see the recommended steps.

See the image below:

Table of Contents.

1- Setting the scenario
2- Sorting the toolbox
3- Assesing the status of your project
4- Preparing the work
5- Executing the work
6- Troubleshooting
7- Read More
8- :wq!

1- Setting the scenario

As a first step we need to select our Drupal installation on which we have to perform the upgrade.

GOAL: Have an enabled Drupal setup.

I’m going to deploy in my local environment a new Drupal 8 site for testing updates to Drupal 9. I will call it “drupal8function”, with an user “admin” and password “admin”, using DDEV as deployment tool, as I wrote down here in this gitlab snippet, just by typing the next commands in prompt:

$ mkdir drupal8function && cd drupal8function
$ ddev config --project-type=drupal8 --docroot=web --create-docroot
$ yes | ddev composer create "drupal/recommended-project:^8"
$ ddev composer require drush/drush drupal/admin_toolbar drupal/devel
$ ddev exec drush si --site-name=drupal8function --account-name=admin --account-pass=admin -y
$ ddev drush en -y admin_toolbar devel
$ ddev drush cr
$ ddev start && ddev launch

I need some custom resources for testing so I’ll clone some Drupal 8 modules that I have in my Gitlab repository. My idea is taking some modules in a Drupal 8 version and then moving to my /custom folder enabling it by drush, so I’ll have Drupal 8 custom resources ready to review for the upgrade to Drupal 9. Let’s see some D8 modules:

$ git clone https://gitlab.com/davidjguru/drupal-custom-modules-examples.git
$ mkdir ./web/modules/custom
$ mv drupal-custom-modules-examples/links_example ./web/modules/custom/
$ mv drupal-custom-modules-examples/headers_manager/ ./web/modules/custom/
$ mv drupal-custom-modules-examples/Migrations/migration_basic_module/ ./web/modules/custom/
$ ddev drush en -y links_example headers_manager migration_basic_module
$ ddev drush

Also, I’ve added some patches for Drupal 8 core modules, by adding some links in composer.json files:

"patches": {
  "drupal/core": {
    "Content_entity, config_entity Plugins": "https://www.drupal.org/files/issues/2020-09-24/3172853-2.patch",
    "AccessAwareRouter HTTP method.": "https://www.drupal.org/files/issues/2018-12-12/2706241-34.patch",
    "Sets langcode.": "https://www.drupal.org/files/issues/2020-06-10/locale-config-langcodes.patch",
    "Layout builder assign inline block access.": "https://www.drupal.org/files/issues/2021-03-22/3047022-71.patch",
    "Generate pathauto aliases.": "https://www.drupal.org/files/issues/2020-11-06/3181070-possible-need-to-change-the-way-menulinkmanager-updates-menu-item-and-menu-tree.patch"
   }
 }

Ok, now I have a semi-real scenario composed by a Drupal 8 core, patches, some contrib modules and custom resources in Drupal 8 too. So when I ask for the Drupal core version to Composer, I’m getting:

The next step will be Preparing the toolbox.

2- Sorting the toolbox

First of all I think we need to prepare a toolbox with some basic utilities for our upgrading plan.

GOAL: Have a set of well-know and tested tools.

I have selected here only five basic resources, very easy to use. These are the following items:

2.1- upgrade_status

Before studying and designing the process for update your Drupal installation, maybe you need a high-level view, a fast summary about what is the current status in your platform. The contrib module “Upgrade Status” can provide you a fast review about this and it provides you with some solutions and tips too.

$ ddev composer require drupal/upgrade_status
$ ddev drush en -y upgrade_status
$ ddev drush cr

(Avoid using the command ddev if you are not using DDEV, of course. Or change by lando command.)

And if you don’t want touch your Drupal core, ask for dev versions for core and dependencies getting first your current core version in terminal:

$ ddev composer show drupal/core | grep versions
versions : * 8.9.18

$ ddev composer require --dev drupal/core-dev:8.9.18 --update-with-all-dependencies
$ ddev composer require drupal/upgrade_status
$ ddev drush -y upgrade_status
$ ddev drush cr

Once installed and enabled, this module will offer you a GUI and a pair of new drush subcommands in order to get the reports. We’ll see the results in a next section.

2.2- php-stan

PHPStan is a PHP Static Analysis Tool created by Ondřej Mirtes and others that combined with a set of custom rules for Drupal can do the job reviewing the status of your code. First, We have to install all the resources:

$ ddev composer require  --dev phpstan/phpstan \
  phpstan/extension-installer \
  mglaman/phpstan-drupal \
  phpstan/phpstan-deprecation-rules

The phpstan/extension.-installer will autoconfigure PHPStan to load phpstan-drupal and the deprecation-rules, the resource that will delivery the reports about using deprecated code in your installation and resources.

Then we can run phpstan against the selected resources by doing something like:

$ ddev php vendor/bin/phpstan.phar analyze web/modules/custom/my_custom_module

And you can change the level for the review, from 0 (the loosest) to 8 (the strictest), passing the parameter -l to the command:

$ ddev php vendor/bin/phpstan -l 4 analyze web/modules/custom/my_custom_module

2.3- drupal-check

Drupal Check is a static analysis tool developed by Matt Glaman based in PHPStan, a very useful resource for these updating processes, allowing us to search for deprecated code in our resources, errors and some more interesting information. You can use it as a new Composer dependency or as VSCode extension.

We’re going to install the tool through Composer, by doing:

$ ddev composer require mglaman/drupal-check --dev

And we’ll have available a new command from prompt: ‘drupal-check’. We’ll cover the use in a next section.

2.4- drupal-rector

Drupal Rector is a fixing deprecated code tool for Drupal, built by Palantir and available as a Composer dependency. It can inspect, proposing and apply patches for changes in your software from its code reviews.

2.5- backward_compatibility

This is a little contrib module oriented to simple cases: there are a lot of contrib modules in Drupal that only requires add a single line in its module.info.yml file, I mean:

core_version_requirement: ^8 || ^9

Well, instead of patching every contrib module with this problem, just install the resource and it will work’round the Issue. Just by doing:

$ composer require drupal/backward_compatibility
$ drush en -y backward_compatibility
$ drush cr

And nothing more! (simple solutions for very specific use cases).

3- Assesing the status of your project

Now It’s time to get a reliable image of the status of our project through the tools selected in the former section.

GOAL: Have a true picture of the work to be done.

It will be no secret to say that our field of work in this updating process will be mainly in the custom zone of our resources: We will dedicate a large percentage of time to the adaptaion of our custom modules, our custom themes and our patches applied to the project that are in local folders and not in Drupal.org (which is another category of custom work for us).

3.1- Taking a snapshot of your platform

Before we can estimate how much time we will spend and how many actions we need to implement, let’s do some initial data collection and prepare some resources. This will give us a useful snapshot of the status of our platform.

Ok, first you have to review the current status of your local deploy. Remember that your installation has five pillars, five main columns:

Composer files: composer.json and composer.lock
Config Files, usually in /config/sync/
The files folder, usually in /web/sites/default/files/
The database of your Drupal installation
All your custom work: custom modules, custom themes and (maybe) custom patches.

Put all the above resources in a safe place of your choice.

Remember that if you did your homework and assignments right, then the alignment between “Active Configuration” (database) and “Passive Configuration” (configuration files) will be complete, I mean:

$ ddev drush cex
[notice] The active configuration is identical to the configuration in the export directory (sites/default/files/sync).
sites/default/files/sync

$ ddev drush cim
[notice] There are no changes to import.

And there will be no change to identify. Otherwise, you should always prioritize the active database configuration as an up-to-date reference of the state of your Drupal installation.

Some Steps:

Preparing your containers and backups
Check your current version for Drupal Core
Enable the upgrade_status module and check results
Check code with drupal-checker and drupal-rector

3.2- Containers and Backups

It’s time to review some special configurations and values in your environments…what versions of Nginx, Apache, MySQL, Postgres or MariaDB you have? check the current versions in your environments, containers, etc.

3.3- Current version for Drupal Core

Let’s confirm our core version by typing in prompt:

$ ddev composer show drupal/core | grep versions
versions : * 8.9.18

3.4- Upgrade Status

After enabling the upgrade_status module, we’ll inspect the current status of our installation accesing to /admin/reports/upgrade-status and seeing the first information area of the status report:

Here you can see the general review of your resources, custom and contrib, and seeing the changes to do, you will be able to estimate the amount of time you will need to dedicate to these updates,, about the hosting, core and environment:

Or by seeing the specific modules:

We can use the new Drush commands provided by the module to explore some specific modules, doing:

 $ ddev drush upgrade_status:analyze admin_toolbar
 [notice] Processing /var/www/html/web/modules/contrib/admin_toolbar.

================================================================================
Admin Toolbar,  3.0.2
Scanned on Mon, 09/20/2021 - 17:15

FILE:
web/modules/contrib/admin_toolbar/admin_toolbar_search/src/Controller/AdminToolb
arSearchController.php

STATUS         LINE                           MESSAGE                           
--------------------------------------------------------------------------------
Check manually 15   Class PHPUnit\Framework\TestCase not found.                 
--------------------------------------------------------------------------------

[...]

--------------------------------------------------------------------------------

This module is ready for Drupal 9.

3.5- Drupal-Checker and Drupal-Rector

$ drupal-check -a web/modules/contrib/admin_toolbar

Install by doing:

$ ddev composer require --dev palantirnet/drupal-rector

You can give to Drupal Rector a GUI from your Drupal installation:

$ ddev composer require 'drupal/upgrade_rector:^1.0@alpha'
$ ddev drush en -y upgrade_rector
$ ddev drush cr

And then go to /admin/reports/upgrade-rector and see the reviews available for contrib and custom resources:

3.6- Get a list of deprecated resources

You can check the current status of all your dependencies, getting a report with related information from Composer. Just run:

$ ddev composer outdated

And you will get:

alchemy/zippy                            0.4.9    1.0.0   Zippy, the archive manager companion
asm89/stack-cors                         1.3.0    v2.1.1  Cross-origin resource sharing library and stack middleware
chi-teck/drupal-code-generator           1.33.1   2.5.1   Drupal code generator
composer/composer                        2.2.6    2.2.9   Composer helps you declare, manage and install dependencies of PHP projects. It ensures you have the right stack everywhere.
composer/installers                      v1.12.0  v2.1.0  A multi-framework Composer library installer
composer/pcre                            1.0.1    3.0.0   PCRE wrapping library that offers type-safe preg_* replacements.
composer/semver                          3.2.6    3.3.1   Semver library that offers utilities, version constraint parsing and validation.
composer/xdebug-handler                  2.0.4    3.0.3   Restarts a process without Xdebug.
consolidation/annotated-command          4.3.2    4.5.2   Initialize Symfony Console commands from annotated command class methods.
consolidation/config                     1.2.1    2.1.0   Pr

[...]

Now you have a little overview about all the pending tasks for upgrading all in your Drupal installation.

4- Preparing the work

Somebody, somewhere - maybe yourself - needs to know what needs to be done, risks we run and how much time / effort the upgrade needs.

GOAL: Haver a reliable estimation of the process.

Ok, after putting in a safe place a copy of the five basic parts of your installation (config, database dump, files folder, composer files and all your custom works -themes, modules, patches, etc. and after checking the state for update, you’ll be able to build a map with all the required tasks related with the update, I mean:

List of resources that only requires basic and fast changes: the quickest and easiest bet to execute. Small and perhaps repetitive changes that you might be able to run the same way in the same bash script or through the same Git patch.
List of resources that contains deprecated code: If you have ignored successive warnings and code deprecation warnings since previous versions (don’t be like me), you may need time to make adjustments to this part. Check the warnings thoroughly now.
List of resources that requires refactoring or advanced changes: Perhaps one of the most complex parts. You may take time to balance this part according to the latest changes in core and contributed modules, to adapt your own functionality in custom modules and to rewrite code. Be cautious and value well this estimation.
List of items to update in environments (versions for PHP, MariaDB and other containers stuff): A part that tends to be overlooked but someone will take care of executing… Do our current Docker images work? are our containers up to date? do they require a higher version of a resource? don’t forget this section.

After reviewing all this and compiling all the information provided by the tools discussed above as output, you can now think about what’s left, what output can we expect, you can combine all your estimates and build a tracking document of all the tasks associated with the upgrade. I use something like this, something primary but very useful as a checklist:

5- Executing the upgrade

The time has come to make the necessary changes and adjustments. Let’s go!

GOAL: Have an experienced process (trial, error, trial, error…)

5.1- Remove some resources

Remove drupal/console: (usually causes problems with symfony dependencies), so If you have installed drupal console, remove it as first step.

$ ddev composer remove drupal/console

Now, remove drupal/core and/or drupal/core-recommended from composer.json. Look for drupal/core-recommended or drupal/core in composer.json and remove the line(s):

"drupal/core-recommended": "^8.9.0"

If you have drush and devel, which are widely used, also manually edit the versions:

"drupal/devel": "^4.0",
"drush/drush": "^10.0.0",

Then run composer update again:

$ ddev composer update

This will remove drupal core from your installation but will update the contrib resources.

5.2- Set your new Drupal Core version

Add drupal/core-recommended:9.0.0 to composer.json

"drupal/core-recommended": "9.0.0",

Now run composer update one more time:

$ ddev composer update

5.3- Delete modules with problems

In my case, During the last tasks of upgrading projects to Drupal 9, I have some problems with a specific contrib module: multivalue_form_element

Problem 1
- Conclusion: remove drupal/core-recommended 9.0.0
- Conclusion: don't install drupal/core-recommended 9.0.0
- Conclusion: don't install drupal/multivalue_form_element 1.0.0-beta2|don't install

Why? Well the module seems to be available only from Drupal 9.1:

When deleting this module from composer.json, run composer update and now You will have the new core version (9.0.0) installed. We will back to this module in next steps.

5.4- Upgrade to latest Drupal 9 version

After this, change your drupal/core-recommended line in composer.json to:

"drupal/core-recommended": "^9.0.0",

And re-run composer update .

5.5- Update patches

Sure you have some patches applied to your Drupal Core from the composer.json file. I have, for example these, mixin’ contrib from drupal.org and custom from a local folder within the project:

"drupal/core": {
"Add content_entity and config_entity DataType plugins (https://www.drupal.org/project/drupal/issues/3172853)": "https://www.drupal.org/files/issues/2020-09-24/3172853-2.patch",
"AccessAwareRouter does not respect HTTP method (https://www.drupal.org/project/drupal/issues/2706241)": "https://www.drupal.org/files/issues/2018-12-12/2706241-34.patch",
"Claro removes contextual links for inline blocks translate - only needed when using claro in layout builder": "./patches/claro-preprocess-contextual.patch",
"Sets the langcode for config export to en": "https://www.drupal.org/files/issues/2020-06-10/locale-config-langcodes.patch",
"Layout builder fails to assign inline block access dependencies for the overrides section storage on entities with pending revisions.": "https://www.drupal.org/files/issues/2020-09-10/3047022-43.patch",
"Generate pathauto aliases when rearranging menu.": "https://www.drupal.org/files/issues/2020-11-06/3181070-possible-need-to-change-the-way-menulinkmanager-updates-menu-item-and-menu-tree.patch",
"Remove entity constraint validator for inline blocks": "./patches/3053881-30-reroll-8.x.patch",
"Replace token module internal link to nodes with a entity link": "./patches/menu_ui-token-uuid-to-id.patch",
"Drupal dialog respect drupalAutoButtons: false": "./patches/drupal-dialog-respect-drupalautobuttons.patch",
"Make drupalAutoButtons a boolean": "./patches/OpenDialogCommand-drupalAutoButtons-boolean.patch",
"Tabledrag nested stuff.": "https://www.drupal.org/files/issues/2020-12-08/3092181-142-8X_0.patch",
"Error in computed image fields (width, height) when saving nodes": "https://www.drupal.org/files/issues/2019-02-11/ignore_width_height_on_untranslatable_field_comparision_2941092_8.patch",
"Vary block definitions on language": "./patches/block-manager-cache-key.patch",
"Expose resetSize from dialog position to the Drupal object": "./patches/expose-reset-size-to-drupal-object.patch"
 },

So when I’m trying update my Drupal core I’m getting these errors:

Could not apply patch! 
  Skipping. The error was: Cannot apply patch 
    https://www.drupal.org/files/issues/2020-09-10/3047022-43.patch
[...]
Could not apply patch! 
  Skipping. The error was: Cannot apply patch ./patches/drupal-dialog-respect-drupalautobuttons.patch
[...]
Could not apply patch! 
  Skipping. The error was: Cannot apply patch 
  https://www.drupal.org/files/issues/2020-12-08/3092181-142-8X_0.patch

As you can see, I’m having problems with three patches for Drupal Core. Ok, so here we have to do some manual work…We have to check for every patch:

If the patch was already applied to the 9.x development branch, so It’s not necesary to apply now in the upgrade.
If the patch was not applied, then you have to look for a new re-rolled patch ready for the new version of Drupal (> 9.x.x.).
If the patch doesn’t come from drupal.org and it’s a custom solution located in /patches directory, then you’ll have to work in the migration of the resource, re-rolling the patch. :-/
If the patch has not been applied and there is no version for Drupal 9 either, so you have to talk with your team, including your scrum master and / or Product Owner. The original problem will come back to the Drupal installation until you or somebody in the Drupal community can port the patch to Drupal 9. Sorry :-/

And looking the original Issues, I can see versions for Drupal 9.2, as here.

I found a solution for Drupal 9 here.

Or there:
www.drupal.org/issues/3092181
Seems to be resolved from Drupal 9.2.

My third patch with problems is a local custom patch in /patches folder, so I have to do some coding review here :-/

5.6- Reinstall modules with Issues

After install the updated core, upgrading to the latest version available and working with patches, now can request the problematic modules, like the multivalue_form_element:

$ ddev composer require drupal/multivalue_form_element

Remember that you can avoid problems pre-testing the installation of dependencies in Composer by using:

$ ddev composer install --dry-run

5.7- Run updates in database

Important: Not run the database updates before install the modules with issues: You can have the module saved as active in database but the codebase doesn’t exist, due to this you’ll get a lot of problems. Remember: first update core, then install modules with issues and after all, launch the database updates.

$ ddev drush updb -y
$ ddev drush cr

5.8- Review all your custom work

You can use the well-formed tools commented in the section about tooling, doing so:

$ ddev composer require mglaman/drupal-check --dev
$ ddev drupal-check -a web/modules/custom

5.9- Checking new installed versions

name     : drupal/smtp
descrip. : Allow for site emails to be sent through an SMTP server of your choice.
keywords : 
versions : * 1.0.0
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved) https://spdx.org/licenses/GPL-2.0-or-later.html#licenseText
homepage : https://www.drupal.org/project/smtp
source   : [git] https://git.drupalcode.org/project/smtp.git 8.x-1.0
dist     : [zip] https://ftp.drupal.org/files/projects/smtp-8.x-1.0.zip 8.x-1.0
path     : /app/backend/web/modules/contrib/smtp
names    : drupal/smtp

support
source : https://git.drupalcode.org/project/smtp
issues : https://www.drupal.org/project/issues/smtp

**requires
drupal/core ^8.8 || ^9**
phpmailer/phpmailer ^6.1.7

Or:

$ ddev composer show drupal/pathauto

Getting:

name     : drupal/pathauto
descrip. : Provides a mechanism for modules to automatically generate aliases for the content they manage.
keywords : 
versions : * 1.8.0
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved) https://spdx.org/licenses/GPL-2.0-or-later.html#licenseText
homepage : https://www.drupal.org/project/pathauto
source   : [git] https://git.drupalcode.org/project/pathauto.git 8.x-1.8
dist     : [zip] https://ftp.drupal.org/files/projects/pathauto-8.x-1.8.zip 8.x-1.8
path     : /app/backend/web/modules/contrib/pathauto
names    : drupal/pathauto

support
source : https://cgit.drupalcode.org/pathauto
issues : https://www.drupal.org/project/issues/pathauto
documentation : https://www.drupal.org/docs/8/modules/pathauto

**requires
drupal/core ^8.8 || ^9**
drupal/ctools *
drupal/token *

and:

$ ddev composer show drupal/core

name     : drupal/core
descrip. : Drupal is an open source content management platform powering millions of websites and applications.
keywords : 
**versions : * 9.2.5**
type     : drupal-core
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved)

5.10- Remove tooling for upgrade

Remove the upgrade_status contrib module or the extra resources by typing:

$ ddev drush pmu -y upgrade_status
$ ddev composer remove drupal/upgrade_status
$ ddev drush cr

$ ddev composer remove mglaman/drupal-check --dev

5.11- Commit files with changes

You will have to commit to repository the new versions of your composer.json and composer.lock files, in order to get the new versions in the next deploys for environments.

6- Troubleshooting

Maybe it’s not your last update or a colleague needs help: make your experience a validated learning experience.

GOAL: Have a reusable set of problems and solutions.

Here I would like to gather some problems and limitations observed in various Drupal 9 upgrade tasks.

6.1- GraphQL is giving me some errors

For Drupal 9, GraphQL module has a dependency in alpha version, which can crash with your Drupal 9 installation: the drupal/typed_data module (https://www.drupal.org/project/typed_data):

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/graphql ^4.1 -> satisfiable by drupal/graphql[4.1.0].
    - drupal/graphql 4.1.0 requires drupal/typed_data * -> found drupal/typed_data[dev-1.x, 1.0.0-alpha1, ..., 1.x-dev (alias of dev-1.x)] but it does not match your minimum-stability.

And so, by doing: ddev composer require drupal/typed_data

$ ddev composer require drupal/typed_data                                                                                                                                        
[InvalidArgumentException]                                                                                                                                                      
Could not find a version of package drupal/typed_data matching your minimum-stability (stable). Require it with an explicit version constraint allowing its desired stability.

It can be changed in composer.json file, where you’ve marked stability as “stable”, so composer won’t accept the dependency in alpha, so you can change your requirements, allowing the minimum-stability of packages in dev or alpha but prefer stable.

This has to be changed in composer.jsonfile in your project root folder:

 "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",  <= THIS
    "prefer-stable": true,**
    "config": {
        "sort-packages": true,
        "platform": {
            "php": "7.4"
        }
    },

But it’s not a good idea, cause you can change the level of minimum-stability and it could be a risk for your project. Instead, use the recommended way that is only install in alpha the dependency, step by step by doing:

$ composer require drupal/typed_data:@alpha
$ composer require drupal/graphql

6.2- I can’t create new users

I noticed that after an update to Drupal 9, in some projects is not possible create a new user in path /admin/people/create, you can’t see the second password field “Confirm Password” and due to this, the form is not saving a new user.

Is this happening to you? Don’t worry, it has an easy solution. Let’s see.

First, this happens when using custom themes based in Claro (https://www.drupal.org/project/claro) now, in Drupal Core -but experimental by now-.
Second, The error is coming from a change in Claro theme. In Claro 9.2.5 (Drupal Core 9.2.5) there is a change in a library declaration and this was moving from one to another block.
Third, you only need edit your custom_theme.info.yml file and move the user/drupal.user: claro/form.password-confirm pair from the block “libraries-override” to the block “libraries-extend”. Now:

libraries-extend: <= HERE
[...]
  core/drupal.ajax:
    - claro/ajax
  user/drupal.user:
    - claro/form.password-confirm  <= HERE
  views/views.module:
    - claro/views
[...]

Fourth, only save the change and clear cache by doing $ drush cr and now you’ll see the second field, saving new users.
Fifth (remember), Claro theme is experimental in Drupal Core, and is not stable.
Sixth (more remember), You must to know that subtheming from Admin Themes are not offically supported in Drupal, so be carefull: Related Issue in Drupal.org.

Read more about this Issue in Drupal.org.

Read more about subtheming an Admin theme in Drupal.org: drupal.org/issues/3118850.

6.3- Config Split not seem to work well in deploys

After updates to Drupal 9 and moving forward versions of contributed modules, in some cases there are problems when importing configuration files orchestrated by the contrib module config split and launched from some task runners tools like dropcat, robo, drobo (robo for Drupal) etc.

Well, some Drush commands aliases has been changed in Config Split and maybe you have to change your commands:

But the most important thing is:… the version 2.x is not recommended by the module maintainers…so please, downgrade to the former version.

6.4- Review (and change) deprecated code

As part of the code upgrades - especially working with custom code - in the move to Drupal 9 you will have to make mandatory changes that were previously only optional: deprecated code that (now yes) you will no longer be able to use in D9 (in D8 you were only informed or warned).

For example, I have received warnings but I have not changed some services (don’t be like me) and when I got to Drupal 9 I received errors like this:

Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException: 
**You have requested a non-existent service "entity.query".** 
Did you mean one of these: "entity.query.config", "entity.query.sql", 
"pgsql.entity.query.sql", "entity.query.null", "entity.query.keyvalue"? 
in Drupal\Component\DependencyInjection\Container->get() 
(line 156 of 
/var/www/html/web/core/lib/Drupal/Component/DependencyInjection/Container.php).

This message is about the Issue: drupal.org/2849874 talking about a service deprecation. It has been around for a long time but now generates errors and should be removed.

In this case I was using the service:

$query = Drupal::service('entity.query')->get('menu_link_content');
$query->condition('link.uri', 'entity:node/' . $node->id());
$result = $query->execute();

It’s just a service provided by the class QueryFactory: api.drupal.org/entity.query

So now you have to use the general manager for entities, the entityTypeManager service:

$query = \Drupal::entityTypeManager()->getStorage('menu_link_content')->getQuery();
$query->condition('link.uri', 'entity:node/' . $node->id());
$result = $query->execute();

And entityQuery is available too:

$query = \Drupal::entityQuery('menu_link_content');
$query->condition('link.uri', 'entity:node/' . $node->id());
$result = $query->execute();

Please, think about using review-code tooling like drupal-check and drupal-rector in order to avoid these kind of errors in code deprecations.

7- Read More

Many others have also written about the upgrade process. There is also information from Drupal.org, as well as many related experiences. If you want to go deeper into this, I recommend here several reading materials from different organisations, companies and authors.

8- :wq!

Recommended song: Panda - Dungen

PHP Coding: Reverse Geocoding for taxonomy terms

2021-06-28T00:00:00+00:00


Picture from Unsplash, user Nafis Al Sadnan, @saddy143

Have you ever had to perform geocoding? I mean, the exercise of getting some kind of data from stuff like a specific naming in order to get values like longitude and latitude. It had been a long time since I had used the know as “Geographic Information Systems” (GIS), when I was a young (and naive) Java developer…

The fact is that I recently had to perform some tasks related to Geocoding: I needed to get some values from latitude and longitude values, the so-called “Reverse Geocoding”. My goal was fill out a Drupal taxonomy and populating a related field in a specific content type. I have decided to compile my case notes here in a sequential way, but the experience was previously published separately as a sucession of Gitlab Snippets and Gist from Github that you can follow from here.

TL;DR -> This is a post about how to execute Reverse Geocoding for populating a taxonomy term field in Drupal 8, 9.

Table of Contents.

1- What am I looking for?
2- Reverse Geocoding for PHP
3- Resources
4- Loading taxonomy terms
5- Read More
6- :wq!
————————————————————————————————

1- What am I looking for?

Well, I’m inside a scenario where I’m creating new nodes of a specific content type, then I’m populating its fields programmatically and finally saving the whole new node. I’m doing all the process by coding, out of the Drupal GUI and from a custom code made by me. This is the big picture.

At a higher zoom level, within all its related fields, I have a Entity Reference field, specfically for a Hierarchical Taxonomy Term, I mean, a field with values to pre-charged taxonomy terms from a populated vocabulary which contains terms in three levels:

My vocabulary is storing places from Peru (Perú, Piruw) in South America. This is a set of more than 2000 terms divided in a tree of hierarchy with three level. In Peru there are 24 departments, 25 regions, 196 provinces and 1838 districts, so It looks pretty extensive, something like:

I receive geocoded files with some values and I take long, lat and I start to consult to wich point they belong, in order to obtain their things like its department, province and district. Then I’ll look for a match with my pre-populated taxonomy and I’ll save the three references to taxonomy terms maintaining hierarchy. Ok? Let’s recap.

Steps:

1- Processing long, lat values.
2- Getting Department, Province and District from the long, lat values.
3- Looking for matching in my vocabulary.
4- Setting the matched values in the taxonomy hierarchy for each new node.
5- Saving the new populated node.

2- Reverse Geocoding for PHP

Since we have to perform Geocoding tasks, let’s first remember some basic concepts and what kind of main resources we have available.

What is “Reverse Geocoding”

First, let us initially review the concept of Reverse Geocoding:

“Reverse geocoding is the process of converting a location as described by geographic coordinates (latitude, longitude) to a human-readable address or place name. It is the opposite of forward geocoding.”

Source: Reverse Geocoding in Wikipedia

Installing geocoder for PHP

Geocoder PHP is the most famous library for Geocoding operations written in PHP language.
Geocoder PHP provides: “an abstraction layer for geocoding manipulations”. You can get the resources from its repository in Github and as a Packagist set of packages. The main library is divided into three basic concepts in order to work with Geocoding: an HttpAdapter for doing requests, several geocoding Providers and various Dumpers to get a formatting output. You will need two basic resources: the main library and a provider. I’m gonna install it in my Drupal local deploy:

$ ddev composer require willdurand/geocoder
$ ddev composer require geocoder-php/locationiq-provider 

Or whithout ddev if you’re out of this tooling:

$ composer require willdurand/geocoder
$ composer require geocoder-php/locationiq-provider 

Now I have the classes available through the class loader. I can use the resources in my installation. Geocoder only requires just a pair of keyresources:

An external provider
A HTTP Client for connections.

I have the provider clasess and I have the HTTP Client too, Guzzle library, present and available in Drupal. So what I need now is get data from an external service. Let’s see some ideas about it.

External Services

As you can see in former sections I’m using the classical https://geocoder-php.org library. It requires the main library and the external providers classes in order to connect to others external APIs.
In this case I’m using LocationIQ to get the addresses: https://locationiq.com in free mode. I’ve created a new user in this platform and I’m getting a token API for queries.

Some Limitations for the free mode of LocationIQ:

1 Access Token
5,000 request/day
2 request/second

But well, only for some tests this may be useful.

Making External Reverse Geocoding Queries

Now I have the Geocoding library, the classes for the provider and data for connections to external provider, I can do some tests for Reverse Geocoding. Now for instance, I wanna build a address like an unique string builded from external received data:

<?php

[...]
use Geocoder\Query\ReverseQuery;
use Http\Adapter\Guzzle6\Client;
use Geocoder\Provider\LocationIQ\LocationIQ;
use Geocoder\StatefulGeocoder; 
[...]

// You can use other options for the HTTP Client. 
// $httpClient = \Drupal::httpClient();
$httpClient = new Client();
$provider = new LocationIQ($httpClient, "MY-APIKEY-FROM-LOCATIONIQ");
$geocoder = new StatefulGeocoder($provider, 'en');

// Launch the external query.
$result = $geocoder->reverseQuery(ReverseQuery::fromCoordinates($latitude_data, $longitude_data));

// Mount the returned address like a unique string. 
$address = $result->first()->getStreetName() . ' ' . $result->first()->getStreetNumber() . ', ' .
            $result->first()->getLocality() . ' ' . $result->first()->getPostalCode() . ', ' .
            $result->first()->getCountry()->getName();

And then I can load the address variable in a Drupal field. But this is not my case, remember I need to populate taxonomy terms fields in a Drupal node, so beyond this example I need to test more resouces.

3- Resources

By searching these resources we can find libraries, services and APIs for consumption that have as few limitations as possible (rate limit, requests per second/day, etc), and finally achieve a more efficient implementation.

External Public Resources

There are some public resources opened to connections, URLs ready to connections, just like this cartociudad.es/geocoder/ with its own Geocoding / Reverse Geocoding services (click and see the results in browser).

The operations are free and simple, just an open URL and query parameters, but maybe the service is not available or stable enough for you. Let’s see other options.

What about Drupal

In Drupal you can use the Drupal Geocoder Module, a contrib module that acting like a wrapper for geocoder library and resolves by itself the installation of the main library, as you can see in: geocoder/composer.json.

You have to follow the instructions from its README file: geocoder/README.md.

I certainly haven’t had much luck using this module. Some strange bug limits me to use it for quick situations. There are some weird issues installing providers, see the Issue 3153678.

Nominatim

Nominatim is an API used to search OSM data by name and address in order to doing Geocoding and Reverse Geocoding. OSM is “Open Street Map”, a collaborative project creating a big, free and editable map of the world. Maria Arias de Reyna, @delawen and Juan Luis Rodríguez, @juanluisrp guided me in this way. They are the greatest GIS experts I ever know and they were very helpful here, givin’ me some ideas and links to resources. Kudos. Specifically, @delawen told me about Nominatim, a heavy-weight solution for Geocoding Operations and I began to play with it. What is ‘Nominatim’? First Bus Stop:

“Nominatim is the geocoding software that powers the official OSM site www.openstreetmap.org. It serves 30 million queries per day on a single server.”

Source: https://nominatim.org

Ok, I see I can do my queries using Nominatim, but what else?

Nominatim offers some Endpoints for specific queries:

/search - search OSM objects by name or type
/reverse - search OSM object by their location
/lookup - look up address details for OSM objects by their ID
/status - query the status of the server
/deletable - list objects that have been deleted in OSM but are held back in Nominatim in case the deletion was accidental
/polygons - list of broken polygons detected by Nominatim
/details - show internal details for an object (for debugging only)

And you can use Nominatim from a local Docker deploy or from live in external server:

https://nominatim.openstreetmap.org/reverse?lat=<value>&lon=<value>&<params>
https://nominatim.openstreetmap.org/reverse?format=json&lat=-12.046374&lon=-77.042793
https://nominatim.openstreetmap.org/reverse?format=json&lat=-9.949480903131423&lon=-78.22368622836807

What will return a Reverse Geocoding Query from Nominatim? Take a look:

{
  "place_id": 159358569,
  "licence": "Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
  "osm_type": "way",
  "osm_id": 286831995,
  "lat": "-9.949531163248126",
  "lon": "-78.22368982770104",
  "display_name": "Culebras, Province of Huarmey, Ancash, Peru",
  "address": {
    "village": "Culebras",
    "region": "Province of Huarmey",
    "state": "Ancash",
    "country": "Peru",
    "country_code": "pe"
  },
  "boundingbox": [
    "-9.9496929",
    "-9.9483904",
    "-78.225981",
    "-78.2215248"
  ]
}

Ok, This seems to be all I need!…
Nominatim is my great solution! well, let’s get to work.

4- Loading taxonomy terms

Well, I feel that throughout the previous sections we have already laid some firm foundations. Well, I feel that throughout the previous sections we have already laid some firm foundations. Now I want to recover my initial goal: to load taxonomy terms. For this I will use Nominatim, but as the local container addresses are not queryable from this article, I will change the URLs to external queries so you can try.

I need to complete the next field:

Getting data for Country

In Peru there are 24 departments, 25 regions, 196 provinces and 1838 districts. I need to save Department, Province and Districts for every item. In the former example I will catch the data from the Nominatim reverse endpoint, where Department=”Ancash” , Province=”Huarmey” District=”Culebras”, so based on the data received from Nominatim that you can see in a former example, I need to adapt this:

Department=state
Province=region
District=village

Mounting the new container

Ok, now I’m gonna launch the Nominatim container using my local Docker Installation and the selected URLs with data from my required country (Peru):

$ docker run -it --rm \
  -e PBF_URL=https://download.geofabrik.de/south-america/peru-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/south-america/peru-updates/ \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:3.7

This new container will expose the API address at:

http://localhost:8080/search.php?q=KEYWORD

Resolving queries

Now I can build some queries to my new Nominatim Server from my PHP code (in a Drupal custom module, like a service or from a Controller, you know):

<?php

[...]
use GuzzleHttp\Client as GuzzleClient;
use Http\Adapter\Guzzle6\Client as GuzzleAdapter;
use GuzzleHttp\Psr7\Request as GuzzleRequest;
[...]

[...]
 // We are getting the values for geofield.
 // 'lat' => $latitude_data, 'lng' => $longitude_data.

$config = ['timeout' => 10];
$httpNominatimClient = new GuzzleClient($config);
$adapter = new GuzzleAdapter($httpNominatimClient);
$request = new GuzzleRequest('GET', 'https://nominatim.openstreetmap.org/reverse?format=json&lat=' . $latitude_data . '&lon=' . $longitude_data);
$response = $adapter->sendRequest($request);
$geo_nominatim = json_decode($response->getBody(), true);
[...]

And I can extract the required values following the returned format in the json response, putting some control in the case of the value for “District” (sometimes is a City, sometimes is a Village):

// First get the initial strings. 
$initial_department = $geo_nominatim['address']['state'];
$initial_province = $geo_nominatim['address']['region'];
if(empty($geo_nominatim['address']['village'])){
  $initial_district = $geo_nominatim['address']['city'];
} else {
  $initial_district = $geo_nominatim['address']['village'];
}

My next problem: the ‘Ñ’ character and the diacritics

The first thing I notice is that Nominatim is returning me some naming without accents or diacritics in some cases, in pure English. Let’s see an example about this. I’m looking for data (lat, lon) for “Huánuco” (Perú) and Nominatim is returning me something like this:

{
  "city":"Chinchao",
  "region":"Province of Huánuco",
  "state":"Huánuco",
  "postcode":"062",
  "country":"Peru",
  "country_code":"pe"
}

As you can see, “Huánuco” is comming with diacritics but the country name “Peru” is returned without accent. Can this cause me problems in the future? So now having observed this issue, I now have to check that there are no misinterpretations, normalizing all the strings on both sides of the matching.

The ‘Ñ’ character is the sign for a very common sound that you can build using ‘gn’ in French or Italian, and using ‘ny’ in Catalan language. But at some point an old transcriber monk decided to use this symbol, and here we are…

So, I normalize some values but maintaining the ‘ñ’ character in names, just in ‘San Vicente de Cañete’.

How Can Normalize string but maintaining special characters like ‘ñ’? well, I was thinking about changes the character, normalize and then return the ‘ñ’ character to the strings, so I can try something like this:

// Changes the 'ñ' character.
$first_step_replace_1 = str_replace('ñ', '\001', $initial_department);
$first_step_replace_2 = str_replace('ñ', '\001', $initial_province);
$first_step_replace_3 = str_replace('ñ', '\001', $initial_district);

// Normalizes all the strings.
$normalized_department = preg_replace('/[\x{0300}-\x{036f}]/u', "", Normalizer::normalize($first_step_replace_1 , Normalizer::FORM_D));
$normalized_province = preg_replace('/[\x{0300}-\x{036f}]/u', "", Normalizer::normalize($first_step_replace_2 , Normalizer::FORM_D));
$normalized_district = preg_replace('/[\x{0300}-\x{036f}]/u', "", Normalizer::normalize($first_step_replace_3 , Normalizer::FORM_D));

// Returns the 'ñ' character to the strings.
$normalized_department = str_replace('\001', 'ñ', $normalized_department);
$normalized_province = str_replace('\001', 'ñ', $normalized_province);
$normalized_district = str_replace('\001', 'ñ', $normalized_district);

Now I can make sure that everything is normalized, flattened without diacritics and maintaining the ‘ñ’ character.

Loading the whole taxonomy

The next step for me is to make sure that I can execute a matching between the received values and the existing ones from my vocabulary. In addition, I must take into account that the same name can be repeated throughout the tree structure: A province and a district may share a name, and it may even be the case that the same name is shared at all three levels of the hierarchy of terms, for instance, see the next example where “Arequipa” is a Department, a Province and a District:

So I have to do some thinking to discern how to know that a specific name is part of the first, second or third level within the taxonomy. For the moment to start with, I load the whole taxonomy, all its terms, using the loadTree() method. Although it is a heavy process to load all the complete entities (TRUE parameter), later I can change this for a “light” load of only name and numeric tid.

[...]
// Load the whole taxonomy tree using values.
$manager = \Drupal::entityTypeManager()->getStorage('taxonomy_term');
$taxonomy_tree = $manager->loadTree(
        'location', // The taxonomy term vocabulary machine name.
        0,               // The "tid" of parent using "0" to get all.
        3,               // Get all available level.
        TRUE             // Get full load of taxonomy term entity.
        );
[...]

Ok, now I’m in a loop going through the values taken from Nominatim, point by point. I need to find the term matching from traversing the taxonomy values and know in which position (by level) I should load the geolocated point. To do this I’m planning to ask each term if it has children or if it has parents, since (I think):

If it has no parents and it has children, it will be first level.
If it has parents and has children, it will be second level.
If it has parents and has no children, it will be third level.

But as first step:

[...]
// Prepares the future array of taxonomy terms to load.
$array_of_tids = [];
[...]

Now I’m gonna loop over all the taxonomy terms in a Normalized way:

[...]
foreach ($taxonomy_tree as $term) {
  $first_step_naming = str_replace('ñ', '\001', $term->getName());
  $second_step_naming = preg_replace('/[\x{0300}-\x{036f}]/u', "", Normalizer::normalize( $first_step_naming, Normalizer::FORM_D));
  $normalized_naming = str_replace('\001', 'ñ', $second_step_naming);
[...]
}

I can play with terms using conditions and testing parents and childrens. I can use the loadChildren() method and the loadParents() method passing by its tid for every term.

// If has not parent and has children is a first level term. 
if((empty($manager->loadParents($term->id()))) && (!empty($manager->loadChildren($term->id()))) && (($normalized_naming === $normalized_department))) {
  $array_of_tids[0] = $term->id();
}

// If has parent and children then is a second level term.
if((!empty($manager->loadParents($term->id()))) && (!empty($manager->loadChildren($term->id()))) && (($normalized_naming === $normalized_province))) {
  $array_of_tids[1] = $term->id();
}

// If has parent and has not children then is a third level term.
if((!empty($manager->loadParents($term->id()))) && (empty($manager->loadChildren($term->id()))) && (($normalized_naming === $normalized_district))) {
  $array_of_tids[2] = $term->id();
  $parent = reset($manager->loadParents($term->id()));
  $array_of_tids[1] = $parent->id();
}

Also I can save the terms in hierarchy using only the final term from the third level, completing the tree from bottom to top, each time I identify a third-level term:

// If has parent and has not children then is a third level term.
if((!empty($manager->loadParents($term->id()))) && (empty($manager->loadChildren($term->id()))) && (($normalized_naming === $normalized_district))) {

  // Third level Term.
  $array_of_tids[2] = $term->id();

  // Second level Term.
  $parent = reset($manager->loadParents($term->id()));
  $array_of_tids[1] = $parent->id();
  
  // First level Term.
  $grandparent = reset($manager->loadParents($parent->id());
  $array_of_tids[0] = $grandparent->id();
}

My goal is to finally have an array composed of three numeric values that will be the tids of the terms found. This array can be saved in the Entity Reference field for the taxonomy and finally when the node is saved, these taxonomy terms will be charged in every node.

$new_node->field_taxonomy_location = $array_of_tids;
$new_node->save();

Done!

5- Read More

6- :wq!

Recommended song: Addio Lugano bella

Giorgio Gaber, Enzo Jannacci, Lino Toffolo, Otello Profazio and Silverio Pisu.

PHP Coding: functions that can save your day

2021-05-30T00:00:00+00:00


Picture from Unsplash, user Umberto, @umby

Well, this month I’ve been doing some things I haven’t practiced for some time: data migrations, intensive connections to Gitlab API v4 and even some web-scraping to get data. Along the way I’ve recovered some old PHP functions that I had forgotten or hadn’t used for a long time. As I didn’t want to forget them again, I took this notebook as a reference and wrote down a small post with some functions of our old friend PHP that can give us a little help in the day to day.

The old man PHP has over 1000 built-in functions that can be called and used directly from your code and will be independient of the platform/CMS/framework (Drupal, Symfony, Laravel…) so you can load the functions in your scripts or classes in order to execute specific tasks. I’ve given a little context to each one and possible uses. You probably already know them or they seem very obvious to you, but I thought it was fun to share them. I’m sure someone might find them useful.

Note: This month (may 2021) I’ve published an article with a review about the book ‘31 Days of Drupal Migrations’ in The Russian Lullaby. This could be interesting for you.

1- basename()

I really don’t know why I had forgotten this function of PHP…but the fact is that I had been processing URLs for some time and cutting them to get paths, directory names and final files.

One day I remembered it again and there it was! basename() is a function that returns the final item in a path and if you add a suffix (maybe you only need the name of a file, not its extension), the function will delete the .extension and will give you just a name.

Examples

<?php

// Do your things.
[...]
// Trying to get the final component in url.
$cut_url = substr($url, 0, -6);
// Same as.
$cut_url = basename($url);

Remember

Function / Method: basename() returns the final element of a path.
Using: basename($path); basename($path, $suffix);
Available: PHP4, PHP5, PHP7, PHP8.
More Info: php.net/basename

2- getElementsByTagName()

Ok, well I was doing some data extractions from web scraping and suddenly remembered the existence of the DOMDocument class in PHP, a set of available functions for getting a HTML document and parsing / extracting diverse information from the DOM object.

This function (really is a method, due to live in a class context): getElementsByTagName() gives you a set of results gettings values only from reffered tags in the processed DOM. For instance, I can do something like:

Examples

<?php
// Create a new DOM Document to get the HTML remote content.
$html_code = new DOMDocument();

// Load the url's contents into the DOM item.
$html_code->loadHTMLFile($url); 

But in my Drupal context will be just like:

<?php
[...]
// Third: Gets a whole post from original website.
$post = file_get_contents($domain_link_post);
$dom = Html::load($post);

It’s the same operation, and then we’ll get specific values:

<?php

// Do your things.
[...]

// Fourth: Gets all the interesting values for the new node. 
// Gets title and subtitle from DOM.
$block_title = $dom->getElementById('title');
$title = $block_title->getElementsByTagName('h1')->item(0)->textContent;
$alias = $block_title->getElementsByTagName('h3')->item(0)->textContent;

In the former codeblock I was extracting some basic data for load in a new Drupal node, strings like the title of the node and information to fill in a field called “alias”. My parsed DOM document is well known to me, this is very important for implementing the correct parsing: I must know well where are the data I am interested in.
In fact Drupal is using this DOMDocument class under the hood through a wrapper, the Html class available from:

/core/lib/Drupal/Component/Utility/Html.php

Inside this class we’re doing an extensive use of the DOMDocument class. Just in the load() main function you can see, for instance:

<?php
[...]
$dom = new \DOMDocument();
// Ignore warnings during HTML soup loading.
@$dom->loadHTML($document);

return $dom;

Remember

Function / Method: DOMDocument::getElementsByTagName(), search all elements from a tag name.
Using: $dom->getElementsByTagName(‘li’); $dom->getElementsByTagName(‘p’); Available: PHP5, PHP7, PHP8.
More Info: php.net/getelementsbytagname

Read More:

3- glob()

What is glob? (baby don’t hurt me… xD) Well the glob() function will return naming of files in a certain path, and I was using the function for treating files and processing values, or delete existing files from a folder.

Examples

Here deleting files by unlink function.

<?php
[...]
$initial_files = glob('sites/default/files/my_folder/*');
// Deleting all the files in the list
foreach($initial_files as $file) {
  if(is_file($file)){
    // Delete the given file
    unlink($file); 
  }
}

And here looping over the files for processing some specific values.

<?php
[...]
// Get all the existing files.
$geodata_files = glob('modules/custom/my_custom_module/geodata_files/*');

// Now we are going to loop over the existing files for doing stuff.
foreach($geodata_files as $geodata_file) {

  // Doing your things.
  [...]
}

Remember

Function / Method: glob(), find pathnames matching a pattern.
Using: count(glob(‘folder/files/’)); $files = glob(‘folder/files/’);
Available: PHP4, PHP5, PHP7, PHP8.
More Info: Glob function of PHP

4- curl_init()

Curl is a function library supported by PHP that allows HTTP request and also is available as command line tool for prompt. In that context I’ve been using it for a while now, but only from the Linux console… The reason? Well, for almost all the code related to REST clients from Drupal I’ve implemented it using Guzzle (the PHP reference library for REST clients and integrated in Drupal) or indirectly using the HTTP Client Manager, a resource that allows you manage HTTP Clients from YAML description files -or JSON- (and is using Guzzle under the hood, by the way).

But the last week I had to implement a small external query from my code to the Google Places API and I thought I’d try curl directly from PHP. This was my case.

Examples

<?php

[...]

// Enables a new cURL resource.
$ch = curl_init();
$url_map = "https://maps.googleapis.com/maps/api/place/details/json?cid=" . $required_cid_code . "&key=" . $this->myConfig->get('google_places_api_key');
curl_setopt($ch, CURLOPT_URL, $url_map);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
curl_setopt($ch, CURLOPT_HEADER, 0);

// Get the geo file from google maps API for places.
$result_value = curl_exec($ch);
$converted_result = json_decode($result_value, true);
curl_close($ch);

[...]

curl is available from your server installation as a PHP resource, in my case the web container of my DDEV local deploy, and only requires initialize a session, set your options for the work, then execute the session and finally close it.

Remember

Function / Method: curl_init(), initializes a new session and return a cURL handle for use.
Using: $ch = curl_init(); $ch = curl_init(“http://www.externaldomain.com/”);
Available: PHP4, PHP5, PHP7, PHP8.
More Info: The PHP’s book of curl
Read More:

5- htmlspecialchars_decode()

I was extracting values from an external web source of HTML, real numbers, but in some cases when loading them in target and then rendering my nodes, some of these fields were showing character encoding using HTML entities, changing a single quote or apostrophe: ' by its HTML encoding: ' (you can see equivalences here). So in a received and then loaded value for a field, instead of get some like from the external source:

Price marked in Soles
From: S/597,604 To: S/1'703,183

I was setting in destiny:

Price marked in Soles
From: S/597,604 To: S/1&#39;703,183

So I needed a way to decode that parameter and set its original format. And there’re a pair of interesting functions in PHP to execute something like this:

htmlspecialchars_decode(): Convert HTML entities to special characters.
htmlspecialchars(): Convert special characters to HTML entities.

My case is exactly the first option, I need to change the HTML codification and save it and I can use a specific flag to mark this required behavior: ENT_QUOTES.

Examples

$value_soles = htmlspecialchars_decode($soles_value_string, ENT_QUOTES);
$value_dolars = htmlspecialchars_decode($dolars_value_string, ENT_QUOTES);

Remember

Function / Method: htmlspecialchars_decode(), convert special HTML entities to characters.
Using: htmlspecialchars_decode($original_string, ENT_QUOTES); htmlspecialchars_decode($original_string);
Available: PHP5, PHP7, PHP8.
More Info: php.net/htmlspecialchars-decode.php
Read More:

:wq!

Recommended song: Quemas - Ede & Xoel López

Drupal Tips: Link fields from Twig

2021-04-30T00:00:00+00:00


Picture from Unsplash, user Michael Dziedzic, @lazycreekimages

In this new issue of the Drupal Fast Tips I would like to share some basic tips and examples related with the Drupal Link field and how to get the data from its sub-fields (title and link) to rendering in a Twig template. Sometimes we need render values from a link field in a structured way through a Twig template, and depending on the location of the Link field, this may have a different articulation.

Because of this, I wanted to show some small examples of extracting and handling the values of this particular type of fields. This will be a post for site-builders or developers with basic knowledge on the Drupal backend (yes, Twig and the rendering are in the backend, sorry.).

Note: This month (april 2021) I’ve published an article about Visibility Condition Plugins for Drupal in The Russian Lullaby. This could be interesting for you.

This article is part of a series of posts about Drupal Tips.

1- Drupal Fast Tips (I) - Using links in Drupal 8
2- Drupal Fast Tips (II) - Prefilling fields in forms
3- Drupal Fast Tips (III) - The Magic of ‘#attached’
4- Drupal Fast Tips (IV) - Xdebug, DDEV and Postman
5- Drupal Fast Tips (V) - Placing a block by code
6- Drupal Fast Tips (VI) - From Arrays to HTML
7- Drupal Fast Tips (VII) - Link Fields from Twig

Using a Link Field

Only starts when you add a new Link field to your content, directly or from a Paragraph using this field. Now you have two sub-fields (title and link) and some config options.

It’s also quite possible that you have added the contrib module ‘Link Attributes’ for adding some extra widgets to the Link field. For instance, this is a very common configuration for saving values in target attribute. After passing by /admin/structure/types/manage/article/form-display and select link with attributes for your Link field display, now you have got something like this:

So you’re building links to render with two subfields and one related attribute.

Building the link with separate items

When you’re getting the link field values from a view and you’ve selected as formatter: separate link element and text in config. First check if title exists and render it. If not, only builds the link data:

  <div class="link-item">
    {% if title %}
      <div class="link-title">{{ title }}</div>
    {% endif %}`
    <div class="link-url">{{ link }}</div>
  </div>

This is basic structure for showing the link from Twig if you have separate values for link and title, and you aren’t returning the values mounting in a single object.

Getting value of the target attribute from a Link Field in Paragraph

Sometimes you can use the Link Field within a Paragraph and then you need get the values of the link from a Twig Template. Here the syntax can be more strange, but let’s see how to extract some values from a field in a Paragraph: Link with url, text and target values.

// key -> target="{{ paragraph.field_link_link[0].options.attributes.target }}
<div>
<a href="{{ paragraph.field_link_link.0.url  }}" target="{{ paragraph.field_link_link[0].options.attributes.target }}">
{{ content.field_link_title }}
</a>
</div>

In the snippet from above, you can see that we’re using as title of the link the data from the content, mixing origins.

Processing and building links from a View to a Twig Template

I’m getting some fields from a specific view, passing values from a Link field (text, link), and a I want review if external / internal links and put some attributes from the link text.

The next snippet is a piece of code from a classic hook_preprocess_views_view_fields() included in a *.theme file, where we’re using a control variable registering if the link was external or not. Then we’ll try to reach this variable and take some decisions about the structure in Twig. This case involves three areas of work:

The view itself, when we configure the exit of the fields to the render system (You can overwrite the fields output here).
The theme_name.theme file, with hooks and methods for altering the future render.
The related Twig template, catching the variables from the theme file and processing values.

Let’s see:

  if ($view->storage->id() == 'user_page' && $view->current_display == 'page_5') {

    // Loads and cuts the field resources_link for processing.
    $link = $variables['fields']['field_resources_link']->content->__toString();

    //$link -> "<a href="https://www.google.com/" target="_blank">External Link Example</a>"
    //$link -> "<a href="/blog">Internal Link Example</a>

    $sliced_link = explode ('"', $link);
    $url_link = $sliced_link[1];

    // Check if external link.
    if(isset($link) && $link!=""  && str_contains($link, 'target="_blank"')){
      $variables['external'] = true;
      $url_title = explode( "<", substr($sliced_link[4], 1))[0];
    } else if (isset($link) && $link!=""  && !str_contains($link, 'target="_blank"')) {
      // Check if internal link.
      $variables['external'] = false;
      $url_title = explode( "<", substr($sliced_link[2], 1))[0];
    }
    
    // Loads the final values for twig template.
    $variables['url_link'] = $url_link;
    $variables['url_title'] = $url_title;
  }

Then mounting the received values in twig:

<div class="row-news">
	<div class="row-news-header {% if external %} resources-link {% endif %}">
	  <a href="{{ url_link }}" alt={{ url_title }} title= {{ url_title }} 
        {% if external %} 
          target="_blank" 
        {% endif %}> 
        {{ fields.title.content }} 
      </a>
	</div>
	{{fields.field_resource_description.content}}
</div>

:wq!

Recommended song: Catalina - Rosalía & Refree

Linux: 70 commands aliases for everyday life

2021-03-29T00:00:00+00:00


Picture from Unsplash, user JJ Ying

Following on from my previous post about “200 Linux commands for everyday life” that I published a few months ago -September 2020, here-, so today I want to share another small article about little everyday utilities. The previous post had a more extensive (and intense) feedback than when I’m writing about Drupal, so this has invited me to think that maybe (just maybe) it could be useful content for someone.

This time, I have chosen to gather my favourite console “aliases” in one post. Aliases are a very dynamic way to add power to the command terminal and also a way to improve your fluency in the Linux terminal and therefore, your own productivity when working on projects. They are really just an exercise in replacing commands that you use frequently with a faster and simpler version of them, so that you can type them faster and more intuitively (especially if they include options and parameters). This way you can type less to request the same long instructions.

Do you often type the same command with similar parameters? make it a console “alias”!…well, you can also create “functions” to do the same but in a much more dynamic and adaptive way, but I’ve left that for another article later on. Today: aliases!.

Introduction

Just go to your /home/user folder and edit your .bashrc file in order to include these aliases. Then reload the bashrc file and you’ll get the new aliases available. You can add the new aliases in a new file too, using a file named .bash_aliases and then load the new file from your original .bashrc file, just by adding the reference in order to set the aliases:

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

if [ -f ~/.bash_functions ]; then
    . ~/.bash_functions
fi

For example, if you wanna use reduced formats for usual instrucions, you can do something like this:

alias yep='sudo apt install $1'
alias nop='sudo apt remove $1'

And so you can type by prompt using the new aliases:

$ yep nmap
$ nop nmap

Remember:

**:~$** cd ~    
**:~$** vim .bashrc // :wq! (after paste the content)      
**:~$** source .bashrc   

You can download the whole file .bash_aliases fullfilled with all the aliases from here like a Github Gist.

Morning Opertures

alias whatsup='service --status-all'  
alias hello='sudo /etc/init.d/apache2 stop && cd workspace/project && ddev start && ddev launch'   
alias hi='sudo systemctl stop apache2'  
alias ports='nmap localhost'  
   **Are my ports open?**
alias dens="sudo systemd-resolve --status | grep 'DNS Servers'"
   **What is my current DNS Server?**
alias iad='systemctl is-active docker'  
alias bye='shutdown -r now'  

Usual Instructions

alias c='clear'  
alias h='history'  
alias hg='history | grep $1'  
    **Search using history command: $ hg somestring.**  
 alias wg='wget -c '  
alias al="echo ------------Your curent aliases are:------------¡';alias"  
    **Returns all your aliases collection.**
alias sup="sudo apt update && sudo apt upgrade -y"  

Content in folders

Getting info from a position in a folder.

14. alias ll='ls -la'  
    **Debian doesnt have the -ll- alias created by default, like Ubuntu.**    
15. alias lf='ls -alF'  
16. alias la='ls -A'  
17. alias ls='ls -CF'  
18. alias lt='ls --human-readable --size -1 -S --classify'  
    **Sort resources by _file_ size.**   
19. alias lu='du -sh * | sort -h'  
    **Same as former but counting size from folders too.**   
20. alias lt='ls -t -1 -long'  
    **Gets an ordered list of files and folders by changes in time.**   
21. alias lc='find . -type f | wc -l'  
    **Gets a total recursirve count of existing files, no folders.**   
22. alias ld='ls -d */'  
    **Gets a list of existing directories from current folder.**   

Info from Drupal: config files, types of content and existing paragraphs

alias lsc='ls -lah config/sync/ | wc -l'   
alias lsn='ls -lah config/sync/node.type.* | wc -l'    
alias lsp='ls -lah config/sync/paragraphs.paragraphs_type.* | wc -l'  

Files, folders and resources

alias fh='find . -name '   
alias ..='cd ..'  
alias ....='cd ../..' 

More Jump Down

alias 1d="cd .."  
alias 2d="cd ..;cd .."  
alias 3d="cd ..;cd ..;cd .."  
alias 4d="cd ..;cd ..;cd ..;cd .."  
alias 5d="cd ..;cd ..;cd ..;cd ..;cd .."  
alias untar='tar -zxvf $1'  
    **Open a tar.gz formated folder.**  
alias tar='tar -czvf $1'  
    **Compressing a folder in tar.gz format.**  
alias mnt="mount | awk -F' ' '{ printf \"%s\t%s\n\",\$1,\$3; }' | column -t | egrep ^/dev/ | sort"  
alias df="df -Tha --total"   
alias exp='nautilus .'  
    **Opens the current directory in file explorer.**  

Basic info

alias gs='git status'  
alias gb='git branch'  
alias gr='git remote -v'  

Getting info from ‘Git log’

42. alias gl='git log --oneline'  
    **Gets a log info view in a single line format.**    
43. alias glc="git log --format=format: --name-only --since=12.month | egrep -v '^$' | sort | uniq -c  | sort -nr | head -50"  
    **Get a list with the most changed files from 12 months ago.**  
44. alias gld='git log –oneline –decorate –graph –all'  
    **Show all the branches in the tree format history with pointers.**  
45. alias glp="git log -g --grep='PHP' -10 --pretty='%h - %s - %cn - %cd'"  
    **Gets info from log filtering by some fixed key in commit messages.**  
    **I am using some keys in message commits, [just like these.](https://gitlab.com/-/snippets/2096890)**  
46. alias glf='git for-each-ref --sort=-committerdate'   
    **Get commits list by DESC order.**

Pushing to basic branches

47. alias gpom='git push origin master'  
48. alias gpod='git push origin develop'  

Drush Commands

I’m using DDEV-Local for my Drupal local deploys.

alias cex='ddev drush cex'  
alias cim='ddev drush cim'  
alias cexy='yes|ddev drush cex'  
alias cimy='yes|ddev drush cim'  
alias dgm='ddev drush generate module'  
alias dws='ddev drush watchdog:show –count=20'  
    **Shows the last 20 problems in your Drupal installation.**  
alias ddc='ddev drush cr'  
alias dpc='ddev drush pmu $@'
alias dec='ddev drush en -y $@'
alias dpe='ddev drush pmu $1 && ddev drush cr && ddev drush en -y $1'
alias dva='ddev drush views:analyze'
alias dpl='ddev drush pml'
alias dle="ddev drush pm-list --status='enabled'"
alias dld="ddev drush pm-list --status='disabled'"
alias dlo='ddev drush pm-list --type=module --status=enabled --no-core'

DDEV Explicit Aliases

alias dsl='ddev start && ddev launch'   
alias ddy='ddev delete -Oy'  
    **Deletes the DDEV folder and resources in the current project folder.**  
alias did='ddev import-db'   
alias ddl='ddev list'  

Reviewing Codestyle Aliases

alias dcs1="ddev exec -d=/var/www/html vendor/bin/phpcs --standard=Drupal --extensions='php,module,inc,install,test,profile,theme,info,txt,md' web/modules/custom/"  
alias dcs2="ddev exec -d=/var/www/html vendor/bin/phpcs --standard=DrupalPractice --extensions='php,module,inc,install,test,profile,theme,info,txt,md' web/modules/custom/"  
alias dcsr='ddev exec -d=/var/www/html vendor/bin/phpcs –report=diff'  
    **With /path/to/code, Will generate a patch file from a style diff.**

davidjguru.github.io

Drupal Tips: Avoid infinite loop in user login

Acknowledgments

Checking the node / entity access

The Issue

1. The way I’m getting the taxonomy terms values

2. The way I’m executing redirection

Read More

:wq!

Recommended song: Qué he sacado con quererte AKA Violeta Parra - Guadalupe Plata

Tooling: Python2 error in SASS building frontend

Acknowledgments

The Issue

Fixin’

Warning

Read More

:wq!

Recommended song: In Spain we call it Soledad - Rigoberta Bandini

Drupal Tips: Changing session lifetime for users

Acknowledgments

Goals and basic concepts

GC_MAXLIFETIME

GARBAGE COLLECTOR (GC values)

COOKIE_LIFETIME

Addressing session cookie management in Drupal

Overwriting default values by switching service class

Session data & Garbage Collector & Cron

Useful Modules

Read More

:wq!

Recommended song: I Feel So Good - Joanna Connor

Decrease noise, gain focus: Reduce your stress level

First: Simplify your daily life

Second: Purify, purify, purify always (debugging)

Third: Learn how to say ‘No’

Conclusion

Read More:

:wq!

Recommended song: Tarantella del Gargano - Daniele Sepe

Drupal Tips: Altering order of execution in resources

Acknowledgments

1- Altering order in submit functions

Use Case

2- Altering the priority order of modules

3- How to change the registry of modules using a Hook

4- How to change the registry of modules using a contrib module

5- :wq!

Recommended song: Lazy Bones - Wooden Shjips

Drupal Techniques: How to upgrade Drupal

Acknowledgments

1- Setting the scenario

2- Sorting the toolbox

2.1- upgrade_status

2.2- php-stan

2.3- drupal-check

2.4- drupal-rector

2.5- backward_compatibility

3- Assesing the status of your project

3.1- Taking a snapshot of your platform

Some Steps:

3.2- Containers and Backups

3.3- Current version for Drupal Core

3.4- Upgrade Status

3.5- Drupal-Checker and Drupal-Rector

3.6- Get a list of deprecated resources

4- Preparing the work

5- Executing the upgrade

5.1- Remove some resources

5.2- Set your new Drupal Core version

5.3- Delete modules with problems

5.4- Upgrade to latest Drupal 9 version

5.5- Update patches

5.6- Reinstall modules with Issues

5.7- Run updates in database

5.8- Review all your custom work

5.9- Checking new installed versions

5.10- Remove tooling for upgrade

5.11- Commit files with changes

6- Troubleshooting

6.1- GraphQL is giving me some errors