Common development workflow

When developing new changes in Git it is recommended to use a development workflow that allows for the changes to be developed, tested, without impacting the primary project. Once everything has been tested and approved those changes can be merged in to the primary branch of the project. For bigger, more complex projects, custom modules and themes will normally be in their own repository, and pulled in via the main project's composer.json.

For smaller simpler projects, you may be able to not use branching, with configuration, custom modules, and themes, etc. in a single repo.

While you can create GitLab MR's via the user interface, if there are  many changes, using Git to create an MR for Drupal contrib module or Drupal core can be most practical.

Very simple workflow for small projects

If you are the only one working on a small project, and looking for a simple workflow, you might not need to use branching. Just clone the remote repo. Make your changes. Commit your changes. Push your commits to the remote repo.

To make things even simpler, you could consider housing custom modules and themes in a single unified project repo, and not in separate repos.

An alternate strategy for custom modules is to commit them directly to the repository of the site where they are used. This is commonly done with modules that are specific to just one site.

From .gitignore patch in #3082958: Add gitignore(s) to Composer-ready project templates.

Simple method to create Drupal GitLab MR

It can seem difficult to create a Merge Request (MR) in GitLab with Git for a Drupal core or contrib module, breaking it down into the minimum required steps, using HTTPS token authentication can help. The example is based on #3256623: Add upgrade path for migrations from 7.x-1.x to Drupal 10.

1. In the issue, click "Create issue fork", if it doesn't already exist and check it says "✓ You have push access".
2. Clone the project and enter the folder:
   

   $ git clone https://git.drupalcode.org/project/login_security
   $ cd login_security/

3. Now set the remote, download it, and switch to that branch. These commands can be found under "Show commands":
   

   $ git remote add login_security-3256623 https://git.drupalcode.org/issue/login_security-3256623.git
   $ git fetch login_security-3256623
   $ git checkout -b '3256623-add-upgrade-path' --track login_security-3256623/'3256623-add-upgrade-path'

4. Create a GitLab token, see Git Authentication for Drupal.org Projects > Authenticating for pushing over HTTPS.
5. Do your code updates, and send them to the branch on GitLab. Use your drupal.org user name (replace my_username) and the GitLab token as password:
   

   $ git add .
   $ git commit -m "Description of the changes"
   $ git push
   Username for 'https://git.drupalcode.org': my_username
   Password for 'https://my_username@git.drupalcode.org':
   Enumerating objects: 28, done.
   [...]
   

Congratulations! If all went well, the code is now added in the GitLab branch, and you can create a Merge Request.

Common development workflow

Describes the steps in a Git managed Drupal project workflow, and the commands.

1. Create a new feature branch from master

The first step in this process is to branch off of the primary branch so you have a place to work on the changes about to be developed.

git checkout -b feature/[your-new-branch]
# Make you code changes. Do not forget to export new configuration.
git add -A # Stages new, modified and deleted
git commit -m "Some message"
git push -u origin feature/[your-new-branch] # First push only, next just 'git push'

• How to Write a Git Commit Message

2. Commands to merge master branch into a feature branch

Now that a new branch has been created work can now be pushed up to the branch. Most features take a while to develop, test, and approve. This often means that the primary branch has been updated be the time the new feature is ready. Before merging the feature into the primary branch you will want to be sure that those new changes are added to your branch to reduce the chance of merge conflicts.

Using the above-mentioned tools Composer, Git and Drush, what terminal commands would you use when we are on an existing feature branch and want to update it with the latest master changed on the remote repository?

# In local development environment:
drush config:export # Export new configuration while on a feature branch
git add -A # Stages new, modified and deleted
git commit -m "Some message"
git checkout master # Go to the master branch
git pull # Get the latest code base
git checkout feature/[my-existing-branch] # Go back to feature branch
git merge master # If you get any merge conflicts, see next paragraph
git push # Include latest master commits in remote feature branch

# On remote staging (or production/live) server:
drush state:set system.maintenance_mode 1 # Switch to maintenance mode
# Pull the code(configuration) from git, if it has not been put on the server automatically
composer install # Get any new dependencies
drush updatedb -y # Run the update scripts
drush config:import # Import new configuration
drush cache:rebuild # Clear all caches
drush state:set system.maintenance_mode 0 # Put the website back online

Note that we merge into our feature branch before we do a configuration import to avoid the loss of new configuration added in the feature branch such as new blocks.

Also, updb should always be run before cim, if you need to update entities after that, create a hook_post_update_NAME and use this alternative workflow:

drush updatedb --no-post-updates -y 
drush config:import -y
drush updatedb -y

Since 10.3 Drush provides a new deploy command that wrap it up:

drush deploy -y

 ⚠ Beware, it changes the previous workflow order, hook_post_update_NAME is fired after drush updatedb and a new hook HOOK_deploy_NAME() is fired after drush config:import, here's the details:

drush updatedb --no-cache-clear   # HOOK_update_N + HOOK_post_update_NAME
drush cache:rebuild
drush config:import
drush cache:rebuild
drush deploy:hook                 # HOOK_deploy_NAME()

How to deal with merge conflicts

Sometimes you may still run into merge conflicts. Use this guide to help with resolving those conflicts.

• Resolving Conflicts | Git How To - Guided Git Tutorial

Allow roll-back: Tie SQL dumps to commit hashes

We've all been there, the database is corrupt or there were errors validating the config synchronization. We want to do a rollback to a certain commit hash but you realize you don't have the database dump that goes along with it. Using an older version would mean you probably lose content.

To avoid this situation regularly make a database dump (locally and/or on your staging environment). Preferably before each merge (or pull) and with the commit hash inside the dump file name to know which state of the codebase it goes with. Now you are able to always restore your site based on a commit hash.

Follow these steps:

GIT ignore SQL dumps

Add to the .gitignore file:

# Ignore the default SQL database dumps
*.sql
*.sql.gz
/db-backups/

Define common tables to skip the data for (not the structure)

Taken from https://github.com/drush-ops/drush/blob/master/examples/example.drush.yml.

Add to the drush.yml file (usually at drush/drush.yml):

sql:
  # List of tables whose *data* is skipped by the 'sql-dump' and 'sql-sync'
  # commands when the "--structure-tables-key=common" option is provided.
  # You may add specific tables to the existing array or add a new element.
  structure-tables:
    common:
      - cache
      - 'cache_*'
      - history
      - 'sessions'
      - 'watchdog'

Optionally you can apply it to any Drush SQL dump by adding:

# This section is for setting command-specific options.
command:
  sql:
    dump:
      options:
        # Omit cache and similar tables (including during a sql:sync).
        structure-tables-key: common

Make a "smart" dump

Make a dump in a dedicated dump folder in the project root (must contain a .git folder) with a traceable file name:

mkdir -p `git rev-parse --show-toplevel`/db-backups; drush sql-dump --structure-tables-key=common --result-file=`git rev-parse --show-toplevel`/db-backups/$(date +%Y%m%d\T%H%M%S-)`git branch | grep \* | cut -d ' ' -f2 | sed -e 's/[^A-Za-z0-9._-]/_/g'`-`git rev-parse HEAD | cut -c 1-8`.sql

• creates a folder db-backups in the project root (mkdir will be ignored if the folder already exists)
• using a date and timestamp in the name of the dump file so they will get listed in chronological order by default
• appended with the current branch (sanitized to be used within the file name)
• appended with an 8 character commit hash (like used by GitLab) to know what state of the codebase it "belongs" to.

resulting in for example 20180726T112941-develop-07b7ff12.sql .

Before first use, in the project root do a git init to create a Git repository. Running git init in an existing repository is safe. It will not overwrite things that are already there. 

Or, if you want to GZIP:

mkdir -p `git rev-parse --show-toplevel`/db-backups; drush sql-dump --structure-tables-key=common --gzip --result-file=`git rev-parse --show-toplevel`/db-backups/$(date +%Y%m%d\T%H%M%S-)`git branch | grep \* | cut -d ' ' -f2 | sed -e 's/[^A-Za-z0-9._-]/_/g'`-`git rev-parse HEAD | cut -c 1-8`.sql

Remember that the date and time in the filename represent the moment that the dump was created, just to keep them in the right order. That is not necessarily the date and time of the commit.

In the next chapter, we are going to automate the generation of the above dump on each git pull from the repo (before we import the configuration into the database).

How to perform a rollback?

Rollback the codebase first:

Look into the filename of your last database dump to get the branch and commit hash of the code base state that "belongs" to it and follow the instructions in the link and subsequent Drush commands mentioned below.

git checkout - How to revert a Git repository to a previous commit - Stack Overflow

If you can not find the commit probably you are on the wrong branch. Finding what branch a git commit came from - Stack Overflow, use for example: git branch --contains 07b7ff12

Then to do a database rollback:

drush sql-drop -y; drush sql-cli < db-backups/[your-filename].sql

Or, if you have a GZIP:

drush sql-drop -y; gunzip db-backups/[your-filename].sql; drush sql-cli < db-backups/[your-filename].sql

Use aliases (functions) instead

If you want to use the shorter aliases dbdump and dbrestore instead of the long lines above, in your terminal do:

nano ~/.bashrc # Open editor.

# Add to the bottom:
dbdump() {
  mkdir -p "$(git rev-parse --show-toplevel)"/db-backups
  drush sql-dump --structure-tables-key=common --gzip --result-file="$(git rev-parse --show-toplevel)"/db-backups/"$(date +%Y%m%d\T%H%M%S-)""$(git branch | grep \* | cut -d ' ' -f2 | sed -e 's/[^A-Za-z0-9._-]/_/g')"-"$(git rev-parse HEAD | cut -c 1-8)".sql
}

dbrestore() {
  drush sql-drop
  gunzip -k "$(git rev-parse --show-toplevel)"/db-backups/$1.sql.gz
  drush sql-cli < "$(git rev-parse --show-toplevel)"/db-backups/$1.sql
}

^X # Exit and save.
source ~/.bashrc # Make the new functions instantly available.

Note:

• Before first use, in the project root do a git init to create a Git repository. Running git init in an existing repository is safe. It will not overwrite things that are already there. 
• Actually, we used functions instead of aliases.
• For dbrestore provide the file name as an argument (without extensions), thus for example dbrestore 20190608T115648-master-efc2b9e7.
• Both "aliases" can be used from inside any folder in your project, except for other nested git folders.
• If on the server you lack permission to create a folder then remove the mkdir and write to /var/tmp instead:
  

  dbdump() {
    drush sql-dump --structure-tables-key=common --gzip --result-file=/var/tmp/"$(basename `git rev-parse --show-toplevel`)"-"$(date +%Y%m%d\T%H%M%S-)""$(git branch | grep \* | cut -d ' ' -f2 | sed -e 's/[^A-Za-z0-9._-]/_/g')"-"$(git rev-parse HEAD | cut -c 1-8)".sql
  }
  
  dbrestore() {
    drush sql-drop
    gunzip -k /var/tmp/$1.sql.gz
    drush sql-cli < /var/tmp/$1.sql
  }

  The repository name has also been added to the filename to distinguish between multiple projects writing their backups to /var/tmp. 
  Be aware that any backups in /var/tmp remain only for about 30 days.

• @TODO: Make that if no argument is supplied with dbrestore, it will take the latest one corresponding to the current branch.
• To check the above script, it is suggested to use ShellCheck – shell script analysis tool.

Commit a database (during initial development) and the corresponding config export

During initial development sometimes you want to include a database in an 'install' folder in the project root. Especially in this case, it is important the database filename contains the commit hash of the codebase it "belongs to". The 'install' directory contains files needed to set up a working environment, either on a local development environment or a server. It usually consists of a compressed database file and, optionally, an archive of files outside of version control that are needed, for example, images like a site logo.

Installation instructions should still be put in the README.md or INSTALL.md file in the project root, including the location of the needed files mentioned above.

The below script can be added to the .bashrc file. The created command dbcommit will take care of the needed operations. Make sure you run this after the other commits.

dbcommit() {
  drush config:export -y
  git add "$(git rev-parse --show-toplevel)"/config/*
  git commit -m "Configuration export"
  mkdir -p "$(git rev-parse --show-toplevel)"/install
  git rm "$(git rev-parse --show-toplevel)"/install/*.sql.gz
  drush sql-dump --structure-tables-key=common --gzip --result-file="$(git rev-parse --show-toplevel)"/install/"$(date +%Y%m%d\T%H%M%S-)""$(git branch | grep \* | cut$ | cut -d ' ' -f2 | sed -e 's/[^A-Za-z0-9._-]/_/g')"-"$(git rev-parse HEAD | cut -c 1-8)".sql
  git add "$(git rev-parse --show-toplevel)"/install/*.sql.gz
  git commit -m "Updating the corresponding database installation file to the current codebase (check commit hash of the filename)"
}

The commit hash for the database filename is never the last one but the previous one. When rolling back, you can use either one.

An example:

git checkout efc2b9e7
dbrestore 20190608T115648-master-efc2b9e7 #filename without extensions
drush cache:rebuild

Additional: How to import the remote database to your local database

Using Drush sql-dump and sql-sync to sync between multiple environments

Note

The "Common development workflow" chapter was moved from another guide. Here is a placeholder page of the old location.