bffs 4eva: drupal 8 and custom databases - badcamp 4eva- drupal 8 and...bffs 4eva: drupal 8 and...

Kristian Ducharme

BADCamp 2016

BFFs 4eva: Drupal 8 and Custom Databases

Introduction

■ Kristian Ducharme - Senior Engineer / Technical Lead @ CivicActions

■ Past Projects: City of Los Angeles, Acquia Projects

■ Development / Drupal Background

Project Background

https://www.digitaldemocracy.org

■ California Legislative Hearings, ML Transcription, Data Aggregation

■ Custom MySQL Database - 100+ tables (DDDB)

■ DB Team supporting schema, data import, transcription

■ Growth Plans - Multi-state, Tiered offerings, Sites for Advocacy Groups + Organizations, Campaigns, Advanced Search

Drupal / Data Architecture

Current D7 Drupal Architecture

■ DDDB is located on same server, connected via read-only MySQL user

■ Nightly import through migration scripts

■ Data imported into fielded nodes with its own structure, incorporating some similarities to DDDB schema

■ Drupal content types are loosely connected via ID fields, not utilizing entity references

Current D7 Issues

■ Synchronization of data between DDDB and Drupal has to be maintained

■ Duplication of data

■ Keeping DDDB an independent DB important

■ Drupal features limited due to architecture (Views / Content Admin / Search)

■ Schema changes in DDDB require 2-3 layers of Drupal changes / testing

=

DDDB Schema

D7 or D8?

Use Cases for D8 External Custom Entities

■ Complex External Relational Databases

■ Too much changing data for periodic migration

■ DB Schema with intricate relationship, foreign key constraints

■ Read-only data sources, need to tie with Drupal content types

Advantages Disadvantages

Advantages

■ Gain all Drupal 8 functionality on dataset

■ Fieldable Entities■ Options for headless

Drupal/dataset APIs■ Search API / Facets■ No import/migration process■ Real-time data■ No Data Duplication■ External DB Replication■ High level of

customization/overrides

Disadvantages

Advantages

■ Gain all Drupal 8 functionality on dataset

■ Fieldable Entities■ Options for headless

Drupal/dataset APIs■ Search API / Facets■ No import/migration process■ Real-time data■ No Data Duplication■ External DB Replication■ High level of

customization/overrides

Disadvantages

■ Complexity of Views■ Configuring Relationships■ Heavy initial development to

establish Custom Entities■ More Performance Tuning

Required■ Requires External DB to be on

same server as Drupal DB■ For local dev, requires copy

of external DB

D8 Custom Entities In Action

Addressing Drupal DB/Table Needs

Possible Issues with External DB■ Multiple Primary Keys, Non-Integer Primary Keys■ Relationship tables (all become entities?)■ DB Not On Same Server as Drupal

Solutions■ Adding new unique ID column to tables■ Generated Columns (MySQL 5.7.6+)■ Computed Fields■ SQL Views, Custom Entity built on views■ Database Replication to sync w/Drupal server

Custom Entity Creation Process

■ Creating Custom Modules To Hold Entities

■ Mapping Tables/Columns to Entities/Fields

■ Drupal Console drupal generate:entity:content

■ Adjusting Annotations

■ Getters/Setters for all fields

■ Adding fields to baseFieldDefinitions

■ Adding Table Prefix to settings.php

Custom Entities Annotation

Custom Entities Services

■ Overriding Uninstall Services

ContentUninstallValidator Override

Custom Entities Class Inheritance

■ DB Tables mapped to entities w/common fields (state, status, location, etc)

■ Extending “base” entity class(es) to provide common functionality to many entities

■ Use of custom SqlContentEntityStorage

“State” Filtering Example on Entity

/ Views

Use of “State” field in

DdBaseStateField Class

Hearing Entity extending

DdBaseStateField Class

Use Of Custom SqlContentEntityStorage Handlers

dd_base.views_execution.inc - hook_views_query_alter() to add State filtering

Custom Entities w/Views

■ Post Drupal Console generation, ready w/only relationships defined in Entity baseFieldDefinitions

■ Modifying xxxxViewsData.php(Reference: views.api.php hook_views_data)

■ Ability to add relationships to tables that have no custom entities

■ Support for table joins w/multiple conditions

Drupal\dd_hearing\Entity\DdHearing

Views “Add Relationships” for Hearing

DdPersonViewsData - “Add Relationships” for Person

Base Table Name

Unique ID For Relationship Column

Table to join

Field to join on

Field on Person Table to join

Resulting SQL Query:SELECT Person.pid AS pid, Legislator_Person.pid AS legislator_person_pid FROM {Person} Person INNER JOIN {Legislator} Legislator_Person ON Person.pid = Legislator_Person.pid

Views “Add Relationships” for Person

Limitation Of Views (w/o Search API)

■ Handling sitewide search

■ Performance

■ No Facets

■ No Big Data Tools

Custom Entities w/Search API

■ Entity Reference fields automatically available

■ Computed Fields allow data to be added to Search Index during entity load

■ Patch to allow computed fields to be visible: https://www.drupal.org/node/2814925

■ Faceted Search on Views built on Search Indexes

dd_hearing.moduleSet Computed Field On Entity Storage Load

Search API Index Fields

The End Result

Q & A

Contact Information

Kristian Ducharmekristian.ducharme@civicactions.com

Drupal.org: kducharm GitHub: kducharm

BFFs 4eva: Drupal 8 and Custom Databases

http://www.civicactions.com