dev/core#6679 - Full-text search: let each MATCH drive its own FULLTEXT index

[dstorozhuk]

Overview

Fixes dev/core#6679.

The first query in CRM_Contact_Form_Search_Custom_FullText_Activity::prepareQueries() puts its two matchText() clauses on either side of an OR. With enable_innodb_fts on those expand to MATCH … AGAINST, and MySQL cannot use a FULLTEXT index as an access path for a MATCH that is only one side of an OR — so it scans every non-deleted activity instead. That one query is ~88% of the full-text search screen, which is why enabling InnoDB full-text search does not make the screen faster.

Splitting the two branches into a UNION lets each drive its own index. Measured 44× overall across ten search terms on a database with ~76k contacts and ~169k activities, with byte-identical results.

Before

SELECT     distinct ca.id
FROM       civicrm_activity ca
INNER JOIN civicrm_activity_contact cat ON cat.activity_id = ca.id
INNER JOIN civicrm_contact c ON cat.contact_id = c.id
LEFT  JOIN civicrm_email e ON cat.contact_id = e.contact_id
LEFT  JOIN civicrm_option_group og ON og.name = 'activity_type'
LEFT  JOIN civicrm_option_value ov ON ( ov.option_group_id = og.id )
WHERE      (
             (MATCH (c.sort_name,c.display_name,c.nick_name) AGAINST ('+"registration"' IN BOOLEAN MODE))
             OR
             (MATCH (e.email) AGAINST ('+"registration"' IN BOOLEAN MODE) AND ca.activity_type_id = ov.value AND ov.name IN ('Inbound Email', 'Email') )
           )
AND        (ca.is_deleted = 0 OR ca.is_deleted IS NULL)
AND        (c.is_deleted = 0 OR c.is_deleted IS NULL)
LIMIT 11

EXPLAIN — the FULLTEXT index is in possible_keys and is not chosen:

table  type    possible_keys                             key                rows
ca     ref     …, civicrm_fts_1b19478711aa261826dd38d…   index_is_deleted  61179
cat    ref     UI_activity_contact, index_record_type    index_record_type      4
c      eq_ref  PRIMARY, index_is_deleted_sort_name       PRIMARY                1
e      ref     FK_civicrm_email_contact_id               FK_…_contact_id        1
og     const   UI_name                                   UI_name                1
ov     ref     FK_civicrm_option_value_option_group_id   FK_…_group_id          8

Slow log, one search: Query_time: 2.963 Rows_examined: 2735023.

After

The two branches become a UNION, and civicrm_option_group / civicrm_option_value move into the e-mail branch — the only place they are used — joined on the activity type instead of cross-joined into every row.

Slow log for the same search after the change: Query_time: 0.009.

Per search term, same box and data, best of 2:

term	before	after	speedup
registration	3.519 s	0.009 s	381×
judge	3.548 s	0.004 s	821×
school	3.659 s	0.083 s	44×
smith	3.753 s	0.058 s	65×
gmail	10.412 s	0.644 s	16×
teacher	3.806 s	0.001 s	3324×
a	2.875 s	0.001 s	4873×
total, 10 terms	43.813 s	0.991 s	44×

End to end, the whole full-text search screen:

term	FTS off	FTS on	FTS on + this PR
registration	4 723 ms	2 911 ms	45 ms
judge	656 ms	279 ms	39 ms
school	141 ms	145 ms	65 ms

Technical Details

The LEFT JOINs to civicrm_email, civicrm_option_group and civicrm_option_value become INNER JOINs in the e-mail branch. That is equivalent because those tables are only referenced by that branch's own predicate: in the original query a row could only satisfy the second OR branch when e.email matched and ca.activity_type_id = ov.value and ov.name IN ('Inbound Email','Email'), all of which require the joins to have produced a row. The contact branch never referenced them at all.

Results were verified as full result sets, not counts: ten search terms compared on both row count and MD5(GROUP_CONCAT(id ORDER BY id)) between the old and new query. All ten identical, including result sets of 11,168 and 16,621 ids.

Comments

Only the Activity partial query has this shape. Contact, Case, Contribution, Membership and Participant each have a single matchText() call and already use their index — no change needed there.

Measured on CiviCRM 6.16.5 with MySQL 8.0, PHP 8.3, Drupal 10.6; the query is unchanged in master, which is what this PR targets.

[civibot]

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...

• If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
• A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
• A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
• If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
• Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
• A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
• To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
• For more information about contributing, see CONTRIBUTING.md.

PR commands & links...

• /rebase <branch-name> will rebase your branch and change the base of the PR.
• /squash will combine all commits (keeping only the first commit messsage).
• /port <branch-name> will create a copy of this PR against a different branch.
• /lintroll will automatically fix linting errors, amending commits as needed.
• retest this please will rerun the tests and rebuild the demo site.
• 📖 Review standards
• 🗒️ Review template (brief or verbose)

➡️ Online demo of this PR 🔗

[civibot]

The issue associated with the Pull Request can be viewed at https://lab.civicrm.org/dev/core/-/issues/6679