Tweak intraspecies alignment packing code

[twalsh-ebi]

Description

eg-web-common PR 133 removed the EnsEMBL::Web::ConfigPacker::_summarise_compara_alignments stub method from eg-web-common code and restored the packing of intraspecies alignments in NV sites. This had two main effects:

• it restored intraspecies alignments (e.g. Wheat polyploidy-aware self-alignment) to Ensembl Plants;
• it increased the size and packing time of the Plants MULTI.db.packed file.

The increased MULTI.db.packed packing time has been posing something of a challenge. This has perhaps been highlighted by recent web code changes to reduce database access by using MULTI.db.packed ( e.g. eg-web-common PR 175 ).

This pull request would tweak the code for identifying intraspecies alignment datasets and summarising Compara alignments, with a view to reducing MULTI.db.packed packing time in Ensembl Plants.

Changes include:

• moving the code of EnsEMBL::Web::ConfigPacker::_summarise_compara_alignments to EnsEMBL::Web::Utils::Compara::_summarise_compara_alignment_data, and calling the latter from the former.
• replacing the query used to fetch alignment endpoints with a more efficient query executing in roughly half the time. When tested directly on ensembl_compara_plants_63_116, the new query returned its 774 rows in 2 min 31.70 sec, with each row containing an alignment endpoint pair. By contrast, the existing query took 4 min 21.58 sec to return its 21,322,846 rows, which require further processing to obtain the alignment endpoint pairs.
• refactoring the intraspecies constraint queries to a method EnsEMBL::Web::Utils::Compara::_fetch_intraspecies_constraints which uses one database query instead of two.
• adding a comment to each of the ConfigPacker methods _get_regions and _comp_se to indicate they are not in use (as far as we know).

Refactoring alignment summary code would also make it possible to implement pre-packing of intraspecies alignments as in eg-web-plants PR 110.

Views affected

Because this change is intended to help with repacking MULTI.db.packed, it should not result in a change to any web view.

It should ideally not even affect the contents of MULTI.db.packed. During testing, changes were observed in the Vertebrates intraspecies alignment region summary. However, a spot-check of one such change confirmed that the new database query provided a more accurate alignment endpoint pair.

Possible complications

None expected.

Merge conflicts

None detected.

Related JIRA Issues (EBI developers only)

• N/A

[sgiorgetti]

This query might take a while to run, as it seems there is hardly any constraint.
Would it be possible - if straightforward - to include this dataset to packed / json, and essentially cache its results?

[twalsh-ebi]

Thanks for having a look at this, @sgiorgetti.

Actually the results of this intraspecies query are only used in the generation of the Compara alignment summary, so we wouldn't need to cache the former if we cache the latter.

If we cache the Compara intraspecies alignment data in a JSON file ( as implemented in eg-web-plants PR 110 ), then this query could be skipped during packing of the Ensembl Plants MULTI.db.packed file.