Move to ElasticSearch and drop Solr/MongoDB#33

Open

markwoodhall wants to merge 157 commits into

developfrom

elastic

Commits on May 11, 2018

Start of structure for elastic indexing

kjw
authored and
markwoodhall
committed
Switch to one type per index, for ES 6+ compatibility

kjw
authored and
markwoodhall
committed
Use keyword and text instead of string type, for ES 5+ compatibility

kjw
authored and
markwoodhall
committed
Disable _all field (not present in ES 6+ anyway)

kjw
authored and
markwoodhall
committed
More complete mappings

kjw
authored and
markwoodhall
committed
Rework member loading to load into ES

kjw
authored and
markwoodhall
committed
index-publishers renamed to index-members

kjw
authored and
markwoodhall
committed
Index journals into ES

kjw
authored and
markwoodhall
committed
Remove some fields from category definition

kjw
authored and
markwoodhall
committed
Index subjects into ES

kjw
authored and
markwoodhall
committed
Rewrite task entry points and user functions

kjw
authored and
markwoodhall
committed
Start of the ES work indexer

kjw
authored and
markwoodhall
committed
Connect parse actions to ES indexing

kjw
authored and
markwoodhall
committed
More complete work indexing for ES

kjw
authored and
markwoodhall
committed
Mostly complete ES work indexer

kjw
authored and
markwoodhall
committed
Complete ES work indexer

kjw
authored and
markwoodhall
committed
Include unstructured field in references and various indexer fixes

kjw
authored and
markwoodhall
committed
Insert work documents into ES

kjw
authored and
markwoodhall
committed
Basic queries and DOI look up via elastic search

kjw
authored and
markwoodhall
committed
Basic filters for ES queries

kjw
authored and
markwoodhall
committed
Mostly working ES filters, dates in ES to citeproc convertion

kjw
authored and
markwoodhall
committed
Nested filters for nested document fields

kjw
authored and
markwoodhall
committed
Additional metadata in ES doc to citeproc conversion

kjw
authored and
markwoodhall
committed
Add journal id to work mapping

kjw
authored and
markwoodhall
committed
Working select parameter for ES

kjw
authored and
markwoodhall
committed
Sorting for ES queries and re-enable debug parameter

kjw
authored and
markwoodhall
committed
Report elastic client config on debug=true

kjw
authored and
markwoodhall
committed
Port work filters to ES

kjw
authored and
markwoodhall
committed
Working work facets in ES

kjw
authored and
markwoodhall
committed
Include published-year work field for published facet

kjw
authored and
markwoodhall
committed
Make all work fields keywords except those ending in .text

kjw
authored and
markwoodhall
committed
Specific field queries for ES

kjw
authored and
markwoodhall
committed
Reverse work look ups for ES

kjw
authored and
markwoodhall
committed
Working sample paramter for ES

kjw
authored and
markwoodhall
committed
Include _seq_no field for random sort (without field is deprecated)

kjw
authored and
markwoodhall
committed
API cursors using ES scroll function

kjw
authored and
markwoodhall
committed
Store member and journal coverage data in ES

kjw
authored and
markwoodhall
committed
Rename published to issued and published-year to issued-year

kjw
authored and
markwoodhall
committed
Index subjects into journal mappings

kjw
authored and
markwoodhall
committed
Handle case of no print publication date

kjw
authored and
markwoodhall
committed
Query for and display members from ES

kjw
authored and
markwoodhall
committed
Journal query and display for ES

kjw
authored and
markwoodhall
committed
Prefix search and display for ES

kjw
authored and
markwoodhall
committed
Remove old code for citation checking

kjw
authored and
markwoodhall
committed
Citation display decision fed by ES member data

kjw
authored and
markwoodhall
committed
Remove old doaj code

kjw
authored and
markwoodhall
committed
Remove old patent deposit code (now handled by event data)

kjw
authored and
markwoodhall
committed
Remove mongo configuration

kjw
authored and
markwoodhall
committed
Use ES to serve /types/:type-id/works

kjw
authored and
markwoodhall
committed
Index feed documents into ES

kjw
authored and
markwoodhall
committed
Removal of old, unused code

kjw
authored and
markwoodhall
committed
Set minimum_should_match for ISSN matching

kjw
authored and
markwoodhall
committed
Auto complete fields for member, journal and funder names

kjw
authored and
markwoodhall
committed
Complete ES to citeproc mapping

kjw
authored and
markwoodhall
committed
Display contributors correctly

kjw
authored and
markwoodhall
committed
Index funders into ES. Now a single pass load

kjw
authored and
markwoodhall
committed
Get funder data from ES for /funders

kjw
authored and
markwoodhall
committed
Fixes for fundref namespace removal

kjw
authored and
markwoodhall
committed
Remove deposits API and all mongo code

kjw
authored and
markwoodhall
committed
Reference count and DOI updates for ES

kjw
authored and
markwoodhall
committed
Remove /licenses route (in favour of license facet)

kjw
authored and
markwoodhall
committed
Store some fields in work parent for aggregation at work level

kjw
authored and
markwoodhall
committed
Add link application facet

kjw
authored and
markwoodhall
committed
Use copy_to to make parent aggregation fields

kjw
authored and
markwoodhall
committed
Index citation-id and book-id crm items

kjw
authored and
markwoodhall
committed
New dates in selection of issued date

kjw
authored and
markwoodhall
committed
Use most recent deposit date if a DOI has absolutely no other date

kjw
authored and
markwoodhall
committed
Don't add nil relation or property values

kjw
authored and
markwoodhall
committed
Support query clauses that need bool must_not occurrence

kjw
authored and
markwoodhall
committed
Use prefix reference distribution flag

kjw
authored and
markwoodhall
committed
Fix reference distribution element name

kjw
authored and
markwoodhall
committed
Public references are labelled "open", not "public"

kjw
authored and
markwoodhall
committed
Projectile config to avoid searching test-data

kjw
authored and
markwoodhall
committed
Minimal commits required after rebasing
markwoodhall
committed
Update config to match master
markwoodhall
committed
Minor formatting fix
markwoodhall
committed
Remove tasks no longer used by elastic implementation
markwoodhall
committed
Add accoc-exists from master
markwoodhall
committed
Parse coverage stats as doubles
markwoodhall
committed
Add some missing index mappings
markwoodhall
committed
Get hour, min, and second where applicable
markwoodhall
committed
Consider same dates as master for issued date, store published date
markwoodhall
committed
Store contributor sequence
markwoodhall
committed
Include contributor sequence in citeproc
markwoodhall
committed
Store journal language and item source
markwoodhall
committed
Replace many used of assoc with assoc-exists, to match master
markwoodhall
committed
Unparse date with no milliseconds
markwoodhall
committed
Make deposited and first-deposited implementation the same as master
markwoodhall
committed
Update member name, prefix, and location to match master
markwoodhall
committed
Add support for member coverage when fetching more than one member
markwoodhall
committed
Add breakdown, subjects defaults to empty vector, coverage support when fetching more than one journal
markwoodhall
committed
Use should instead of filter to influence score
markwoodhall
committed
Don't wait forever when indexing if counts don't match
markwoodhall
committed
docker-compose down before starting system
markwoodhall
committed
Renames and formatting
markwoodhall
committed
Formatting and whitespace fixes
markwoodhall
committed
Port peer review from master
markwoodhall
committed
Port institutions from master
markwoodhall
committed
Port journal-issue from master
markwoodhall
committed
Port free-to-read to master
markwoodhall
committed
published-print should be explicit or default
markwoodhall
committed
ORCID should include URI as per master, optionally include authenticated-orcid
markwoodhall
committed
Remove rel from citeproc relations
markwoodhall
committed
Port crossmark-unaware content-domains from master
markwoodhall
committed
Add missing institution index mapping
markwoodhall
committed
Only get date-parts for various published dates
markwoodhall
committed
Default titles so output matches master, always assoc publisher
markwoodhall
committed
Integer is not applicable for reference year
markwoodhall
committed
Use match query for field queries
markwoodhall
committed
Add score to the list of selectable scores
markwoodhall
committed
Formatting
markwoodhall
committed
Add sorter to api-get
markwoodhall
committed
Keep waiting for indexing while doc count increases
markwoodhall
committed
Remove extra fields from title text
markwoodhall
committed
Make loading differnt corpus data easier
markwoodhall
committed
Add citation matching tests
markwoodhall
committed
Formatting
markwoodhall
committed
Update assertion data for minor elastic variations
markwoodhall
committed
Institution and assertion don't need to be nested
markwoodhall
committed
Disble xpack security
markwoodhall
committed
Add a simple test to prove sample still works
markwoodhall
committed
Debug message
markwoodhall
committed
Port missing agency fix from master
markwoodhall
committed
Bring funder indexing and api in line with master
markwoodhall
committed
Formatting
markwoodhall
committed
Formatting
markwoodhall
committed
Fix typo in rdf node name
markwoodhall
committed
Support hierarchy and hierarchy names as per master
markwoodhall
committed
Fully implement funder hierarchy
markwoodhall
committed
Wait for funder to be indexed
markwoodhall
committed
Sort funder data for deterministic compare
markwoodhall
committed
Update works assertion data
markwoodhall
committed
Extract long doi for transform resources
markwoodhall
committed
expose issn type as :type not :kind
markwoodhall
committed
Remove score comparsion from certain works tests
markwoodhall
committed
Use one shard for work index
markwoodhall
committed
Add helper function to pretty print resource to path
markwoodhall
committed
Remove score comparison from journal work test
markwoodhall
committed
Add breakdowns to member coverage
markwoodhall
committed
Update subtypes for books
markwoodhall
committed
Ignore score during member works test
markwoodhall
committed
Update assertion data to reflect isbn and type fixes
markwoodhall
committed
Remove load-test-journals
markwoodhall
committed
Set cr-funder-registry on start
markwoodhall
committed
Delay to give coverage time to index
markwoodhall
committed
Update test corpus after rebase
markwoodhall
committed
Remove solr and outdated test documentation from readme
markwoodhall
committed
Fix typo
markwoodhall
committed

Commits on Jul 11, 2018

Commits on Jul 12, 2018

Commits on Jul 13, 2018

Commits on Jul 24, 2018

Add complete count-types and coverage-type implementation
markwoodhall
committed

Commits on Jul 25, 2018

Fix failing funder works test
markwoodhall
committed