ElasticSearch feature | patchroute refactoring -> elastic export -> data migration -> refactoring participant list

[ScalaChamp]

Things have done:

• Introduced OO principles to separate responsibilities into different classes.
• Created BasePatch abstract class as a parent of different classes, which are serving common feature.
• Introduced Template method design pattern in BasePatch to extract common code to avoid code duplication.
• Created abstract methods in BasePatch to specify what children should do to patch data.
• Introduced factory method to hide low level details about creation of special instance of BasePatch.

---

• Introduced Exportable interface for having an abstraction for exporting data and mappings to ElasticSearch.
• Created BaseExporter (abstract class) and ElasticExportAdapter using "Adapter Design Pattern" to loose coupling from ElasticSearch API, which wraps all the necessary API-s from ElasticSearch library.
• We created two DTO classes in order to reduce verbosity/complexity of ElasticExportAdapter, these classes are: UpsertDataRequestPayload and UpsertMappingRequestPayload which are used by ElasticExportAdapter in order to make request to ElasticSearch.
• We introduced Generator and Collector abstractions in order to generate and collect data/mapping before making upsert request to ElasticSearch.
• Concrete implementations of Generator and Collector are: BaseGenerator (abstract class), MappingGenerator and SourceGenerator which are responsible for generating and collecting either mapping or data.
• Created another DTO class GeneratorPayload for Generator classes which are used generating mapping/data.
• Introduced Processor abstraction to process data before "upserting" to ElasticSearch to avoid replacing existing ElasticSearch data.
• Concrete implementations of Processor is: CollectionProcessor whose purpose is to work on data represented as "1 to many relationships".
• Introduced Parser abstraction in order to parse text values to concrete types, it works for mapping and data as well.
• Concrete implementations of Parser are: TypeParser and ValueParser which are responsible for parsing types for mapping and parsing values for data.
• Introduced ExportFacade class which uses "Facade Design Pattern" which is a front-facing protocol masking more underlying complexity and details of exporting to ElasticSearch.
• Created another DTO class: ExportFacadePayload which is needed for transferring all the necessary data to ExportFacade
• Added Util class for ElasticSearch which is responsible for providing only utility methods which do not have side effects (any connection to database, network or 3rd party vendor).
• Refactored legacy code based on new design.
• Eliminated the code whose functionality is covered by new feature.

---

• Made BaseExporter more generic by leaving only one method - export(). Created 2 subclasses: ElasticDataExportAdapter for exporting data and ElasticMappingExportAdapter for exporting mappings to ES. The subclasses use Adapter pattern in order to adapt to any occurring changes in ES API.
• Introduced the abstraction of Merger with method merge that helps us to merge different mappings into one final mapping, Merger can be used for other cases as well.
• Since it was agreed beforehand that we would use camel case pattern for keys we created a new method underscoresToCamelCase() in Util.java which handles this issue.
• Created an abstract class BaseMigrator which works for both cases (one to one, one to many). BaseMigrator uses following interfaces: Generator and Exportable.
• Added a new class BulkExportFacade which provides a simple API and hides the complexity associated with exporting data to ES.
• Introduced the class BaseCollectionMigrator which is a special case for classes which work on the data represented as one-to-many relationships.
• Created several concrete classes of Migrator abstraction: KitRequestShippingMigrator, MedicalRecordMigrator, OncHistoryDetailsMigrator, ParticipantDataMigrator and ParticipantMigrator.
• Used already existing Pub/Sub functionality as a control unit to make it easier to trigger exporting existing data to ES.
• Details of Pub/Sub: taskType = ELASTIC_EXPORT, body: {"study": "<study_name>", "isMigration": true}
• Made additional refactoring in order to clean up the code.

---

• Created filtering for ElasticSearch based on existing filtering strategy.
• Introduced splitter abstraction for splitting filters by different criterias, e.g (<=, >=, <=>, LIKE, JSON_EXTRACT, DATE, STR_DATE and etc)
• Created the single and collection query builders to build ElasticSearch queries depending on operators.

P.S code was written using pair programming with @gmakharat .