Story-1870 ICMA Phase 2 changes (finos#1890)

* Remove deprecated components

RELEASE.md

@@ -1,21 +1,59 @@
-# *Product Model - Package Price and Spread*
+# *ICMA Contribution - Removal of Deprecated Components*
 
 _Background_
 
-Coverage for package price was introduced in FpML 5.13. This release allows for the mapping of package prices and spreads, and refactors `BusinessEvent` and `EventInstruction` to improve consistency.
+As part of ICMA's contribution to the CDM for Repo and Bonds, several model components that have been ear-marked as deprecated need to be removed. Those component are now supersded by new components from that contribution.
 
 _What is being released?_
 
-This release adjusts the model and adds FpML mapping coverage for package prices and spreads.
+This release removes the following deprecated components:
+
+**Data types**
+
+- `SecurityFinancePayout` (superseded by `AssetPayout`)
+- `SecurityFinanceLeg` (superseded by `AssetLeg`)
+
+**Attributes**
+
+- In `Payout` and `SettlementOrigin`: `securityFinancePayout` (superseded by `assetPayout`)
+- In `Collateral`: `marginPercentage` (already represented within `eligibleCollateral`)
+- In `ProductTaxonomy`: `taxonomySource` and `taxonomyValue` (superseded by `source` and `value` in `Taxonomy` super-type)
+- In `AssignedIdentifier`: `identifierType` (moved to `TradeIdentifier`)
+
+**Annotations**
+
+- In `ExecutionDetails`: the `[metadata reference]` annotation for `packageReference`
+
+**Functions**
+
+In addition, the functional model has been amended as follows:
+
+- The logic previously relying on the `securityFinancePayout` attribute now uses `assetPayout` and `collateral->collateralProvisions`, including the sec-lending product and event qualification logic. The functions impacted are:
+
+  - `CalculateTransfer`
+  - `Create_SecurityTransfer`
+  - `Create_SecurityFinanceTransfer`
+  - `ResolveTransfer`
+  - `SecurityFinanceCashSettlementAmount`
+  - `Create_BillingRecord`
+  - `ResolveSecurityFinanceBillingAmount`
+  - `Create_AssetPayoutTradeStateWithObservations`
+  - `Qualify_SecurityLendingAgreement`
+  - `Qualify_Repurchase`
+  - `Qualify_FullReturn`
+
+- The sec-lending samples and mappers have been adjusted to reflect the new structure
+- Function names referring to "Security Finance" have replaced it with the more generic "Asset"
+- Model descriptions that use the term "Security Finance Payout" have replaced it with "Asset Payout"
+
+**Other**
+
+- Various comments that were left-over from the contribution have been removed
 
-- Mappings added to populate CDM attribute `ExecutionDetails->packageReference->price` from FpML element `quote`.
-- `BusinessEvent` updated to extend `EventInstruction`, with all attributes (except `eventQualifier` and `after`) moved to `EventInstruction`.
-
 _Review directions_
 
-In the CDM Portal, select the Textual Browser and inspect each of the changes identified above.
+In the CDM Portal, select the Textual Browser and inspect the types and functions.
 
-In the CDM Portal, select Ingestion and review the following samples:
+In the CDM Portal, select Ingestion and review the sample in the "fis" folder.
 
-- fpml-5-10 > processes > msg-package-price
-- fpml-5-10 > processes > msg-package-spread
+In Rosetta, select the Visualisation tab and review the event examples in the "Security Lending" folder.

documentation/source/product-model.rst

@@ -478,8 +478,6 @@ The ``Payout`` type defines the composable payout types, each of which describes
    fixedPricePayout FixedPricePayout (0..*)
    securityPayout SecurityPayout (0..*)
         [deprecated]
-   securityFinancePayout SecurityFinancePayout (0..*)
-        [deprecated]
    cashflow Cashflow (0..*)
    performancePayout PerformancePayout (0..*)
    assetPayout AssetPayout (0..*)
@@ -691,19 +689,13 @@ The ``ProductTaxonomy`` data structure and an instance of a CDM object (`seriali
          [metadata scheme]
      secondaryAssetClass AssetClassEnum (0..*)
          [metadata scheme]
-     taxonomyValue string (0..1)
-         [metadata scheme]
-         [deprecated]
-     taxonomySource TaxonomySourceEnum (0..1)
-         [deprecated]
      productQualifier string (0..1)
 
      condition TaxonomyType:
-         required choice source, primaryAssetClass, secondaryAssetClass, taxonomySource
+         required choice source, primaryAssetClass, secondaryAssetClass
 
      condition TaxonomySource:
-         if source exists then value exists and
-		 if taxonomySource exists then productQualifier exists
+         if source exists then ( value exists or productQualifier exists )
 
      condition TaxonomyValue:
          optional choice value, productQualifier

isda-demo/src/test/java/com/regnosys/cdm/example/functions/CreateNovationEventTest.java

@@ -87,8 +87,8 @@ private WorkflowStep getWorkflowStepInstruction() throws IOException {
                                         .setRole(CounterpartyRoleEnum.PARTY_2))
                                 .setTradeId(Lists.newArrayList(TradeIdentifier.builder()
                                         .addAssignedIdentifier(AssignedIdentifier.builder()
-                                                .setIdentifierValue("UTI-Trade-Party-3")
-                                                .setIdentifierType(TradeIdentifierTypeEnum.UNIQUE_TRANSACTION_IDENTIFIER))
+                                                .setIdentifierValue("UTI-Trade-Party-3"))
+                                        .setIdentifierType(TradeIdentifierTypeEnum.UNIQUE_TRANSACTION_IDENTIFIER)
                                         .setIssuerValue("LEI-PARTY-3")))))
                 // Split breakdown to terminate the original trade
                 .addBreakdown(PrimitiveInstruction.builder()

rosetta-source/src/main/java/cdm/base/staticdata/asset/common/processor/TaxonomySourceMappingProcessor.java

@@ -1,18 +1,15 @@
 package cdm.base.staticdata.asset.common.processor;
 
 import cdm.base.staticdata.asset.common.TaxonomySourceEnum;
-import com.regnosys.rosetta.common.translation.MappingContext;
-import com.regnosys.rosetta.common.translation.MappingProcessor;
-import com.regnosys.rosetta.common.translation.MappingProcessorUtils;
-import com.regnosys.rosetta.common.translation.Path;
+import com.regnosys.rosetta.common.translation.*;
 import com.regnosys.rosetta.common.util.PathUtils;
 import com.rosetta.model.lib.RosettaModelObjectBuilder;
 import com.rosetta.model.lib.path.RosettaPath;
 
 import java.util.List;
+import java.util.Optional;
 
 import static cdm.base.staticdata.asset.common.ProductTaxonomy.ProductTaxonomyBuilder;
-import static com.rosetta.model.metafields.FieldWithMetaString.FieldWithMetaStringBuilder;
 
 @SuppressWarnings("unused")
 public class TaxonomySourceMappingProcessor extends MappingProcessor {
@@ -22,32 +19,33 @@ public TaxonomySourceMappingProcessor(RosettaPath modelPath, List<Path> synonymP
     }
 
     @Override
-    public void map(Path synonymPath, RosettaModelObjectBuilder builder, RosettaModelObjectBuilder parent) {
-        MappingProcessorUtils.getNonNullMappingForModelPath(getMappings(), PathUtils.toPath(getModelPath().newSubPath("value")))
-                .map(m -> m.getXmlPath())
+    public <T> void mapBasic(Path synonymPath, Optional<T> instance, RosettaModelObjectBuilder parent) {
+        Path productTaxonomyModelPath = PathUtils.toPath(getModelPath()).getParent();
+        Path taxomomyValueModelPath = productTaxonomyModelPath.addElement("value");
+        Path nameModelPath = taxomomyValueModelPath.addElement("name").addElement("value");
+        // Find xml path from name model path due to mapping bug where schemes on multi-cardinality basic types get
+        // mapped to the wrong list item
+        MappingProcessorUtils.getNonNullMappingForModelPath(getMappings(), nameModelPath)
+                .map(Mapping::getXmlPath)
                 .ifPresent(xmlPath -> {
                     ProductTaxonomyBuilder productTaxonomyBuilder = (ProductTaxonomyBuilder) parent;
-                    FieldWithMetaStringBuilder taxonomyValueBuilder = (FieldWithMetaStringBuilder) builder;
-
-                    updateSchemeAndSource(xmlPath, productTaxonomyBuilder, taxonomyValueBuilder);
+                    updateSchemeAndSource(xmlPath, productTaxonomyBuilder);
 
                     // If unset, set to OTHER
-                    if (productTaxonomyBuilder.getTaxonomySource() == null) {
-                        productTaxonomyBuilder.setTaxonomySource(TaxonomySourceEnum.OTHER);
+                    if (productTaxonomyBuilder.getSource() == null) {
+                        productTaxonomyBuilder.setSource(TaxonomySourceEnum.OTHER);
                     }
                 });
     }
-
-    protected void updateSchemeAndSource(Path xmlPath, ProductTaxonomyBuilder productTaxonomyBuilder, FieldWithMetaStringBuilder taxonomyValueBuilder) {
+    protected void updateSchemeAndSource(Path xmlPath, ProductTaxonomyBuilder productTaxonomyBuilder) {
         setValueAndUpdateMappings(xmlPath.addElement("productTypeScheme"),
                 xmlValue -> {
                     // Update scheme
-                    taxonomyValueBuilder.getOrCreateMeta().setScheme(xmlValue);
+                    productTaxonomyBuilder.getOrCreateValue().getOrCreateName().getOrCreateMeta().setScheme(xmlValue);
                     // Update taxonomySource
-                    productTaxonomyBuilder.setTaxonomySource(getTaxonomySourceEnum(xmlValue));
+                    productTaxonomyBuilder.setSource(getTaxonomySourceEnum(xmlValue));
                 });
     }
-
     protected TaxonomySourceEnum getTaxonomySourceEnum(String scheme) {
         if (scheme.contains("www.fpml.org/coding-scheme/product-taxonomy")) {
             return TaxonomySourceEnum.ISDA;

rosetta-source/src/main/java/cdm/event/workflow/processor/FISMapperMappingProcessor.java

@@ -16,8 +16,10 @@
 import cdm.event.workflow.WorkflowStep.WorkflowStepBuilder;
 import cdm.observable.asset.PriceExpression;
 import cdm.observable.asset.PriceTypeEnum;
+import cdm.product.collateral.CollateralProvisions;
 import cdm.product.collateral.CollateralTypeEnum;
 import cdm.product.common.settlement.DeliveryMethodEnum;
+import cdm.product.template.AssetPayout;
 import cdm.product.template.DurationTypeEnum;
 import cdm.product.template.EconomicTerms.EconomicTermsBuilder;
 import cdm.product.template.TradableProduct;
@@ -50,7 +52,6 @@
 import static cdm.event.common.TradeState.TradeStateBuilder;
 import static cdm.product.asset.InterestRatePayout.InterestRatePayoutBuilder;
 import static cdm.product.common.settlement.PriceQuantity.PriceQuantityBuilder;
-import static cdm.product.template.SecurityFinancePayout.SecurityFinancePayoutBuilder;
 
 /**
  * This instance override the version in CDM so it can be kept up to date with ISLA model changes.
@@ -301,18 +302,19 @@ private Multimap<String, MappingConsumer<TradeStateBuilder>> buildCommonMappings
 		});
 
 		commonMappings.put("Collateral_Type_IND", (indexes, value, tradeState) -> {
-			getSecPO(tradeState)
+			getColPro(tradeState)
 					.getValue()
-					.getOrCreateCollateralProvisions()
 					.setCollateralType(parseCollateralType(value));
 			return Collections.singletonList(new PathValue<>(tradeState.getModelPath(), value));
 		});
 
 		commonMappings.put("Required_Trade_Mrgn", (indexes, value, tradeState) -> {
-			getSecPO(tradeState)
+			getColPro(tradeState)
 					.getValue()
-					.getOrCreateCollateralProvisions()
-					.getOrCreateMarginPercentage()
+					.getOrCreateEligibleCollateral(0)
+					.getOrCreateCriteria(0)
+					.getOrCreateTreatment()
+					.getOrCreateValuationTreatment()
 					.setMarginPercentage(parseDecimal(value).divide(BigDecimal.valueOf(100)));
 			return Collections.singletonList(new PathValue<>(tradeState.getModelPath(), value));
 		});
@@ -347,7 +349,7 @@ private Multimap<String, MappingConsumer<TradeStateBuilder>> buildCommonMappings
 					.addProductIdentifierValue(productIdentifier, 0);
 			// reference
 			Reference.ReferenceBuilder reference = Reference.builder();
-			PathValue<SecurityFinancePayoutBuilder> secLendingPayout = getSecPO(tradeState);
+			PathValue<AssetPayout.AssetPayoutBuilder> secLendingPayout = getSecPO(tradeState);
 			secLendingPayout
 					.getValue()
 					.getOrCreateSecurityInformation()
@@ -363,21 +365,21 @@ private Multimap<String, MappingConsumer<TradeStateBuilder>> buildCommonMappings
 		commonMappings.put("DVP_Indicator", (indexes, value, tradeState) -> {
 			getSecPO(tradeState)
 					.getValue()
-					.getOrCreateSecurityFinanceLeg(0)
+					.getOrCreateAssetLeg(0)
 					.setDeliveryMethod(parseDeliveryMethod(value));
 			return Collections.singletonList(new PathValue<>(tradeState.getModelPath(), value));
 		});
 
 		commonMappings.put("Security_SettDueDt", (indexes, value, tradeState) -> {
 			getSecPO(tradeState)
 					.getValue()
-					.getOrCreateSecurityFinanceLeg(0)
+					.getOrCreateAssetLeg(0)
 					.getOrCreateSettlementDate()
 					.getOrCreateAdjustableDate()
 					.setAdjustedDateValue(parseISODate(value));
 			getSecPO(tradeState)
 					.getValue()
-					.getOrCreateSecurityFinanceLeg(0)
+					.getOrCreateAssetLeg(0)
 					.getOrCreateSettlementDate()
 					.getOrCreateAdjustableDate()
 					.getOrCreateDateAdjustments()
@@ -553,10 +555,16 @@ private PathValue<EconomicTermsBuilder> getEcTerms(PathValue<TradeStateBuilder>
 				tp.getValue().getOrCreateProduct().getOrCreateContractualProduct().getOrCreateEconomicTerms());
 	}
 
-	private PathValue<SecurityFinancePayoutBuilder> getSecPO(PathValue<TradeStateBuilder> ts) {
+	private PathValue<AssetPayout.AssetPayoutBuilder> getSecPO(PathValue<TradeStateBuilder> ts) {
 		PathValue<EconomicTermsBuilder> et = getEcTerms(ts);
 		return new PathValue<>(et.getModelPath().addElement("payout").addElement("securityFinancePayout", 0),
-				et.getValue().getOrCreatePayout().getOrCreateSecurityFinancePayout(0));
+				et.getValue().getOrCreatePayout().getOrCreateAssetPayout(0));
+	}
+
+	private PathValue<CollateralProvisions.CollateralProvisionsBuilder> getColPro(PathValue<TradeStateBuilder> ts) {
+		PathValue<EconomicTermsBuilder> et = getEcTerms(ts);
+		return new PathValue<>(et.getModelPath().addElement("collateral").addElement("collateralProvisions", 0),
+				et.getValue().getOrCreateCollateral().getOrCreateCollateralProvisions());
 	}
 
 	private PathValue<PriceQuantityBuilder> getPriceQuantityForInterestRatePayout(PathValue<TradeStateBuilder> ts) {

rosetta-source/src/main/java/cdm/security/lending/functions/SettlementFunctionHelper.java

@@ -74,13 +74,13 @@ public LocalDate farSettlementDate(BusinessEvent businessEvent) {
         return settlementDate(businessEvent, Iterables::getLast);
     }
 
-    private LocalDate settlementDate(BusinessEvent businessEvent, Function<List<? extends SecurityFinanceLeg>, SecurityFinanceLeg> financeLegSelector) {
+    private LocalDate settlementDate(BusinessEvent businessEvent, Function<List<? extends AssetLeg>, AssetLeg> assetLegSelector) {
         return getSecurityPayout(businessEvent)
-                .map(Payout::getSecurityFinancePayout)
+                .map(Payout::getAssetPayout)
                 .filter(x -> !x.isEmpty()).map(Iterables::getLast)
-                .map(SecurityFinancePayout::getSecurityFinanceLeg)
-                .filter(x -> !x.isEmpty()).map(financeLegSelector)
-                .map(SecurityFinanceLeg::getSettlementDate)
+                .map(AssetPayout::getAssetLeg)
+                .filter(x -> !x.isEmpty()).map(assetLegSelector)
+                .map(AssetLeg::getSettlementDate)
                 .map(AdjustableOrRelativeDate::getAdjustableDate)
                 .map(AdjustableDate::getAdjustedDate)
                 .map(FieldWithMetaDate::getValue)
@@ -145,14 +145,14 @@ private Quantity getShareQuantity(List<? extends Quantity> quantities) {
 
     private Optional<PayerReceiver> getPayerReceiver(Payout payout) {
         return Optional.ofNullable(payout)
-                .map(Payout::getSecurityFinancePayout)
+                .map(Payout::getAssetPayout)
                 .filter(x -> !x.isEmpty()).map(Iterables::getLast)
-                .map(SecurityFinancePayout::getPayerReceiver);
+                .map(AssetPayout::getPayerReceiver);
     }
 
     private Optional<PayerReceiver> getReturnPayerReceiver(Payout payout) {
         return Optional.ofNullable(payout)
-                .map(Payout::getSecurityFinancePayout)
+                .map(Payout::getAssetPayout)
                 .filter(x -> !x.isEmpty()).map(Iterables::getLast)
                 .map(x -> invert(x.getPayerReceiver()));
     }
@@ -176,8 +176,8 @@ private Optional<Payout> getPayout(BusinessEvent executionBusinessEvent) {
 
     private Optional<Payout> getSecurityPayout(BusinessEvent executionBusinessEvent) {
         return getPayout(executionBusinessEvent)
-                .map(Payout::getSecurityFinancePayout)
-                .map(securityFinancePayouts -> Payout.builder().setSecurityFinancePayout(securityFinancePayouts)
+                .map(Payout::getAssetPayout)
+                .map(securityFinancePayouts -> Payout.builder().setAssetPayout(securityFinancePayouts)
                         .build());
     }
 

rosetta-source/src/main/java/org/isda/cdm/qualify/EconomicTermsQualificationHandler.java

@@ -54,12 +54,12 @@ public void setQualifier(ContractualProduct.ContractualProductBuilder contractua
         if (productTaxonomyBuilder != null) {
             productTaxonomyBuilder
                     .setProductQualifier(qualifier)
-                    .setTaxonomySource(TaxonomySourceEnum.ISDA);
+                    .setSource(TaxonomySourceEnum.ISDA);
         } else {
             // Or add new ProductTaxonomy
             contractualProductBuilder.addProductTaxonomy(ProductTaxonomy.builder()
                     .setProductQualifier(qualifier)
-                    .setTaxonomySource(TaxonomySourceEnum.ISDA));
+                    .setSource(TaxonomySourceEnum.ISDA));
         }
     }
 }

rosetta-source/src/main/resources/cdm-sample-files/fis/expectations.json

@@ -1,2 +1,2 @@
 [
-{"fileName":"cdm-sample-files/fis/isla1.xml","excludedPaths":0,"externalPaths":120,"outstandingMappings":10,"validationFailures":31,"qualificationExpectation":{"success":false,"qualifyResults":[{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"}],"qualifiableObjectCount":7}}]
+{"fileName":"cdm-sample-files/fis/isla1.xml","excludedPaths":0,"externalPaths":120,"outstandingMappings":10,"validationFailures":37,"qualificationExpectation":{"success":false,"qualifyResults":[{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"},{"qualifiedName":"SecurityLendingAgreement","qualifiedObjectClass":"cdm.product.template.EconomicTerms"}],"qualifiableObjectCount":7}}]