== Physical Plan ==
AdaptiveSparkPlan (11)
+- == Final Plan ==
ResultQueryStage (7), Statistics(sizeInBytes=8.0 EiB)
+- ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=8.6 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #2 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-common-unitofmeasurecodes-v2/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (10)
+- Exchange (9)
+- ObjectHashAggregate (8)
+- Project (2)
+- Scan ExistingRDD Delta Table State #2 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-common-unitofmeasurecodes-v2/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #2 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-common-unitofmeasurecodes-v2/_delta_log [codegen id : 1]
Output [10]: [txn#827190, add#827191, remove#827192, metaData#827193, protocol#827194, cdc#827195, checkpointMetadata#827196, sidecar#827197, domainMetadata#827198, commitInfo#827199]
Arguments: [txn#827190, add#827191, remove#827192, metaData#827193, protocol#827194, cdc#827195, checkpointMetadata#827196, sidecar#827197, domainMetadata#827198, commitInfo#827199], Delta Table State #2 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-common-unitofmeasurecodes-v2/_delta_log MapPartitionsRDD[38885] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#827190, add#827191, remove#827192, metaData#827193, protocol#827194, domainMetadata#827198]
Input [10]: [txn#827190, add#827191, remove#827192, metaData#827193, protocol#827194, cdc#827195, checkpointMetadata#827196, sidecar#827197, domainMetadata#827198, commitInfo#827199]
(3) ObjectHashAggregate
Input [6]: [txn#827190, add#827191, remove#827192, metaData#827193, protocol#827194, domainMetadata#827198]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#827198, 0, 0), partial_collect_set(txn#827190, 0, 0), partial_count(protocol#827194), partial_sum(add#827191.size), partial_last(metaData#827193, true), partial_count(metaData#827193), partial_last(protocol#827194, true), partial_count(remove#827192), partial_count(add#827191), partial_count(txn#827190)]
Aggregate Attributes [12]: [buf#827300, buf#827301, count#827302L, sum#827303L, last#827304, valueSet#827305, count#827306L, last#827307, valueSet#827308, count#827309L, count#827310L, count#827311L]
Results [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
(4) Exchange
Input [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=325390]
(5) ShuffleQueryStage
Output [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
Keys: []
Functions [10]: [collect_set(domainMetadata#827198, 0, 0), collect_set(txn#827190, 0, 0), count(protocol#827194), sum(add#827191.size), last(metaData#827193, true), count(metaData#827193), last(protocol#827194, true), count(remove#827192), count(add#827191), count(txn#827190)]
Aggregate Attributes [10]: [collect_set(domainMetadata#827198, 0, 0)#827246, collect_set(txn#827190, 0, 0)#827247, count(protocol#827194)#827248L, sum(add#827191.size)#827249L, last(metaData#827193) ignore nulls#827250, count(metaData#827193)#827251L, last(protocol#827194) ignore nulls#827252, count(remove#827192)#827253L, count(add#827191)#827254L, count(txn#827190)#827255L]
Results [14]: [collect_set(domainMetadata#827198, 0, 0)#827246 AS domainMetadata#827231, null AS fileSizeHistogram#827232, collect_set(txn#827190, 0, 0)#827247 AS setTransactions#827233, null AS numDeletedRecordsOpt#827234, count(protocol#827194)#827248L AS numOfProtocol#827235L, coalesce(sum(add#827191.size)#827249L, 0) AS sizeInBytes#827236L, last(metaData#827193) ignore nulls#827250 AS metadata#827237, count(metaData#827193)#827251L AS numOfMetadata#827238L, null AS deletedRecordCountsHistogramOpt#827239, null AS numDeletionVectorsOpt#827240, last(protocol#827194) ignore nulls#827252 AS protocol#827241, count(remove#827192)#827253L AS numOfRemoves#827242L, count(add#827191)#827254L AS numOfFiles#827243L, count(txn#827190)#827255L AS numOfSetTransactions#827244L]
(7) ResultQueryStage
Output [14]: [domainMetadata#827231, fileSizeHistogram#827232, setTransactions#827233, numDeletedRecordsOpt#827234, numOfProtocol#827235L, sizeInBytes#827236L, metadata#827237, numOfMetadata#827238L, deletedRecordCountsHistogramOpt#827239, numDeletionVectorsOpt#827240, protocol#827241, numOfRemoves#827242L, numOfFiles#827243L, numOfSetTransactions#827244L]
Arguments: 1
(8) ObjectHashAggregate
Input [6]: [txn#827190, add#827191, remove#827192, metaData#827193, protocol#827194, domainMetadata#827198]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#827198, 0, 0), partial_collect_set(txn#827190, 0, 0), partial_count(protocol#827194), partial_sum(add#827191.size), partial_last(metaData#827193, true), partial_count(metaData#827193), partial_last(protocol#827194, true), partial_count(remove#827192), partial_count(add#827191), partial_count(txn#827190)]
Aggregate Attributes [12]: [buf#827300, buf#827301, count#827302L, sum#827303L, last#827304, valueSet#827305, count#827306L, last#827307, valueSet#827308, count#827309L, count#827310L, count#827311L]
Results [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
(9) Exchange
Input [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=325379]
(10) ObjectHashAggregate
Input [12]: [buf#827312, buf#827313, count#827314L, sum#827315L, last#827316, valueSet#827317, count#827318L, last#827319, valueSet#827320, count#827321L, count#827322L, count#827323L]
Keys: []
Functions [10]: [collect_set(domainMetadata#827198, 0, 0), collect_set(txn#827190, 0, 0), count(protocol#827194), sum(add#827191.size), last(metaData#827193, true), count(metaData#827193), last(protocol#827194, true), count(remove#827192), count(add#827191), count(txn#827190)]
Aggregate Attributes [10]: [collect_set(domainMetadata#827198, 0, 0)#827246, collect_set(txn#827190, 0, 0)#827247, count(protocol#827194)#827248L, sum(add#827191.size)#827249L, last(metaData#827193) ignore nulls#827250, count(metaData#827193)#827251L, last(protocol#827194) ignore nulls#827252, count(remove#827192)#827253L, count(add#827191)#827254L, count(txn#827190)#827255L]
Results [14]: [collect_set(domainMetadata#827198, 0, 0)#827246 AS domainMetadata#827231, null AS fileSizeHistogram#827232, collect_set(txn#827190, 0, 0)#827247 AS setTransactions#827233, null AS numDeletedRecordsOpt#827234, count(protocol#827194)#827248L AS numOfProtocol#827235L, coalesce(sum(add#827191.size)#827249L, 0) AS sizeInBytes#827236L, last(metaData#827193) ignore nulls#827250 AS metadata#827237, count(metaData#827193)#827251L AS numOfMetadata#827238L, null AS deletedRecordCountsHistogramOpt#827239, null AS numDeletionVectorsOpt#827240, last(protocol#827194) ignore nulls#827252 AS protocol#827241, count(remove#827192)#827253L AS numOfRemoves#827242L, count(add#827191)#827254L AS numOfFiles#827243L, count(txn#827190)#827255L AS numOfSetTransactions#827244L]
(11) AdaptiveSparkPlan
Output [14]: [domainMetadata#827231, fileSizeHistogram#827232, setTransactions#827233, numDeletedRecordsOpt#827234, numOfProtocol#827235L, sizeInBytes#827236L, metadata#827237, numOfMetadata#827238L, deletedRecordCountsHistogramOpt#827239, numDeletionVectorsOpt#827240, protocol#827241, numOfRemoves#827242L, numOfFiles#827243L, numOfSetTransactions#827244L]
Arguments: isFinalPlan=true