== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=11.9 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (9)
+- Exchange (8)
+- ObjectHashAggregate (7)
+- Project (2)
+- Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log [codegen id : 1]
Output [10]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, cdc#1082, checkpointMetadata#1083, sidecar#1084, domainMetadata#1085, commitInfo#1086]
Arguments: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, cdc#1082, checkpointMetadata#1083, sidecar#1084, domainMetadata#1085, commitInfo#1086], Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log MapPartitionsRDD[61] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]
Input [10]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, cdc#1082, checkpointMetadata#1083, sidecar#1084, domainMetadata#1085, commitInfo#1086]
(3) ObjectHashAggregate
Input [6]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#1085, 0, 0), partial_collect_set(txn#1077, 0, 0), partial_count(protocol#1081), partial_sum(add#1078.size), partial_last(metaData#1080, true), partial_count(metaData#1080), partial_last(protocol#1081, true), partial_count(remove#1079), partial_count(add#1078), partial_count(txn#1077)]
Aggregate Attributes [12]: [buf#20382, buf#20383, count#20384L, sum#20385L, last#20386, valueSet#20387, count#20388L, last#20389, valueSet#20390, count#20391L, count#20392L, count#20393L]
Results [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
(4) Exchange
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=4797]
(5) ShuffleQueryStage
Output [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Keys: []
Functions [10]: [collect_set(domainMetadata#1085, 0, 0), collect_set(txn#1077, 0, 0), count(protocol#1081), sum(add#1078.size), last(metaData#1080, true), count(metaData#1080), last(protocol#1081, true), count(remove#1079), count(add#1078), count(txn#1077)]
Aggregate Attributes [10]: [collect_set(domainMetadata#1085, 0, 0)#20289, collect_set(txn#1077, 0, 0)#20288, count(protocol#1081)#20287L, sum(add#1078.size)#20282L, last(metaData#1080) ignore nulls#20290, count(metaData#1080)#20286L, last(protocol#1081) ignore nulls#20291, count(remove#1079)#20285L, count(add#1078)#20284L, count(txn#1077)#20283L]
Results [14]: [collect_set(domainMetadata#1085, 0, 0)#20289 AS domainMetadata#20295, null AS fileSizeHistogram#20296, collect_set(txn#1077, 0, 0)#20288 AS setTransactions#20297, null AS numDeletedRecordsOpt#20298, count(protocol#1081)#20287L AS numOfProtocol#20299L, coalesce(sum(add#1078.size)#20282L, 0) AS sizeInBytes#20300L, last(metaData#1080) ignore nulls#20290 AS metadata#20301, count(metaData#1080)#20286L AS numOfMetadata#20302L, null AS deletedRecordCountsHistogramOpt#20303, null AS numDeletionVectorsOpt#20304, last(protocol#1081) ignore nulls#20291 AS protocol#20305, count(remove#1079)#20285L AS numOfRemoves#20306L, count(add#1078)#20284L AS numOfFiles#20307L, count(txn#1077)#20283L AS numOfSetTransactions#20308L]
(7) ObjectHashAggregate
Input [6]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#1085, 0, 0), partial_collect_set(txn#1077, 0, 0), partial_count(protocol#1081), partial_sum(add#1078.size), partial_last(metaData#1080, true), partial_count(metaData#1080), partial_last(protocol#1081, true), partial_count(remove#1079), partial_count(add#1078), partial_count(txn#1077)]
Aggregate Attributes [12]: [buf#20382, buf#20383, count#20384L, sum#20385L, last#20386, valueSet#20387, count#20388L, last#20389, valueSet#20390, count#20391L, count#20392L, count#20393L]
Results [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
(8) Exchange
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=4787]
(9) ObjectHashAggregate
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Keys: []
Functions [10]: [collect_set(domainMetadata#1085, 0, 0), collect_set(txn#1077, 0, 0), count(protocol#1081), sum(add#1078.size), last(metaData#1080, true), count(metaData#1080), last(protocol#1081, true), count(remove#1079), count(add#1078), count(txn#1077)]
Aggregate Attributes [10]: [collect_set(domainMetadata#1085, 0, 0)#20289, collect_set(txn#1077, 0, 0)#20288, count(protocol#1081)#20287L, sum(add#1078.size)#20282L, last(metaData#1080) ignore nulls#20290, count(metaData#1080)#20286L, last(protocol#1081) ignore nulls#20291, count(remove#1079)#20285L, count(add#1078)#20284L, count(txn#1077)#20283L]
Results [14]: [collect_set(domainMetadata#1085, 0, 0)#20289 AS domainMetadata#20295, null AS fileSizeHistogram#20296, collect_set(txn#1077, 0, 0)#20288 AS setTransactions#20297, null AS numDeletedRecordsOpt#20298, count(protocol#1081)#20287L AS numOfProtocol#20299L, coalesce(sum(add#1078.size)#20282L, 0) AS sizeInBytes#20300L, last(metaData#1080) ignore nulls#20290 AS metadata#20301, count(metaData#1080)#20286L AS numOfMetadata#20302L, null AS deletedRecordCountsHistogramOpt#20303, null AS numDeletionVectorsOpt#20304, last(protocol#1081) ignore nulls#20291 AS protocol#20305, count(remove#1079)#20285L AS numOfRemoves#20306L, count(add#1078)#20284L AS numOfFiles#20307L, count(txn#1077)#20283L AS numOfSetTransactions#20308L]
(10) AdaptiveSparkPlan
Output [14]: [domainMetadata#20295, fileSizeHistogram#20296, setTransactions#20297, numDeletedRecordsOpt#20298, numOfProtocol#20299L, sizeInBytes#20300L, metadata#20301, numOfMetadata#20302L, deletedRecordCountsHistogramOpt#20303, numDeletionVectorsOpt#20304, protocol#20305, numOfRemoves#20306L, numOfFiles#20307L, numOfSetTransactions#20308L]
Arguments: isFinalPlan=true