crp-order-qty-opt-service - Details for Query 53

Submitted Time: 2026/02/06 06:56:00
Duration: 0.4 s
Succeeded Jobs: 104 105

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="ObjectHashAggregate number of output rows: 1 time in aggregation build: 36 ms spill size: 0.0 B number of sort fallback tasks: 0"]; 2 [labelType="html" label="Exchange shuffle records written: 50 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 11 ms (0 ms, 0 ms, 0 ms (stage 206.0: task 1102)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 50 local bytes read: 2.6 KiB fetch wait time: 0 ms remote bytes read: 3.2 KiB merged fetch fallback count: 0 local blocks read: 26 remote merged chunks fetched: 0 remote blocks read: 24 data size total (min, med, max (stageId: taskId)) 11.9 KiB (152.0 B, 152.0 B, 4.6 KiB (stage 206.0: task 1144)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 5 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 5.8 KiB (101.0 B, 101.0 B, 923.0 B (stage 206.0: task 1144))"]; 3 [labelType="html" label="ObjectHashAggregate number of output rows: 50 time in aggregation build total (min, med, max (stageId: taskId)) 858 ms (6 ms, 10 ms, 47 ms (stage 206.0: task 1108)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 206.0: task 1103)) number of sort fallback tasks: 0"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n754 ms (5 ms, 9 ms, 37 ms (stage 206.0: task 1108))"; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log number of output rows: 16"]; } 1->0; 2->1; 3->2; 5->3; 6->5; }

AdaptiveSparkPlan isFinalPlan=true

ObjectHashAggregate(keys=[], functions=[collect_set(domainMetadata#1085, 0, 0), collect_set(txn#1077, 0, 0), count(protocol#1081), sum(add#1078.size), last(metaData#1080, true), count(metaData#1080), last(protocol#1081, true), count(remove#1079), count(add#1078), count(txn#1077)])

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=4797]

ObjectHashAggregate(keys=[], functions=[partial_collect_set(domainMetadata#1085, 0, 0), partial_collect_set(txn#1077, 0, 0), partial_count(protocol#1081), partial_sum(add#1078.size), partial_last(metaData#1080, true), partial_count(metaData#1080), partial_last(protocol#1081, true), partial_count(remove#1079), partial_count(add#1078), partial_count(txn#1077)])

Project [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]

Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log[txn#1077,add#1078,remove#1079,metaData#1080,protocol#1081,cdc#1082,checkpointMetadata#1083,sidecar#1084,domainMetadata#1085,commitInfo#1086]

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
   ObjectHashAggregate (6)
   +- ShuffleQueryStage (5), Statistics(sizeInBytes=11.9 KiB, rowCount=50)
      +- Exchange (4)
         +- ObjectHashAggregate (3)
            +- * Project (2)
               +- * Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log (1)
+- == Initial Plan ==
   ObjectHashAggregate (9)
   +- Exchange (8)
      +- ObjectHashAggregate (7)
         +- Project (2)
            +- Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log (1)


(1) Scan ExistingRDD Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log [codegen id : 1]
Output [10]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, cdc#1082, checkpointMetadata#1083, sidecar#1084, domainMetadata#1085, commitInfo#1086]
Arguments: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, cdc#1082, checkpointMetadata#1083, sidecar#1084, domainMetadata#1085, commitInfo#1086], Delta Table State #143 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-sourceofsupply-sourceofsupply/_delta_log MapPartitionsRDD[61] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)

(2) Project [codegen id : 1]
Output [6]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]
Input [10]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, cdc#1082, checkpointMetadata#1083, sidecar#1084, domainMetadata#1085, commitInfo#1086]

(3) ObjectHashAggregate
Input [6]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#1085, 0, 0), partial_collect_set(txn#1077, 0, 0), partial_count(protocol#1081), partial_sum(add#1078.size), partial_last(metaData#1080, true), partial_count(metaData#1080), partial_last(protocol#1081, true), partial_count(remove#1079), partial_count(add#1078), partial_count(txn#1077)]
Aggregate Attributes [12]: [buf#20382, buf#20383, count#20384L, sum#20385L, last#20386, valueSet#20387, count#20388L, last#20389, valueSet#20390, count#20391L, count#20392L, count#20393L]
Results [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]

(4) Exchange
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=4797]

(5) ShuffleQueryStage
Output [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Arguments: 0

(6) ObjectHashAggregate
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Keys: []
Functions [10]: [collect_set(domainMetadata#1085, 0, 0), collect_set(txn#1077, 0, 0), count(protocol#1081), sum(add#1078.size), last(metaData#1080, true), count(metaData#1080), last(protocol#1081, true), count(remove#1079), count(add#1078), count(txn#1077)]
Aggregate Attributes [10]: [collect_set(domainMetadata#1085, 0, 0)#20289, collect_set(txn#1077, 0, 0)#20288, count(protocol#1081)#20287L, sum(add#1078.size)#20282L, last(metaData#1080) ignore nulls#20290, count(metaData#1080)#20286L, last(protocol#1081) ignore nulls#20291, count(remove#1079)#20285L, count(add#1078)#20284L, count(txn#1077)#20283L]
Results [14]: [collect_set(domainMetadata#1085, 0, 0)#20289 AS domainMetadata#20295, null AS fileSizeHistogram#20296, collect_set(txn#1077, 0, 0)#20288 AS setTransactions#20297, null AS numDeletedRecordsOpt#20298, count(protocol#1081)#20287L AS numOfProtocol#20299L, coalesce(sum(add#1078.size)#20282L, 0) AS sizeInBytes#20300L, last(metaData#1080) ignore nulls#20290 AS metadata#20301, count(metaData#1080)#20286L AS numOfMetadata#20302L, null AS deletedRecordCountsHistogramOpt#20303, null AS numDeletionVectorsOpt#20304, last(protocol#1081) ignore nulls#20291 AS protocol#20305, count(remove#1079)#20285L AS numOfRemoves#20306L, count(add#1078)#20284L AS numOfFiles#20307L, count(txn#1077)#20283L AS numOfSetTransactions#20308L]

(7) ObjectHashAggregate
Input [6]: [txn#1077, add#1078, remove#1079, metaData#1080, protocol#1081, domainMetadata#1085]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#1085, 0, 0), partial_collect_set(txn#1077, 0, 0), partial_count(protocol#1081), partial_sum(add#1078.size), partial_last(metaData#1080, true), partial_count(metaData#1080), partial_last(protocol#1081, true), partial_count(remove#1079), partial_count(add#1078), partial_count(txn#1077)]
Aggregate Attributes [12]: [buf#20382, buf#20383, count#20384L, sum#20385L, last#20386, valueSet#20387, count#20388L, last#20389, valueSet#20390, count#20391L, count#20392L, count#20393L]
Results [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]

(8) Exchange
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=4787]

(9) ObjectHashAggregate
Input [12]: [buf#20394, buf#20395, count#20396L, sum#20397L, last#20398, valueSet#20399, count#20400L, last#20401, valueSet#20402, count#20403L, count#20404L, count#20405L]
Keys: []
Functions [10]: [collect_set(domainMetadata#1085, 0, 0), collect_set(txn#1077, 0, 0), count(protocol#1081), sum(add#1078.size), last(metaData#1080, true), count(metaData#1080), last(protocol#1081, true), count(remove#1079), count(add#1078), count(txn#1077)]
Aggregate Attributes [10]: [collect_set(domainMetadata#1085, 0, 0)#20289, collect_set(txn#1077, 0, 0)#20288, count(protocol#1081)#20287L, sum(add#1078.size)#20282L, last(metaData#1080) ignore nulls#20290, count(metaData#1080)#20286L, last(protocol#1081) ignore nulls#20291, count(remove#1079)#20285L, count(add#1078)#20284L, count(txn#1077)#20283L]
Results [14]: [collect_set(domainMetadata#1085, 0, 0)#20289 AS domainMetadata#20295, null AS fileSizeHistogram#20296, collect_set(txn#1077, 0, 0)#20288 AS setTransactions#20297, null AS numDeletedRecordsOpt#20298, count(protocol#1081)#20287L AS numOfProtocol#20299L, coalesce(sum(add#1078.size)#20282L, 0) AS sizeInBytes#20300L, last(metaData#1080) ignore nulls#20290 AS metadata#20301, count(metaData#1080)#20286L AS numOfMetadata#20302L, null AS deletedRecordCountsHistogramOpt#20303, null AS numDeletionVectorsOpt#20304, last(protocol#1081) ignore nulls#20291 AS protocol#20305, count(remove#1079)#20285L AS numOfRemoves#20306L, count(add#1078)#20284L AS numOfFiles#20307L, count(txn#1077)#20283L AS numOfSetTransactions#20308L]

(10) AdaptiveSparkPlan
Output [14]: [domainMetadata#20295, fileSizeHistogram#20296, setTransactions#20297, numDeletedRecordsOpt#20298, numOfProtocol#20299L, sizeInBytes#20300L, metadata#20301, numOfMetadata#20302L, deletedRecordCountsHistogramOpt#20303, numDeletionVectorsOpt#20304, protocol#20305, numOfRemoves#20306L, numOfFiles#20307L, numOfSetTransactions#20308L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.activity.hdlfsEnabled	true
spark.activity.isolateSession	true
spark.sql.adaptive.applyFinalStageShuffleOptimizations	false
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true
spark.sql.sources.bucketing.autoBucketedScan.enabled	false
temporal.activity.id	4ed5a2f4-9ab7-3eed-9f6f-05c008033ee1
temporal.activity.type	BufferDataSnapShot