crp-order-qty-opt-service - Details for Query 68

Submitted Time: 2026/02/06 06:56:08
Duration: 0.3 s
Succeeded Jobs: 122 123

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="ObjectHashAggregate number of output rows: 1 time in aggregation build: 12 ms spill size: 0.0 B number of sort fallback tasks: 0"]; 2 [labelType="html" label="Exchange shuffle records written: 50 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 10 ms (0 ms, 0 ms, 0 ms (stage 238.0: task 1591)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 50 local bytes read: 3.0 KiB fetch wait time: 0 ms remote bytes read: 2.5 KiB merged fetch fallback count: 0 local blocks read: 25 remote merged chunks fetched: 0 remote blocks read: 25 data size total (min, med, max (stageId: taskId)) 8.6 KiB (152.0 B, 152.0 B, 1376.0 B (stage 238.0: task 1560)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 5 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 5.5 KiB (101.0 B, 101.0 B, 596.0 B (stage 238.0: task 1560))"]; 3 [labelType="html" label="ObjectHashAggregate number of output rows: 50 time in aggregation build total (min, med, max (stageId: taskId)) 405 ms (5 ms, 6 ms, 48 ms (stage 238.0: task 1580)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 238.0: task 1550)) number of sort fallback tasks: 0"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n370 ms (4 ms, 5 ms, 47 ms (stage 238.0: task 1580))"; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="Scan ExistingRDD Delta Table State #29 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/supplier-minimum-profile/_delta_log number of output rows: 6"]; } 1->0; 2->1; 3->2; 5->3; 6->5; }

AdaptiveSparkPlan isFinalPlan=true

ObjectHashAggregate(keys=[], functions=[collect_set(domainMetadata#24463, 0, 0), collect_set(txn#24455, 0, 0), count(protocol#24459), sum(add#24456.size), last(metaData#24458, true), count(metaData#24458), last(protocol#24459, true), count(remove#24457), count(add#24456), count(txn#24455)])

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=5684]

ObjectHashAggregate(keys=[], functions=[partial_collect_set(domainMetadata#24463, 0, 0), partial_collect_set(txn#24455, 0, 0), partial_count(protocol#24459), partial_sum(add#24456.size), partial_last(metaData#24458, true), partial_count(metaData#24458), partial_last(protocol#24459, true), partial_count(remove#24457), partial_count(add#24456), partial_count(txn#24455)])

Project [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, domainMetadata#24463]

Scan ExistingRDD Delta Table State #29 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/supplier-minimum-profile/_delta_log[txn#24455,add#24456,remove#24457,metaData#24458,protocol#24459,cdc#24460,checkpointMetadata#24461,sidecar#24462,domainMetadata#24463,commitInfo#24464]

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
   ObjectHashAggregate (6)
   +- ShuffleQueryStage (5), Statistics(sizeInBytes=8.6 KiB, rowCount=50)
      +- Exchange (4)
         +- ObjectHashAggregate (3)
            +- * Project (2)
               +- * Scan ExistingRDD Delta Table State #29 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/supplier-minimum-profile/_delta_log (1)
+- == Initial Plan ==
   ObjectHashAggregate (9)
   +- Exchange (8)
      +- ObjectHashAggregate (7)
         +- Project (2)
            +- Scan ExistingRDD Delta Table State #29 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/supplier-minimum-profile/_delta_log (1)


(1) Scan ExistingRDD Delta Table State #29 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/supplier-minimum-profile/_delta_log [codegen id : 1]
Output [10]: [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, cdc#24460, checkpointMetadata#24461, sidecar#24462, domainMetadata#24463, commitInfo#24464]
Arguments: [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, cdc#24460, checkpointMetadata#24461, sidecar#24462, domainMetadata#24463, commitInfo#24464], Delta Table State #29 - hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/prp/supplier-minimum-profile/_delta_log MapPartitionsRDD[632] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)

(2) Project [codegen id : 1]
Output [6]: [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, domainMetadata#24463]
Input [10]: [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, cdc#24460, checkpointMetadata#24461, sidecar#24462, domainMetadata#24463, commitInfo#24464]

(3) ObjectHashAggregate
Input [6]: [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, domainMetadata#24463]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#24463, 0, 0), partial_collect_set(txn#24455, 0, 0), partial_count(protocol#24459), partial_sum(add#24456.size), partial_last(metaData#24458, true), partial_count(metaData#24458), partial_last(protocol#24459, true), partial_count(remove#24457), partial_count(add#24456), partial_count(txn#24455)]
Aggregate Attributes [12]: [buf#24602, buf#24603, count#24604L, sum#24605L, last#24606, valueSet#24607, count#24608L, last#24609, valueSet#24610, count#24611L, count#24612L, count#24613L]
Results [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]

(4) Exchange
Input [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=5684]

(5) ShuffleQueryStage
Output [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]
Arguments: 0

(6) ObjectHashAggregate
Input [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]
Keys: []
Functions [10]: [collect_set(domainMetadata#24463, 0, 0), collect_set(txn#24455, 0, 0), count(protocol#24459), sum(add#24456.size), last(metaData#24458, true), count(metaData#24458), last(protocol#24459, true), count(remove#24457), count(add#24456), count(txn#24455)]
Aggregate Attributes [10]: [collect_set(domainMetadata#24463, 0, 0)#24526, collect_set(txn#24455, 0, 0)#24525, count(protocol#24459)#24524L, sum(add#24456.size)#24519L, last(metaData#24458) ignore nulls#24527, count(metaData#24458)#24523L, last(protocol#24459) ignore nulls#24528, count(remove#24457)#24522L, count(add#24456)#24521L, count(txn#24455)#24520L]
Results [14]: [collect_set(domainMetadata#24463, 0, 0)#24526 AS domainMetadata#24529, null AS fileSizeHistogram#24530, collect_set(txn#24455, 0, 0)#24525 AS setTransactions#24531, null AS numDeletedRecordsOpt#24532, count(protocol#24459)#24524L AS numOfProtocol#24533L, coalesce(sum(add#24456.size)#24519L, 0) AS sizeInBytes#24534L, last(metaData#24458) ignore nulls#24527 AS metadata#24535, count(metaData#24458)#24523L AS numOfMetadata#24536L, null AS deletedRecordCountsHistogramOpt#24537, null AS numDeletionVectorsOpt#24538, last(protocol#24459) ignore nulls#24528 AS protocol#24539, count(remove#24457)#24522L AS numOfRemoves#24540L, count(add#24456)#24521L AS numOfFiles#24541L, count(txn#24455)#24520L AS numOfSetTransactions#24542L]

(7) ObjectHashAggregate
Input [6]: [txn#24455, add#24456, remove#24457, metaData#24458, protocol#24459, domainMetadata#24463]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#24463, 0, 0), partial_collect_set(txn#24455, 0, 0), partial_count(protocol#24459), partial_sum(add#24456.size), partial_last(metaData#24458, true), partial_count(metaData#24458), partial_last(protocol#24459, true), partial_count(remove#24457), partial_count(add#24456), partial_count(txn#24455)]
Aggregate Attributes [12]: [buf#24602, buf#24603, count#24604L, sum#24605L, last#24606, valueSet#24607, count#24608L, last#24609, valueSet#24610, count#24611L, count#24612L, count#24613L]
Results [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]

(8) Exchange
Input [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=5674]

(9) ObjectHashAggregate
Input [12]: [buf#24614, buf#24615, count#24616L, sum#24617L, last#24618, valueSet#24619, count#24620L, last#24621, valueSet#24622, count#24623L, count#24624L, count#24625L]
Keys: []
Functions [10]: [collect_set(domainMetadata#24463, 0, 0), collect_set(txn#24455, 0, 0), count(protocol#24459), sum(add#24456.size), last(metaData#24458, true), count(metaData#24458), last(protocol#24459, true), count(remove#24457), count(add#24456), count(txn#24455)]
Aggregate Attributes [10]: [collect_set(domainMetadata#24463, 0, 0)#24526, collect_set(txn#24455, 0, 0)#24525, count(protocol#24459)#24524L, sum(add#24456.size)#24519L, last(metaData#24458) ignore nulls#24527, count(metaData#24458)#24523L, last(protocol#24459) ignore nulls#24528, count(remove#24457)#24522L, count(add#24456)#24521L, count(txn#24455)#24520L]
Results [14]: [collect_set(domainMetadata#24463, 0, 0)#24526 AS domainMetadata#24529, null AS fileSizeHistogram#24530, collect_set(txn#24455, 0, 0)#24525 AS setTransactions#24531, null AS numDeletedRecordsOpt#24532, count(protocol#24459)#24524L AS numOfProtocol#24533L, coalesce(sum(add#24456.size)#24519L, 0) AS sizeInBytes#24534L, last(metaData#24458) ignore nulls#24527 AS metadata#24535, count(metaData#24458)#24523L AS numOfMetadata#24536L, null AS deletedRecordCountsHistogramOpt#24537, null AS numDeletionVectorsOpt#24538, last(protocol#24459) ignore nulls#24528 AS protocol#24539, count(remove#24457)#24522L AS numOfRemoves#24540L, count(add#24456)#24521L AS numOfFiles#24541L, count(txn#24455)#24520L AS numOfSetTransactions#24542L]

(10) AdaptiveSparkPlan
Output [14]: [domainMetadata#24529, fileSizeHistogram#24530, setTransactions#24531, numDeletedRecordsOpt#24532, numOfProtocol#24533L, sizeInBytes#24534L, metadata#24535, numOfMetadata#24536L, deletedRecordCountsHistogramOpt#24537, numDeletionVectorsOpt#24538, protocol#24539, numOfRemoves#24540L, numOfFiles#24541L, numOfSetTransactions#24542L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.activity.hdlfsEnabled	true
spark.activity.isolateSession	true
spark.sql.adaptive.applyFinalStageShuffleOptimizations	false
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true
spark.sql.sources.bucketing.autoBucketedScan.enabled	false
temporal.activity.id	4ed5a2f4-9ab7-3eed-9f6f-05c008033ee1
temporal.activity.type	BufferDataSnapShot