digraph G {
0 [labelType="html" label="<br><b>AdaptiveSparkPlan</b><br><br>"];
1 [labelType="html" label="<b>Execute InsertIntoHadoopFsRelationCommand</b><br><br>task commit time total (min, med, max (stageId: taskId))<br>497 ms (243 ms, 254 ms, 254 ms (stage 140.0: task 718))<br>number of written files: 2<br>job commit time: 747 ms<br>number of output rows: 179<br>number of dynamic part: 0<br>written output: 5.5 KiB"];
2 [labelType="html" label="<br><b>WriteFiles</b><br><br>"];
subgraph cluster3 {
isCluster="true";
label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n267 ms (87 ms, 180 ms, 180 ms (stage 140.0: task 717))";
4 [labelType="html" label="<b>HashAggregate</b><br><br>spill size: 0.0 B<br>time in aggregation build total (min, med, max (stageId: taskId))<br>76 ms (31 ms, 45 ms, 45 ms (stage 140.0: task 718))<br>peak memory total (min, med, max (stageId: taskId))<br>4.5 MiB (2.3 MiB, 2.3 MiB, 2.3 MiB (stage 140.0: task 718))<br>number of output rows: 179<br>number of sort fallback tasks: 0<br>avg hash probes per key (min, med, max (stageId: taskId)):<br>(1, 1, 1 (stage 140.0: task 718))"];
}
5 [labelType="html" label="<b>AQEShuffleRead</b><br><br>number of partitions: 2<br>partition data size total (min, med, max (stageId: taskId))<br>132.4 KiB (52.2 KiB, 80.2 KiB, 80.2 KiB (driver))<br>number of coalesced partitions: 2"];
6 [labelType="html" label="<b>Exchange</b><br><br>shuffle records written: 1,032<br>local merged chunks fetched: 0<br>shuffle write time total (min, med, max (stageId: taskId))<br>43 ms (1 ms, 1 ms, 2 ms (stage 135.0: task 696))<br>remote merged bytes read: 0.0 B<br>local merged blocks fetched: 0<br>corrupt merged block chunks: 0<br>remote merged reqs duration: 0 ms<br>remote merged blocks fetched: 0<br>records read: 1,032<br>local bytes read total (min, med, max (stageId: taskId))<br>125.8 KiB (49.5 KiB, 76.2 KiB, 76.2 KiB (stage 140.0: task 718))<br>fetch wait time total (min, med, max (stageId: taskId))<br>0 ms (0 ms, 0 ms, 0 ms (stage 140.0: task 718))<br>remote bytes read: 0.0 B<br>merged fetch fallback count: 0<br>local blocks read: 497<br>remote merged chunks fetched: 0<br>remote blocks read: 0<br>data size total (min, med, max (stageId: taskId))<br>126.3 KiB (4.0 KiB, 5.0 KiB, 6.1 KiB (stage 135.0: task 702))<br>local merged bytes read: 0.0 B<br>number of partitions: 25<br>remote reqs duration: 0 ms<br>remote bytes read to disk: 0.0 B<br>shuffle bytes written total (min, med, max (stageId: taskId))<br>125.8 KiB (4.1 KiB, 5.0 KiB, 5.9 KiB (stage 135.0: task 696))"];
subgraph cluster7 {
isCluster="true";
label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n266 ms (4 ms, 8 ms, 35 ms (stage 135.0: task 712))";
8 [labelType="html" label="<b>HashAggregate</b><br><br>spill size: 0.0 B<br>time in aggregation build total (min, med, max (stageId: taskId))<br>6 ms (0 ms, 0 ms, 1 ms (stage 135.0: task 693))<br>peak memory total (min, med, max (stageId: taskId))<br>56.3 MiB (2.3 MiB, 2.3 MiB, 2.3 MiB (stage 135.0: task 693))<br>number of output rows: 1,032<br>number of sort fallback tasks: 0<br>avg hash probes per key (min, med, max (stageId: taskId)):<br>(1, 1, 1 (stage 135.0: task 693))"];
9 [labelType="html" label="<br><b>Project</b><br><br>"];
}
10 [labelType="html" label="<b>Exchange</b><br><br>shuffle records written: 1,196<br>local merged chunks fetched: 0<br>shuffle write time: 1 ms<br>remote merged bytes read: 0.0 B<br>local merged blocks fetched: 0<br>corrupt merged block chunks: 0<br>remote merged reqs duration: 0 ms<br>remote merged blocks fetched: 0<br>records read: 1,196<br>local bytes read total (min, med, max (stageId: taskId))<br>66.6 KiB (2.2 KiB, 2.7 KiB, 3.0 KiB (stage 135.0: task 690))<br>fetch wait time total (min, med, max (stageId: taskId))<br>0 ms (0 ms, 0 ms, 0 ms (stage 135.0: task 693))<br>remote bytes read: 0.0 B<br>merged fetch fallback count: 0<br>local blocks read: 25<br>remote merged chunks fetched: 0<br>remote blocks read: 0<br>data size: 193.0 KiB<br>local merged bytes read: 0.0 B<br>number of partitions: 25<br>remote reqs duration: 0 ms<br>remote bytes read to disk: 0.0 B<br>shuffle bytes written: 66.6 KiB"];
subgraph cluster11 {
isCluster="true";
label="WholeStageCodegen (1)\n \nduration: 139 ms";
12 [labelType="html" label="<b>ColumnarToRow</b><br><br>number of output rows: 1,196<br>number of input batches: 1"];
}
13 [labelType="html" label="<b>Scan parquet </b><br><br>number of files read: 1<br>scan time: 124 ms<br>metadata time: 0 ms<br>size of files read: 37.4 KiB<br>number of output rows: 1,196"];
1->0;
2->1;
4->2;
5->4;
6->5;
8->6;
9->8;
10->9;
12->10;
13->12;
}
14
AdaptiveSparkPlan isFinalPlan=true
Execute InsertIntoHadoopFsRelationCommand hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/supply-network-list/10000000095/shardId=0_1_10000000095, false, Parquet, [path=hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/supply-network-list/10000000095/shardId=0_1_10000000095], Overwrite, [supplierGroup, supplier, plant, deliveryDateTime, subrange]
WriteFiles
HashAggregate(keys=[plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680], functions=[])
WholeStageCodegen (3)
AQEShuffleRead coalesced
Exchange hashpartitioning(plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680, 25), ENSURE_REQUIREMENTS, [plan_id=3377]
HashAggregate(keys=[plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680], functions=[])
Project [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
WholeStageCodegen (2)
Exchange hashpartitioning(product#11682, plant#11679, 25), REPARTITION_BY_NUM, [plan_id=3280]
ColumnarToRow
WholeStageCodegen (1)
FileScan parquet [supplierGroup#11677,supplier#11678,plant#11679,deliveryDateTime#11680,subrange#11681,product#11682] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<supplierGroup:string,supplier:string,plant:string,deliveryDateTime:timestamp,subrange:stri...
== Physical Plan ==
AdaptiveSparkPlan (20)
+- == Final Plan ==
Execute InsertIntoHadoopFsRelationCommand (12)
+- WriteFiles (11)
+- * HashAggregate (10)
+- AQEShuffleRead (9)
+- ShuffleQueryStage (8), Statistics(sizeInBytes=126.3 KiB, rowCount=1.03E+3)
+- Exchange (7)
+- * HashAggregate (6)
+- * Project (5)
+- ShuffleQueryStage (4), Statistics(sizeInBytes=193.0 KiB, rowCount=1.20E+3)
+- Exchange (3)
+- * ColumnarToRow (2)
+- Scan parquet (1)
+- == Initial Plan ==
Execute InsertIntoHadoopFsRelationCommand (19)
+- WriteFiles (18)
+- HashAggregate (17)
+- Exchange (16)
+- HashAggregate (15)
+- Project (14)
+- Exchange (13)
+- Scan parquet (1)
(1) Scan parquet
Output [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
Batched: true
Location: InMemoryFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/order-proposal-boundary-snapshot/10000000095/shardId=0_1_10000000095]
ReadSchema: struct<supplierGroup:string,supplier:string,plant:string,deliveryDateTime:timestamp,subrange:string,product:string>
(2) ColumnarToRow [codegen id : 1]
Input [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
(3) Exchange
Input [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
Arguments: hashpartitioning(product#11682, plant#11679, 25), REPARTITION_BY_NUM, [plan_id=3280]
(4) ShuffleQueryStage
Output [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
Arguments: 0
(5) Project [codegen id : 2]
Output [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
Input [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
(6) HashAggregate [codegen id : 2]
Input [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
Keys [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Functions: []
Aggregate Attributes: []
Results [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
(7) Exchange
Input [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Arguments: hashpartitioning(plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680, 25), ENSURE_REQUIREMENTS, [plan_id=3377]
(8) ShuffleQueryStage
Output [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Arguments: 1
(9) AQEShuffleRead
Input [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Arguments: coalesced
(10) HashAggregate [codegen id : 3]
Input [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Keys [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Functions: []
Aggregate Attributes: []
Results [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
(11) WriteFiles
Input [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
(12) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/supply-network-list/10000000095/shardId=0_1_10000000095, false, Parquet, [path=hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/supply-network-list/10000000095/shardId=0_1_10000000095], Overwrite, [supplierGroup, supplier, plant, deliveryDateTime, subrange]
(13) Exchange
Input [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
Arguments: hashpartitioning(product#11682, plant#11679, 25), REPARTITION_BY_NUM, [plan_id=3255]
(14) Project
Output [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
Input [6]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681, product#11682]
(15) HashAggregate
Input [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
Keys [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Functions: []
Aggregate Attributes: []
Results [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
(16) Exchange
Input [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Arguments: hashpartitioning(plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680, 25), ENSURE_REQUIREMENTS, [plan_id=3262]
(17) HashAggregate
Input [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Keys [5]: [plant#11679, supplierGroup#11677, subrange#11681, supplier#11678, deliveryDateTime#11680]
Functions: []
Aggregate Attributes: []
Results [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
(18) WriteFiles
Input [5]: [supplierGroup#11677, supplier#11678, plant#11679, deliveryDateTime#11680, subrange#11681]
(19) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/supply-network-list/10000000095/shardId=0_1_10000000095, false, Parquet, [path=hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/supply-network-list/10000000095/shardId=0_1_10000000095], Overwrite, [supplierGroup, supplier, plant, deliveryDateTime, subrange]
(20) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true