crp-order-qty-opt-service - Details for Query 2837

Submitted Time: 2026/08/10 09:30:01
Duration: 2 s
Succeeded Jobs: 4348 4349 4350

Plan Visualization

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [id="node0" labelType="html" label="CollectLimit shuffle records written: 8 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 434.0 B merged fetch fallback count: 0 local blocks read: 7 remote merged chunks fetched: 0 remote blocks read: 0 local merged bytes read: 0.0 B remote reqs duration: 0 ms local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 1 ms (0 ms, 0 ms, 0 ms (stage 51117.0: task 136940)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time: 0 ms remote bytes read: 0.0 B remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 496.0 B (62.0 B, 62.0 B, 62.0 B (stage 51117.0: task 136944))" tooltip="CollectLimit 1"]; subgraph cluster1 { isCluster="true"; id="cluster1"; label="WholeStageCodegen (8)\n \nduration: total (min, med, max (stageId: taskId))\n35 ms (0 ms, 0 ms, 7 ms (stage 51117.0: task 136938))"; tooltip="WholeStageCodegen (8)"; 2 [id="node2" labelType="html" label=" Project " tooltip="Project"]; 3 [id="node3" labelType="html" label="BroadcastHashJoin number of output rows: 8" tooltip="BroadcastHashJoin [worklistShardItemId#657622L], [worklistShardItemId#658167L], LeftOuter, BuildRight, (shardId#643168 = 0_1_10000002106), false"]; 4 [id="node4" labelType="html" label=" LocalLimit " tooltip="LocalLimit 1"]; 5 [id="node5" labelType="html" label=" Project " tooltip="Project [shardId#643168, insight#657620.worklistShardItemId AS worklistShardItemId#657622L]"]; 6 [id="node6" labelType="html" label="Filter number of output rows: 87" tooltip="Filter insight#657620.internalInsightType IN (OP_CALC_ITEM_ACCEPT_THR_NOT_MET,OP_CALC_ITEM_NEGL_THR_NOT_MET,OP_CALC_ITEM_OPT_MISSING_MD,OP_CALC_LOCKED_QUANTITY_PREVENTS_DISCARDING_INSIGHT,OP_CALC_MIN_NOT_MET)"]; 7 [id="node7" labelType="html" label="Generate number of output rows: 87" tooltip="Generate explode(optimizationInsights#643174), [shardId#643168], false, [insight#657620]"]; 8 [id="node8" labelType="html" label=" Project " tooltip="Project [shardId#643168, optimizationInsights#643174]"]; 9 [id="node9" labelType="html" label="SortMergeJoin number of output rows: 8 spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 51117.0: task 136938))" tooltip="SortMergeJoin [orderProposalId#643169L], [orderProposalId#645037L], Inner"]; } subgraph cluster10 { isCluster="true"; id="cluster10"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n1.4 s (0 ms, 0 ms, 286 ms (stage 51117.0: task 136947))"; tooltip="WholeStageCodegen (2)"; 11 [id="node11" labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 51117.0: task 136938)) peak memory total (min, med, max (stageId: taskId)) 17.3 MiB (64.0 KiB, 64.0 KiB, 2.1 MiB (stage 51117.0: task 136944)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 51117.0: task 136938))" tooltip="Sort [orderProposalId#643169L ASC NULLS FIRST], false, 0"]; } 12 [id="node12" labelType="html" label="Exchange shuffle records written: 11 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 11 local bytes read total (min, med, max (stageId: taskId)) 3.8 KiB (269.0 B, 403.0 B, 921.0 B (stage 51117.0: task 136946)) merged fetch fallback count: 0 local blocks read: 10 remote merged chunks fetched: 0 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 21.5 KiB (240.0 B, 1600.0 B, 9.3 KiB (stage 51115.0: task 136913)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 4 ms (0 ms, 0 ms, 0 ms (stage 51115.0: task 136902)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 51117.0: task 136944)) remote bytes read: 255.0 B number of partitions: 20 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 4.1 KiB (255.0 B, 344.0 B, 881.0 B (stage 51115.0: task 136913))" tooltip="Exchange hashpartitioning(orderProposalId#643169L, 20), ENSURE_REQUIREMENTS, [plan_id=267716]"]; subgraph cluster13 { isCluster="true"; id="cluster13"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n157 ms (1 ms, 7 ms, 12 ms (stage 51115.0: task 136905))"; tooltip="WholeStageCodegen (1)"; 14 [id="node14" labelType="html" label=" Project " tooltip="Project [shardId#643168, orderProposalId#643169L, optimizationInsights#643174]"]; 15 [id="node15" labelType="html" label="Filter number of output rows: 11" tooltip="Filter ((isnotnull(shardId#643168) AND (shardId#643168 = 0_1_10000002106)) AND ((isnotnull(orderProposalId#643169L) AND (size(optimizationInsights#643174, true) > 0)) AND isnotnull(optimizationInsights#643174)))"]; 16 [id="node16" labelType="html" label="Scan ExistingRDD number of output rows: 59" tooltip="Scan ExistingRDD[shardId#643168,orderProposalId#643169L,items#643170,optimizationGroups#643171,isOptimal#643172,lastModified#643173,optimizationInsights#643174]"]; } subgraph cluster17 { isCluster="true"; id="cluster17"; label="WholeStageCodegen (6)\n \nduration: total (min, med, max (stageId: taskId))\n140 ms (1 ms, 139 ms, 139 ms (stage 51117.0: task 136940))"; tooltip="WholeStageCodegen (6)"; 18 [id="node18" labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 51117.0: task 136944)) peak memory total (min, med, max (stageId: taskId)) 16.5 MiB (2.1 MiB, 2.1 MiB, 2.1 MiB (stage 51117.0: task 136944)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 51117.0: task 136944))" tooltip="Sort [orderProposalId#645037L ASC NULLS FIRST], false, 0"]; 19 [id="node19" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 3 ms (0 ms, 0 ms, 1 ms (stage 51117.0: task 136942)) peak memory total (min, med, max (stageId: taskId)) 2.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 51117.0: task 136944)) number of output rows: 29 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[orderProposalId#645037L], functions=[])"]; } 20 [id="node20" labelType="html" label="Exchange shuffle records written: 59 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 29 local bytes read total (min, med, max (stageId: taskId)) 2.1 KiB (75.0 B, 75.0 B, 375.0 B (stage 51117.0: task 136957)) merged fetch fallback count: 0 local blocks read: 29 remote merged chunks fetched: 0 remote blocks read: 25 data size total (min, med, max (stageId: taskId)) 944.0 B (16.0 B, 48.0 B, 96.0 B (stage 51116.0: task 136928)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 8 ms (0 ms, 0 ms, 0 ms (stage 51116.0: task 136934)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 51117.0: task 136944)) remote bytes read total (min, med, max (stageId: taskId)) 1885.0 B (75.0 B, 225.0 B, 311.0 B (stage 51117.0: task 136946)) number of partitions: 20 remote reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 51117.0: task 136938)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 4.2 KiB (75.0 B, 224.0 B, 450.0 B (stage 51116.0: task 136928))" tooltip="Exchange hashpartitioning(orderProposalId#645037L, 20), ENSURE_REQUIREMENTS, [plan_id=267736]"]; subgraph cluster21 { isCluster="true"; id="cluster21"; label="WholeStageCodegen (5)\n \nduration: total (min, med, max (stageId: taskId))\n272 ms (7 ms, 10 ms, 22 ms (stage 51116.0: task 136927))"; tooltip="WholeStageCodegen (5)"; 22 [id="node22" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 148 ms (0 ms, 7 ms, 10 ms (stage 51116.0: task 136927)) peak memory total (min, med, max (stageId: taskId)) 5.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 51116.0: task 136910)) number of output rows: 59 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[orderProposalId#645037L], functions=[])"]; 23 [id="node23" labelType="html" label=" Project " tooltip="Project [orderProposalId#645037L]"]; 24 [id="node24" labelType="html" label="BroadcastHashJoin number of output rows: 980" tooltip="BroadcastHashJoin [worklistShardItemId#643816L], [worklistShardItemId#645060L], Inner, BuildRight, false"]; 25 [id="node25" labelType="html" label=" Project " tooltip="Project [item#643815L AS worklistShardItemId#643816L]"]; 26 [id="node26" labelType="html" label="Generate number of output rows: 980" tooltip="Generate explode(_extract_worklistShardItemId#658683), false, [item#643815L]"]; 27 [id="node27" labelType="html" label=" Project " tooltip="Project [items#657614.worklistShardItemId AS _extract_worklistShardItemId#658683]"]; 28 [id="node28" labelType="html" label="Filter number of output rows: 59" tooltip="Filter ((isnotnull(shardId#657612) AND (shardId#657612 = 0_1_10000002106)) AND ((size(items#657614.worklistShardItemId, true) > 0) AND isnotnull(items#657614.worklistShardItemId)))"]; 29 [id="node29" labelType="html" label="Scan ExistingRDD number of output rows: 59" tooltip="Scan ExistingRDD[shardId#657612,orderProposalId#657613L,items#657614,optimizationGroups#657615,isOptimal#657616,lastModified#657617,optimizationInsights#657618]"]; } 30 [id="node30" labelType="html" label="BroadcastExchange time to broadcast: 0 ms time to build: 1 ms time to collect: 616 ms number of output rows: 980 data size: 1056.0 KiB" tooltip="BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true]),false), [plan_id=267730]"]; subgraph cluster31 { isCluster="true"; id="cluster31"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n38 ms (1 ms, 2 ms, 3 ms (stage 51027.0: task 136860))"; tooltip="WholeStageCodegen (4)"; 32 [id="node32" labelType="html" label=" Project " tooltip="Project [worklistShardItemId#645060L, orderProposalId#645037L]"]; } 33 [id="node33" labelType="html" label="Exchange shuffle records written: 980 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 980 local bytes read total (min, med, max (stageId: taskId)) 22.1 KiB (901.0 B, 1142.0 B, 1438.0 B (stage 51027.0: task 136860)) merged fetch fallback count: 0 local blocks read: 216 remote merged chunks fetched: 0 remote blocks read: 220 data size total (min, med, max (stageId: taskId)) 23.0 KiB (648.0 B, 960.0 B, 1152.0 B (stage 51026.0: task 136811)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 24 ms (0 ms, 0 ms, 1 ms (stage 51026.0: task 136810)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 51027.0: task 136862)) remote bytes read total (min, med, max (stageId: taskId)) 22.5 KiB (846.0 B, 1200.0 B, 1525.0 B (stage 51027.0: task 136864)) number of partitions: 20 remote reqs duration total (min, med, max (stageId: taskId)) 17 ms (0 ms, 1 ms, 1 ms (stage 51027.0: task 136862)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 44.6 KiB (1511.0 B, 1790.0 B, 2.1 KiB (stage 51026.0: task 136810))" tooltip="Exchange hashpartitioning(worklistShardItemId#645060L, 20), REPARTITION_BY_NUM, [plan_id=267726]"]; subgraph cluster34 { isCluster="true"; id="cluster34"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n3.0 s (94 ms, 113 ms, 164 ms (stage 51026.0: task 136816))"; tooltip="WholeStageCodegen (3)"; 35 [id="node35" labelType="html" label=" Project " tooltip="Project [orderProposalId#645037L, worklistShardItemId#645060L]"]; 36 [id="node36" labelType="html" label="Filter number of output rows: 980" tooltip="Filter ((isnotnull(worklistShardItemId#645060L) AND NOT (isFallbackOrderOnly#645054 <=> true)) AND isnotnull(orderProposalId#645037L))"]; 37 [id="node37" labelType="html" label="ColumnarToRow number of output rows: 980 number of input batches: 25" tooltip="ColumnarToRow"]; } 38 [id="node38" labelType="html" label="Scan parquet number of files read: 25 scan time total (min, med, max (stageId: taskId)) 3.0 s (92 ms, 111 ms, 161 ms (stage 51026.0: task 136816)) dynamic partition pruning time: 0 ms metadata time: 0 ms size of files read: 674.1 KiB number of output rows: 980 number of partitions read: 1" tooltip="FileScan parquet [orderProposalId#645037L,isFallbackOrderOnly#645054,worklistShardItemId#645060L,shardId#645081,sequenceId#645059] Batched: true, DataFilters: [isnotnull(worklistShardItemId#645060L), NOT (isFallbackOrderOnly#645054 <=> true), isnotnull(ord..., Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20...., PartitionFilters: [isnotnull(shardId#645081), isnotnull(sequenceId#645059), (shardId#645081 = 0_1_10000002106), (se..., PushedFilters: [IsNotNull(worklistShardItemId), Not(EqualNullSafe(isFallbackOrderOnly,true)), IsNotNull(orderPro..., ReadSchema: struct<orderProposalId:bigint,isFallbackOrderOnly:boolean,worklistShardItemId:bigint>"]; 39 [id="node39" labelType="html" label="BroadcastExchange time to broadcast: 0 ms time to build: 0 ms time to collect: 652 ms number of output rows: 980 data size: 1056.0 KiB" tooltip="BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true]),false), [plan_id=267754]"]; 40 [id="node40" labelType="html" label="Exchange shuffle records written: 980 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 980 local bytes read total (min, med, max (stageId: taskId)) 17.9 KiB (512.0 B, 934.0 B, 1282.0 B (stage 51029.0: task 136883)) merged fetch fallback count: 0 local blocks read: 215 remote merged chunks fetched: 0 remote blocks read: 221 data size total (min, med, max (stageId: taskId)) 15.3 KiB (432.0 B, 640.0 B, 768.0 B (stage 51028.0: task 136821)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 23 ms (0 ms, 0 ms, 1 ms (stage 51028.0: task 136855)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 3 ms (0 ms, 0 ms, 1 ms (stage 51029.0: task 136880)) remote bytes read total (min, med, max (stageId: taskId)) 18.8 KiB (636.0 B, 941.0 B, 1349.0 B (stage 51029.0: task 136880)) number of partitions: 20 remote reqs duration total (min, med, max (stageId: taskId)) 28 ms (0 ms, 2 ms, 3 ms (stage 51029.0: task 136882)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 36.7 KiB (1259.0 B, 1490.0 B, 1741.0 B (stage 51028.0: task 136820))" tooltip="Exchange hashpartitioning(worklistShardItemId#658167L, 20), REPARTITION_BY_NUM, [plan_id=267753]"]; subgraph cluster41 { isCluster="true"; id="cluster41"; label="WholeStageCodegen (7)\n \nduration: total (min, med, max (stageId: taskId))\n2.2 s (76 ms, 88 ms, 140 ms (stage 51028.0: task 136820))"; tooltip="WholeStageCodegen (7)"; 42 [id="node42" labelType="html" label=" Project " tooltip="Project [worklistShardItemId#658167L]"]; 43 [id="node43" labelType="html" label="Filter number of output rows: 980" tooltip="Filter isnotnull(worklistShardItemId#658167L)"]; 44 [id="node44" labelType="html" label="ColumnarToRow number of output rows: 980 number of input batches: 25" tooltip="ColumnarToRow"]; } 45 [id="node45" labelType="html" label="Scan parquet number of files read: 25 scan time total (min, med, max (stageId: taskId)) 2.2 s (74 ms, 85 ms, 136 ms (stage 51028.0: task 136820)) dynamic partition pruning time: 0 ms metadata time: 0 ms size of files read: 674.1 KiB number of output rows: 980 number of partitions read: 1" tooltip="FileScan parquet [worklistShardItemId#658167L,shardId#658188,sequenceId#658166] Batched: true, DataFilters: [isnotnull(worklistShardItemId#658167L)], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20...., PartitionFilters: [isnotnull(shardId#658188), isnotnull(sequenceId#658166), (shardId#658188 = 0_1_10000002106), (se..., PushedFilters: [IsNotNull(worklistShardItemId)], ReadSchema: struct<worklistShardItemId:bigint>"]; 2->0; 3->2; 4->3; 5->4; 6->5; 7->6; 8->7; 9->8; 11->9; 12->11; 14->12; 15->14; 16->15; 18->9; 19->18; 20->19; 22->20; 23->22; 24->23; 25->24; 26->25; 27->26; 28->27; 29->28; 30->24; 32->30; 33->32; 35->33; 36->35; 37->36; 38->37; 39->3; 40->39; 42->40; 43->42; 44->43; 45->44; }

Details

== Physical Plan ==
CollectLimit (38)
+- * Project (37)
   +- * BroadcastHashJoin LeftOuter BuildRight (36)
      :- * LocalLimit (29)
      :  +- * Project (28)
      :     +- * Filter (27)
      :        +- * Generate (26)
      :           +- * Project (25)
      :              +- * SortMergeJoin Inner (24)
      :                 :- * Sort (5)
      :                 :  +- Exchange (4)
      :                 :     +- * Project (3)
      :                 :        +- * Filter (2)
      :                 :           +- * Scan ExistingRDD (1)
      :                 +- * Sort (23)
      :                    +- * HashAggregate (22)
      :                       +- Exchange (21)
      :                          +- * HashAggregate (20)
      :                             +- * Project (19)
      :                                +- * BroadcastHashJoin Inner BuildRight (18)
      :                                   :- * Project (10)
      :                                   :  +- * Generate (9)
      :                                   :     +- * Project (8)
      :                                   :        +- * Filter (7)
      :                                   :           +- * Scan ExistingRDD (6)
      :                                   +- BroadcastExchange (17)
      :                                      +- * Project (16)
      :                                         +- Exchange (15)
      :                                            +- * Project (14)
      :                                               +- * Filter (13)
      :                                                  +- * ColumnarToRow (12)
      :                                                     +- Scan parquet  (11)
      +- BroadcastExchange (35)
         +- Exchange (34)
            +- * Project (33)
               +- * Filter (32)
                  +- * ColumnarToRow (31)
                     +- Scan parquet  (30)


(1) Scan ExistingRDD [codegen id : 1]
Output [7]: [shardId#643168, orderProposalId#643169L, items#643170, optimizationGroups#643171, isOptimal#643172, lastModified#643173, optimizationInsights#643174]
Arguments: [shardId#643168, orderProposalId#643169L, items#643170, optimizationGroups#643171, isOptimal#643172, lastModified#643173, optimizationInsights#643174], MapPartitionsRDD[30592] at start at FileStorageAdapterImpl.java:592, ExistingRDD, UnknownPartitioning(0)

(2) Filter [codegen id : 1]
Input [7]: [shardId#643168, orderProposalId#643169L, items#643170, optimizationGroups#643171, isOptimal#643172, lastModified#643173, optimizationInsights#643174]
Condition : ((isnotnull(shardId#643168) AND (shardId#643168 = 0_1_10000002106)) AND ((isnotnull(orderProposalId#643169L) AND (size(optimizationInsights#643174, true) > 0)) AND isnotnull(optimizationInsights#643174)))

(3) Project [codegen id : 1]
Output [3]: [shardId#643168, orderProposalId#643169L, optimizationInsights#643174]
Input [7]: [shardId#643168, orderProposalId#643169L, items#643170, optimizationGroups#643171, isOptimal#643172, lastModified#643173, optimizationInsights#643174]

(4) Exchange
Input [3]: [shardId#643168, orderProposalId#643169L, optimizationInsights#643174]
Arguments: hashpartitioning(orderProposalId#643169L, 20), ENSURE_REQUIREMENTS, [plan_id=267716]

(5) Sort [codegen id : 2]
Input [3]: [shardId#643168, orderProposalId#643169L, optimizationInsights#643174]
Arguments: [orderProposalId#643169L ASC NULLS FIRST], false, 0

(6) Scan ExistingRDD [codegen id : 5]
Output [7]: [shardId#657612, orderProposalId#657613L, items#657614, optimizationGroups#657615, isOptimal#657616, lastModified#657617, optimizationInsights#657618]
Arguments: [shardId#657612, orderProposalId#657613L, items#657614, optimizationGroups#657615, isOptimal#657616, lastModified#657617, optimizationInsights#657618], MapPartitionsRDD[30592] at start at FileStorageAdapterImpl.java:592, ExistingRDD, UnknownPartitioning(0)

(7) Filter [codegen id : 5]
Input [7]: [shardId#657612, orderProposalId#657613L, items#657614, optimizationGroups#657615, isOptimal#657616, lastModified#657617, optimizationInsights#657618]
Condition : ((isnotnull(shardId#657612) AND (shardId#657612 = 0_1_10000002106)) AND ((size(items#657614.worklistShardItemId, true) > 0) AND isnotnull(items#657614.worklistShardItemId)))

(8) Project [codegen id : 5]
Output [1]: [items#657614.worklistShardItemId AS _extract_worklistShardItemId#658683]
Input [7]: [shardId#657612, orderProposalId#657613L, items#657614, optimizationGroups#657615, isOptimal#657616, lastModified#657617, optimizationInsights#657618]

(9) Generate [codegen id : 5]
Input [1]: [_extract_worklistShardItemId#658683]
Arguments: explode(_extract_worklistShardItemId#658683), false, [item#643815L]

(10) Project [codegen id : 5]
Output [1]: [item#643815L AS worklistShardItemId#643816L]
Input [1]: [item#643815L]

(11) Scan parquet 
Output [5]: [orderProposalId#645037L, isFallbackOrderOnly#645054, worklistShardItemId#645060L, shardId#645081, sequenceId#645059]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/internal/prod-plant-snapshot/10000002106]
PartitionFilters: [isnotnull(shardId#645081), isnotnull(sequenceId#645059), (shardId#645081 = 0_1_10000002106), (sequenceId#645059 = 2)]
PushedFilters: [IsNotNull(worklistShardItemId), Not(EqualNullSafe(isFallbackOrderOnly,true)), IsNotNull(orderProposalId)]
ReadSchema: struct<orderProposalId:bigint,isFallbackOrderOnly:boolean,worklistShardItemId:bigint>

(12) ColumnarToRow [codegen id : 3]
Input [5]: [orderProposalId#645037L, isFallbackOrderOnly#645054, worklistShardItemId#645060L, shardId#645081, sequenceId#645059]

(13) Filter [codegen id : 3]
Input [5]: [orderProposalId#645037L, isFallbackOrderOnly#645054, worklistShardItemId#645060L, shardId#645081, sequenceId#645059]
Condition : ((isnotnull(worklistShardItemId#645060L) AND NOT (isFallbackOrderOnly#645054 <=> true)) AND isnotnull(orderProposalId#645037L))

(14) Project [codegen id : 3]
Output [2]: [orderProposalId#645037L, worklistShardItemId#645060L]
Input [5]: [orderProposalId#645037L, isFallbackOrderOnly#645054, worklistShardItemId#645060L, shardId#645081, sequenceId#645059]

(15) Exchange
Input [2]: [orderProposalId#645037L, worklistShardItemId#645060L]
Arguments: hashpartitioning(worklistShardItemId#645060L, 20), REPARTITION_BY_NUM, [plan_id=267726]

(16) Project [codegen id : 4]
Output [2]: [worklistShardItemId#645060L, orderProposalId#645037L]
Input [2]: [orderProposalId#645037L, worklistShardItemId#645060L]

(17) BroadcastExchange
Input [2]: [worklistShardItemId#645060L, orderProposalId#645037L]
Arguments: HashedRelationBroadcastMode(List(input[0, bigint, true]),false), [plan_id=267730]

(18) BroadcastHashJoin [codegen id : 5]
Left keys [1]: [worklistShardItemId#643816L]
Right keys [1]: [worklistShardItemId#645060L]
Join type: Inner
Join condition: None

(19) Project [codegen id : 5]
Output [1]: [orderProposalId#645037L]
Input [3]: [worklistShardItemId#643816L, worklistShardItemId#645060L, orderProposalId#645037L]

(20) HashAggregate [codegen id : 5]
Input [1]: [orderProposalId#645037L]
Keys [1]: [orderProposalId#645037L]
Functions: []
Aggregate Attributes: []
Results [1]: [orderProposalId#645037L]

(21) Exchange
Input [1]: [orderProposalId#645037L]
Arguments: hashpartitioning(orderProposalId#645037L, 20), ENSURE_REQUIREMENTS, [plan_id=267736]

(22) HashAggregate [codegen id : 6]
Input [1]: [orderProposalId#645037L]
Keys [1]: [orderProposalId#645037L]
Functions: []
Aggregate Attributes: []
Results [1]: [orderProposalId#645037L]

(23) Sort [codegen id : 6]
Input [1]: [orderProposalId#645037L]
Arguments: [orderProposalId#645037L ASC NULLS FIRST], false, 0

(24) SortMergeJoin [codegen id : 8]
Left keys [1]: [orderProposalId#643169L]
Right keys [1]: [orderProposalId#645037L]
Join type: Inner
Join condition: None

(25) Project [codegen id : 8]
Output [2]: [shardId#643168, optimizationInsights#643174]
Input [4]: [shardId#643168, orderProposalId#643169L, optimizationInsights#643174, orderProposalId#645037L]

(26) Generate [codegen id : 8]
Input [2]: [shardId#643168, optimizationInsights#643174]
Arguments: explode(optimizationInsights#643174), [shardId#643168], false, [insight#657620]

(27) Filter [codegen id : 8]
Input [2]: [shardId#643168, insight#657620]
Condition : insight#657620.internalInsightType IN (OP_CALC_ITEM_ACCEPT_THR_NOT_MET,OP_CALC_ITEM_NEGL_THR_NOT_MET,OP_CALC_ITEM_OPT_MISSING_MD,OP_CALC_LOCKED_QUANTITY_PREVENTS_DISCARDING_INSIGHT,OP_CALC_MIN_NOT_MET)

(28) Project [codegen id : 8]
Output [2]: [shardId#643168, insight#657620.worklistShardItemId AS worklistShardItemId#657622L]
Input [2]: [shardId#643168, insight#657620]

(29) LocalLimit [codegen id : 8]
Input [2]: [shardId#643168, worklistShardItemId#657622L]
Arguments: 1

(30) Scan parquet 
Output [3]: [worklistShardItemId#658167L, shardId#658188, sequenceId#658166]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/internal/prod-plant-snapshot/10000002106]
PartitionFilters: [isnotnull(shardId#658188), isnotnull(sequenceId#658166), (shardId#658188 = 0_1_10000002106), (sequenceId#658166 = 2)]
PushedFilters: [IsNotNull(worklistShardItemId)]
ReadSchema: struct<worklistShardItemId:bigint>

(31) ColumnarToRow [codegen id : 7]
Input [3]: [worklistShardItemId#658167L, shardId#658188, sequenceId#658166]

(32) Filter [codegen id : 7]
Input [3]: [worklistShardItemId#658167L, shardId#658188, sequenceId#658166]
Condition : isnotnull(worklistShardItemId#658167L)

(33) Project [codegen id : 7]
Output [1]: [worklistShardItemId#658167L]
Input [3]: [worklistShardItemId#658167L, shardId#658188, sequenceId#658166]

(34) Exchange
Input [1]: [worklistShardItemId#658167L]
Arguments: hashpartitioning(worklistShardItemId#658167L, 20), REPARTITION_BY_NUM, [plan_id=267753]

(35) BroadcastExchange
Input [1]: [worklistShardItemId#658167L]
Arguments: HashedRelationBroadcastMode(List(input[0, bigint, true]),false), [plan_id=267754]

(36) BroadcastHashJoin [codegen id : 8]
Left keys [1]: [worklistShardItemId#657622L]
Right keys [1]: [worklistShardItemId#658167L]
Join type: LeftOuter
Join condition: (shardId#643168 = 0_1_10000002106)

(37) Project [codegen id : 8]
Output: []
Input [3]: [shardId#643168, worklistShardItemId#657622L, worklistShardItemId#658167L]

(38) CollectLimit
Input: []
Arguments: 1

SQL / DataFrame Properties

Name	Value
spark.activity.hdlfsEnabled	true
spark.activity.isolateSession	true
spark.databricks.delta.lastCommitVersionInSession	3
spark.sql.adaptive.enabled	false
spark.sql.cbo.enabled	false
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true
spark.sql.requireAllClusterKeysForDistribution	false
spark.sql.shuffle.partitions	20
temporal.activity.id	10a6a3d0-5fcf-39e3-a20a-09c5b3945c32
temporal.activity.type	RunStreamWithId