== Physical Plan ==
AdaptiveSparkPlan (39)
+- == Final Plan ==
Execute InsertIntoHadoopFsRelationCommand (26)
+- WriteFiles (25)
+- * Sort (24)
+- * Project (23)
+- * SortMergeJoin FullOuter (22)
:- * Sort (16)
: +- AQEShuffleRead (15)
: +- ShuffleQueryStage (14), Statistics(sizeInBytes=43.2 KiB, rowCount=70)
: +- Exchange (13)
: +- * Project (12)
: +- * SortMergeJoin FullOuter (11)
: :- * Sort (5)
: : +- AQEShuffleRead (4)
: : +- ShuffleQueryStage (3), Statistics(sizeInBytes=41.6 KiB, rowCount=70)
: : +- Exchange (2)
: : +- LocalTableScan (1)
: +- * Sort (10)
: +- AQEShuffleRead (9)
: +- ShuffleQueryStage (8), Statistics(sizeInBytes=3.3 KiB, rowCount=70)
: +- Exchange (7)
: +- LocalTableScan (6)
+- * Sort (21)
+- AQEShuffleRead (20)
+- ShuffleQueryStage (19), Statistics(sizeInBytes=4.4 KiB, rowCount=70)
+- Exchange (18)
+- LocalTableScan (17)
+- == Initial Plan ==
Execute InsertIntoHadoopFsRelationCommand (38)
+- WriteFiles (37)
+- Sort (36)
+- Project (35)
+- SortMergeJoin FullOuter (34)
:- Sort (32)
: +- Exchange (31)
: +- Project (30)
: +- SortMergeJoin FullOuter (29)
: :- Sort (27)
: : +- Exchange (2)
: : +- LocalTableScan (1)
: +- Sort (28)
: +- Exchange (7)
: +- LocalTableScan (6)
+- Sort (33)
+- Exchange (18)
+- LocalTableScan (17)
(1) LocalTableScan
Output [73]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L]
Arguments: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, ... 49 more fields]
(2) Exchange
Input [73]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L]
Arguments: hashpartitioning(jobId#57475, 25), ENSURE_REQUIREMENTS, [plan_id=17699]
(3) ShuffleQueryStage
Output [73]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L]
Arguments: 0
(4) AQEShuffleRead
Input [73]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L]
Arguments: coalesced
(5) Sort [codegen id : 1]
Input [73]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L]
Arguments: [jobId#57475 ASC NULLS FIRST], false, 0
(6) LocalTableScan
Output [4]: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
(7) Exchange
Input [4]: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: hashpartitioning(jobId#57621, 25), ENSURE_REQUIREMENTS, [plan_id=17700]
(8) ShuffleQueryStage
Output [4]: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: 1
(9) AQEShuffleRead
Input [4]: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: coalesced
(10) Sort [codegen id : 2]
Input [4]: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: [jobId#57621 ASC NULLS FIRST], false, 0
(11) SortMergeJoin [codegen id : 3]
Left keys [1]: [jobId#57475]
Right keys [1]: [jobId#57621]
Join type: FullOuter
Join condition: None
(12) Project [codegen id : 3]
Output [76]: [coalesce(jobId#57475, jobId#57621) AS jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Input [77]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
(13) Exchange
Input [76]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: hashpartitioning(jobId#57641, 25), ENSURE_REQUIREMENTS, [plan_id=17860]
(14) ShuffleQueryStage
Output [76]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: 3
(15) AQEShuffleRead
Input [76]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: coalesced
(16) Sort [codegen id : 4]
Input [76]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: [jobId#57641 ASC NULLS FIRST], false, 0
(17) LocalTableScan
Output [6]: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
Arguments: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
(18) Exchange
Input [6]: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
Arguments: hashpartitioning(jobId#57629, 25), ENSURE_REQUIREMENTS, [plan_id=17708]
(19) ShuffleQueryStage
Output [6]: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
Arguments: 2
(20) AQEShuffleRead
Input [6]: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
Arguments: coalesced
(21) Sort [codegen id : 5]
Input [6]: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
Arguments: [jobId#57629 ASC NULLS FIRST], false, 0
(22) SortMergeJoin [codegen id : 6]
Left keys [1]: [jobId#57641]
Right keys [1]: [jobId#57629]
Join type: FullOuter
Join condition: None
(23) Project [codegen id : 6]
Output [92]: [coalesce(jobId#57641, jobId#57629) AS jobId#57718, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634, 10000000096 AS replenishmentRunId#57800L, 6944346648028224943 AS tenantId#57883, 8378 AS cornerstoneTenantId#57967L, Daily_grocery AS marketUnit#58052, STANDARD AS scenario#58138, 9aa5a928-b219-375b-ac11-54040e15cd53 AS activityId#58225, BufferDataSnapShot AS activityType#58313, PrepareDataSnapshotWorkflow AS workflowType#58402, b23ef421-1eee-3d30-a73f-c7878eb04008 AS workflowId#58492, 1 AS activityAttempt#58583, AS gitCommitId#58768]
Input [82]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
(24) Sort [codegen id : 6]
Input [92]: [jobId#57718, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634, replenishmentRunId#57800L, tenantId#57883, cornerstoneTenantId#57967L, marketUnit#58052, scenario#58138, activityId#58225, activityType#58313, workflowType#58402, workflowId#58492, activityAttempt#58583, gitCommitId#58768]
Arguments: [activityType#58313 ASC NULLS FIRST], false, 0
(25) WriteFiles
Input [92]: [jobId#57718, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634, replenishmentRunId#57800L, tenantId#57883, cornerstoneTenantId#57967L, marketUnit#58052, scenario#58138, activityId#58225, activityType#58313, workflowType#58402, workflowId#58492, activityAttempt#58583, gitCommitId#58768]
(26) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/internal/performance-metrics/10000000096, false, [activityType#58313], Parquet, [__partition_columns=["activityType"], path=hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/internal/performance-metrics/10000000096], Append, [jobId, stageCount, totalTaskCount, minTaskCountPerStage, maxTaskCountPerStage, medianTaskCountPerStage, bytesSpilledToDiskSerializedTotal, bytesSpilledToDiskSerializedMin, bytesSpilledToDiskSerializedMax, bytesSpilledToDiskSerializedMedian, taskCpuTimeTotal, taskCpuTimeMin, taskCpuTimeMax, taskCpuTimeMedian, taskDeserializeCpuTimeTotal, taskDeserializeCpuTimeMin, taskDeserializeCpuTimeMax, taskDeserializeCpuTimeMedian, taskDeserializeTimeTotal, taskDeserializeTimeMin, taskDeserializeTimeMax, taskDeserializeTimeMedian, taskRunTimeTotal, taskRunTimeMin, ... 68 more fields]
(27) Sort
Input [73]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L]
Arguments: [jobId#57475 ASC NULLS FIRST], false, 0
(28) Sort
Input [4]: [jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: [jobId#57621 ASC NULLS FIRST], false, 0
(29) SortMergeJoin
Left keys [1]: [jobId#57475]
Right keys [1]: [jobId#57621]
Join type: FullOuter
Join condition: None
(30) Project
Output [76]: [coalesce(jobId#57475, jobId#57621) AS jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Input [77]: [jobId#57475, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, jobId#57621, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
(31) Exchange
Input [76]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: hashpartitioning(jobId#57641, 25), ENSURE_REQUIREMENTS, [plan_id=17707]
(32) Sort
Input [76]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L]
Arguments: [jobId#57641 ASC NULLS FIRST], false, 0
(33) Sort
Input [6]: [jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
Arguments: [jobId#57629 ASC NULLS FIRST], false, 0
(34) SortMergeJoin
Left keys [1]: [jobId#57641]
Right keys [1]: [jobId#57629]
Join type: FullOuter
Join condition: None
(35) Project
Output [92]: [coalesce(jobId#57641, jobId#57629) AS jobId#57718, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634, 10000000096 AS replenishmentRunId#57800L, 6944346648028224943 AS tenantId#57883, 8378 AS cornerstoneTenantId#57967L, Daily_grocery AS marketUnit#58052, STANDARD AS scenario#58138, 9aa5a928-b219-375b-ac11-54040e15cd53 AS activityId#58225, BufferDataSnapShot AS activityType#58313, PrepareDataSnapshotWorkflow AS workflowType#58402, b23ef421-1eee-3d30-a73f-c7878eb04008 AS workflowId#58492, 1 AS activityAttempt#58583, AS gitCommitId#58768]
Input [82]: [jobId#57641, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, jobId#57629, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634]
(36) Sort
Input [92]: [jobId#57718, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634, replenishmentRunId#57800L, tenantId#57883, cornerstoneTenantId#57967L, marketUnit#58052, scenario#58138, activityId#58225, activityType#58313, workflowType#58402, workflowId#58492, activityAttempt#58583, gitCommitId#58768]
Arguments: [activityType#58313 ASC NULLS FIRST], false, 0
(37) WriteFiles
Input [92]: [jobId#57718, stageCount#57476L, totalTaskCount#57477L, minTaskCountPerStage#57478L, maxTaskCountPerStage#57479L, medianTaskCountPerStage#57480L, bytesSpilledToDiskSerializedTotal#57481L, bytesSpilledToDiskSerializedMin#57482L, bytesSpilledToDiskSerializedMax#57483L, bytesSpilledToDiskSerializedMedian#57484L, taskCpuTimeTotal#57485L, taskCpuTimeMin#57486L, taskCpuTimeMax#57487L, taskCpuTimeMedian#57488L, taskDeserializeCpuTimeTotal#57489L, taskDeserializeCpuTimeMin#57490L, taskDeserializeCpuTimeMax#57491L, taskDeserializeCpuTimeMedian#57492L, taskDeserializeTimeTotal#57493L, taskDeserializeTimeMin#57494L, taskDeserializeTimeMax#57495L, taskDeserializeTimeMedian#57496L, taskRunTimeTotal#57497L, taskRunTimeMin#57498L, taskRunTimeMax#57499L, taskRunTimeMedian#57500L, jvmGcTimeTotal#57501L, jvmGcTimeMin#57502L, jvmGcTimeMax#57503L, jvmGcTimeMedian#57504L, bytesSpilledToDiskRawTotal#57505L, bytesSpilledToDiskRawMin#57506L, bytesSpilledToDiskRawMax#57507L, bytesSpilledToDiskRawMedian#57508L, resultSerializationTimeTotal#57509L, resultSerializationTimeMin#57510L, resultSerializationTimeMax#57511L, resultSerializationTimeMedian#57512L, resultSizeToDriverTotal#57513L, resultSizeToDriverMin#57514L, resultSizeToDriverMax#57515L, resultSizeToDriverMedian#57516L, taskPeakExecutionMemoryMin#57517L, taskPeakExecutionMemoryMax#57518L, taskPeakExecutionMemoryMedian#57519L, persistencyBytesReadTotal#57520L, persistencyBytesReadMin#57521L, persistencyBytesReadMax#57522L, persistencyBytesReadMedian#57523L, persistencyRecordsReadTotal#57524L, persistencyRecordsReadMin#57525L, persistencyRecordsReadMax#57526L, persistencyRecordsReadMedian#57527L, persistencyBytesWrittenTotal#57528L, persistencyBytesWrittenMin#57529L, persistencyBytesWrittenMax#57530L, persistencyBytesWrittenMedian#57531L, persistencyRecordsWrittenTotal#57532L, persistencyRecordsWrittenMin#57533L, persistencyRecordsWrittenMax#57534L, persistencyRecordsWrittenMedian#57535L, shuffleConsumeRecordsReadTotal#57536L, shuffleConsumeRemoteBlocksFetchedTotal#57537L, shuffleConsumeLocalBlocksFetchedTotal#57538L, shuffleConsumeTotalBlocksFetchedTotal#57539L, shuffleConsumeRemoteBytesReadTotal#57540L, shuffleConsumeLocalBytesReadTotal#57541L, shuffleConsumeTotalBytesReadTotal#57542L, shuffleConsumeRemoteBytesReadToDiskTotal#57543L, shuffleConsumeFetchWaitTimeTotal#57544L, shuffleProduceBytesWrittenTotal#57545L, shuffleProduceRecordsWrittenTotal#57546L, shuffleProduceWriteTimeTotal#57547L, peakExecutionMemory#57622L, peakExecutionMemoryCliqueDuration#57623L, peakExecutionMemoryCliqueSize#57624L, rddStorageDiskSizeTotal#57630L, rddStorageMemSizeTotal#57631L, rddNumCachedPartitionsTotal#57632, rddNumPartitionsTotal#57633, numberOfExecutors#57634, replenishmentRunId#57800L, tenantId#57883, cornerstoneTenantId#57967L, marketUnit#58052, scenario#58138, activityId#58225, activityType#58313, workflowType#58402, workflowId#58492, activityAttempt#58583, gitCommitId#58768]
(38) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/internal/performance-metrics/10000000096, false, [activityType#58313], Parquet, [__partition_columns=["activityType"], path=hdlfs://2e93940d-4be8-4f12-830d-f0b8d392c03a.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/internal/performance-metrics/10000000096], Append, [jobId, stageCount, totalTaskCount, minTaskCountPerStage, maxTaskCountPerStage, medianTaskCountPerStage, bytesSpilledToDiskSerializedTotal, bytesSpilledToDiskSerializedMin, bytesSpilledToDiskSerializedMax, bytesSpilledToDiskSerializedMedian, taskCpuTimeTotal, taskCpuTimeMin, taskCpuTimeMax, taskCpuTimeMedian, taskDeserializeCpuTimeTotal, taskDeserializeCpuTimeMin, taskDeserializeCpuTimeMax, taskDeserializeCpuTimeMedian, taskDeserializeTimeTotal, taskDeserializeTimeMin, taskDeserializeTimeMax, taskDeserializeTimeMedian, taskRunTimeTotal, taskRunTimeMin, ... 68 more fields]
(39) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true