pig 调试（explain&illerstrate）
grunt> cat t.txt
kw1 2
kw3 1
kw2 4
kw1 5
kw2 2 cat test.pig
A = LOAD '/user/input/t.txt' as (k:chararray,c:int);
B = group A BY k;
C = foreach B generate group,SUM(A.c);
-- DUMP C;
store C into 'test.output';$ pig -e 'illustrate -script test.pig'
2014-05-03 17:11:25,182 [main] INFO org.apache.pig.Main - Logging error messages to: /opt/dataset/pig_1399108285179.log
2014-05-03 17:11:25,330 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.0.3.142:9000
2014-05-03 17:11:25,514 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.0.3.142:9001
2014-05-03 17:11:26,103 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.0.3.142:9000
2014-05-03 17:11:26,104 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.0.3.142:9001
2014-05-03 17:11:26,291 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,305 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,306 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,315 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,330 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,474 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-05-03 17:11:26,475 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2014-05-03 17:11:26,513 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,520 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,521 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,521 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,522 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,531 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,534 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,534 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,597 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,599 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,599 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,600 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,601 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,601 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,608 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,611 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,611 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,639 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,641 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,643 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,643 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,650 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,677 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,679 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,679 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,680 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,680 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,681 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,686 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,688 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,688 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,710 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic?false
2014-05-03 17:11:26,712 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,712 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,713 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,714 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,721 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,724 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,724 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,744 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,746 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,746 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,747 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,747 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,748 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,754 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,757 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,757 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,775 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,775 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,776 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,782 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,784 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,784 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,804 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,806 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,806 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,807 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,807 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,808 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,812 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,821 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,821 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
(kw1,2)
2014-05-03 17:11:26,840 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic?false
2014-05-03 17:11:26,842 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,842 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,842 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,843 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,843 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,849 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,849 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,862 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,863 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,863 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,864 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,865 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,868 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,870 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,870 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic?false
2014-05-03 17:11:26,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,885 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,885 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,887 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,889 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,890 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,901 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,903 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,903 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,903 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,904 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,904 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,906 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,908 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,908 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,919 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic?false
2014-05-03 17:11:26,920 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,920 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,921 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,921 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,922 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,924 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,926 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,926 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,937 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic?false
2014-05-03 17:11:26,938 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,938 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,938 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,939 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,939 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,941 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,943 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,943 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,954 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:11:26,955 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,955 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,956 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,956 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,956 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,959 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-05-03 17:11:26,973 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic?false
2014-05-03 17:11:26,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:11:26,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-05-03 17:11:26,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode.
2014-05-03 17:11:26,975 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-05-03 17:11:26,975 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-05-03 17:11:26,978 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-05-03 17:11:26,980 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30
2014-05-03 17:11:26,980 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
-------------------------------------
| A | k:chararray | c:int |
-------------------------------------
| | kw1 | 2 |
| | kw1 | 5 |
-------------------------------------
-----------------------------------------------------------------------------
| B | group:chararray | A:bag{:tuple(k:chararray,c:int)} |
-----------------------------------------------------------------------------
| | kw1 | {(kw1, 2), (kw1, 5)} |
-----------------------------------------------------------------------------
-----------------------------------------
| C | group:chararray | :long |
-----------------------------------------
| | kw1 | 7 |
-----------------------------------------
-------------------------------------------------
| Store : C | group:chararray | :long |
-------------------------------------------------
| | kw1 | 7 |
-------------------------------------------------$ pig -e 'explain -script test.pig'
2014-05-03 17:19:59,359 [main] INFO org.apache.pig.Main - Logging error messages to: /opt/dataset/pig_1399108799355.log
2014-05-03 17:19:59,497 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.0.3.142:9000
2014-05-03 17:19:59,685 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.0.3.142:9001
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
C: (Name: LOStore Schema: group#19:chararray,#34:long)
|
|---C: (Name: LOForEach Schema: group#19:chararray,#34:long)
| |
| (Name: LOGenerate[false,false] Schema: group#19:chararray,#34:long)ColumnPrune:InputUids=[19, 30]ColumnPrune:OutputUids=[34, 19]
| | |
| | group:(Name: Project Type: chararray Uid: 19 Input: 0 Column: (*))
| | |
| | (Name: UserFunc(org.apache.pig.builtin.IntSum) Type: long Uid: 34)
| | |
| | |---(Name: Dereference Type: bag Uid: 33 Column:[1])
| | |
| | |---A:(Name: Project Type: bag Uid: 30 Input: 1 Column: (*))
| |
| |---(Name: LOInnerLoad[0] Schema: group#19:chararray)
| |
| |---A: (Name: LOInnerLoad[1] Schema: k#19:chararray,c#20:int)
|
|---B: (Name: LOCogroup Schema: group#19:chararray,A#30:bag{#37:tuple(k#19:chararray,c#20:int)})
| |
| k:(Name: Project Type: chararray Uid: 19 Input: 0 Column: 0)
|
|---A: (Name: LOForEach Schema: k#19:chararray,c#20:int)
| |
| (Name: LOGenerate[false,false] Schema: k#19:chararray,c#20:int)ColumnPrune:InputUids=[19, 20]ColumnPrune:OutputUids=[19, 20]
| | |
| | (Name: Cast Type: chararray Uid: 19)
| | |
| | |---k:(Name: Project Type: bytearray Uid: 19 Input: 0 Column: (*))
| | |
| | (Name: Cast Type: int Uid: 20)
| | |
| | |---c:(Name: Project Type: bytearray Uid: 20 Input: 1 Column: (*))
| |
| |---(Name: LOInnerLoad[0] Schema: k#19:bytearray)
| |
| |---(Name: LOInnerLoad[1] Schema: c#20:bytearray)
|
|---A: (Name: LOLoad Schema: k#19:bytearray,c#20:bytearray)RequiredFields:null #-----------------------------------------------
# Physical Plan:
#-----------------------------------------------
C: Store(hdfs://namenode:9000/user/deve_test_user/test.output:org.apache.pig.builtin.PigStorage) - scope-19
|
|---C: New For Each(false,false)[bag] - scope-18
| |
| Project[chararray][0] - scope-12
| |
| POUserFunc(org.apache.pig.builtin.IntSum)[long] - scope-16
| |
| |---Project[bag][1] - scope-15
| |
| |---Project[bag][1] - scope-14
|
|---B: Package[tuple]{chararray} - scope-9
|
|---B: Global Rearrange[tuple] - scope-8
|
|---B: Local Rearrange[tuple]{chararray}(false) - scope-10
| |
| Project[chararray][0] - scope-11
|
|---A: New For Each(false,false)[bag] - scope-7
| |
| Cast[chararray] - scope-2
| |
| |---Project[bytearray][0] - scope-1
| |
| Cast[int] - scope-5
| |
| |---Project[bytearray][1] - scope-4
|
|---A: Load(/user/input/t.txt:org.apache.pig.builtin.PigStorage) - scope-0 2014-05-03 17:20:00,316 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-05-03 17:20:00,326 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2014-05-03 17:20:00,349 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-05-03 17:20:00,349 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node scope-20
Map Plan
B: Local Rearrange[tuple]{chararray}(false) - scope-33
| |
| Project[chararray][0] - scope-34
|
|---C: New For Each(false,false)[bag] - scope-21
| |
| Project[chararray][0] - scope-22
| |
| POUserFunc(org.apache.pig.builtin.IntSum$Initial)[tuple] - scope-23
| |
| |---Project[bag][1] - scope-24
| |
| |---Project[bag][1] - scope-25
|
|---Pre Combiner Local Rearrange[tuple]{Unknown} - scope-35
|
|---A: New For Each(false,false)[bag] - scope-7
| |
| Cast[chararray] - scope-2
| |
| |---Project[bytearray][0] - scope-1
| |
| Cast[int] - scope-5
| |
| |---Project[bytearray][1] - scope-4
|
|---A: Load(/user/input/t.txt:org.apache.pig.builtin.PigStorage) - scope-0--------
Combine Plan
B: Local Rearrange[tuple]{chararray}(false) - scope-37
| |
| Project[chararray][0] - scope-38
|
|---C: New For Each(false,false)[bag] - scope-26
| |
| Project[chararray][0] - scope-27
| |
| POUserFunc(org.apache.pig.builtin.IntSum$Intermediate)[tuple] - scope-28
| |
| |---Project[bag][1] - scope-29
|
|---POCombinerPackage[tuple]{chararray} - scope-31--------
Reduce Plan
C: Store(hdfs://namenode:9000/user/deve_test_user/test.output:org.apache.pig.builtin.PigStorage) - scope-19
|
|---C: New For Each(false,false)[bag] - scope-18
| |
| Project[chararray][0] - scope-12
| |
| POUserFunc(org.apache.pig.builtin.IntSum$Final)[long] - scope-16
| |
| |---Project[bag][1] - scope-30
|
|---POCombinerPackage[tuple]{chararray} - scope-39--------
Global sort: false
----------------
pig 调试（explain&illerstrate）的更多相关文章
- Pig Latin程序设计1
Pig是一个大规模数据分析平台.Pig的基础结构层包括一个产生MapReduce程序的编译器.在编译器中,大规模并行执行依据存在.Pig的语言包括一个叫Pig Latin的文本语言,此语言有如下特性: ...
- Apache Pig入门学习文档（一）
1,Pig的安装 (一)软件要求 (二)下载Pig (三)编译Pig 2,运行Pig (一)Pig的所有执行模式 (二)pig的交互式模式 (三)使用pig脚本 ...
- 微信调试、API、AJAX的调试 SocketLog
SocketLog适合Ajax调试和API调试, 举一个常见的场景,用SocketLog来做微信调试, 我们在做微信API开发的时候,如果API有bug,微信只提示"改公众账号暂时无法提供服 ...
- HiveQL(HiveSQL)跟普通SQL最大区别一直使用PIG，而今也需要兼顾HIVE
HiveQL(Hive SQL)跟普通SQL最大区别 一直使用PIG,而今也需要兼顾HIVE.网上搜了点资料,感觉挺有用,这里翻译过来.翻译估计不太准确,待自己熟悉HIVE后再慢慢总结. * No t ...
- 监听调试web service的好工具TCPMon
监听调试web service的好工具TCPMonhttp://ws.apache.org/commons/tcpmon/download.cgi TCPMon Tutorial Content In ...
- 异步调试神器Slog，“从此告别看日志，清日志文件了”
微信调试.API调试和AJAX的调试的工具,能将日志通过WebSocket输出到Chrome浏览器的console中 — Edit 92 commits 4 branches 3 releases ...
- 本地调试WordPress计划终告失败
小猪本来想把博客的网站数据迁移到自己的电脑上面,mysql数据库还是放在主机供应商,这样就能缓解一下每次写博客时访问速度捉急的状况. 计划是美满的,但是只到实施的时候才发现各种问题.先是直接运行程序时 ...
- Hadoop、Pig、Hive、Storm、NOSQL 学习资源收集
(一)hadoop 相关安装部署 1.hadoop在windows cygwin下的部署: http://lib.open-open.com/view/1333428291655 http://blo ...
- Pig On Mac
Install 首先是 Mac OS 下的安装 1 2 export JAVA_HOME=$(/usr/libexec/java_home) brew install pig Run Pig 运行分为 ...
随机推荐
- Java 获取环境变量
Java 获取环境变量Java 获取环境变量的方式很简单: System.getEnv() 得到所有的环境变量System.getEnv(key) 得到某个环境变量的值 由于某些需要,可能要下载某些 ...
- 数据结构 - 树形选择排序 (tree selection sort) 具体解释 及 代码(C++)
树形选择排序 (tree selection sort) 具体解释 及 代码(C++) 本文地址: http://blog.csdn.net/caroline_wendy 算法逻辑: 依据节点的大小, ...
- VC 下加载 JPG / JPEG / GIF / PNG 图片最简单的方法
VC MFC 提供的 API LoadBitmap / LoadImage 类 CBitmap 等都只能操作 BMP 位图,图标.对于其他常用的 JPG / JPEG / GIF / PNG 格式,它 ...
- VC下加载多种格式图片的方法总结IPicture, CxImage, CImage(AtlImage), CPictureEx
尽管VC有提供相应的API和类来操作bmp位图.图标和(增强)元文件,但却不支持jpg.gif和png等格式的图片,而这几种格式却是常常要用到的.这里我给大家介绍两种办法来操作这些格式的图片. 1.用 ...
- Binary Indexed Tree
我借鉴了这个视频中的讲解的填坑法,我认为非常易于理解.有FQ能力和基本英语听力能力请直接去看视频,并不需要继续阅读. naive 算法 考虑一个这样的场景: 给定一个int数组, 我们想知道它的连续子 ...
- DBMS_XPLAN详细说明
执行计划的组成部分 正确的看执行计划 DBMS_XPLAN 这个包是一个很好查看执行计划,显示很多格式,来分析执行计划中存在的问题 format:控制详细执行计划输出的格式,包含以下内容: BASIC ...
- html5plus 从相册选择图片后获取图片的大小
plus.gallery.pick(function (filePath) { plus.io.resolveLocalFileSystemURL(filePath, function (entry) ...
- Linux与Windows信息交互快捷方法
要把windows上的D盘挂载的Linux上,首先要知道windows的用户名和密码 假设用户名是administrator,密码是123456 首先,在linux上创建一个挂载的目标目录 mkdir ...
- Enable .Net 4.5 in IIS on Windows 8.1
Setting up a new development box for myself I had forgotten all about the necessity to use theaspnet ...
- 优动漫PAINT绘制紫阳花教程
紫阳花是插画.漫画很常见的绘画画材.这个教程非常好懂.而且很方便就能绘制出漂亮的效果.因为这种花一个月内能变化三种颜色,故而人们赋予它的花语是善变.背叛. 教程是简单,呃.... 没有优动漫PAINT ...