大数据python词频统计之hdfs分发-cacheFile

-cacheFile 分发，文件事先上传至Hdfs上，分发的是一个文件

1.找一篇文章The_Man_of_Property.txt：

He was proud of him! He could not but feel that in similar circumstances he himself would have been tempted to enlarge his replies, but his instinct told him that this taciturnity was the very thing. He sighed with relief, however, when Soames, slowly turning, and without any change of expression, descended from the box.
When it came to the turn of Bosinney’s Counsel to address the Judge, James redoubled his attention, and he searched the Court again and again to see if Bosinney were not somewhere concealed.
Young Chankery began nervously; he was placed by Bosinney’s absence in an awkward position. He therefore did his best to turn that absence to account.
He could not but fear — he said — that his client had met with an accident. He had fully expected him there to give evidence; they had sent round that morning both to Mr. Bosinney’s office and to his rooms (though he knew they were one and the same, he thought it was as well not to say so), but it was not known where he was, and this he considered to be ominous, knowing how anxious Mr. Bosinney had been to give his evidence. He had not, however, been instructed to apply for an adjournment, and in default of such instruction he conceived it his duty to go on. The plea on which he somewhat confidently relied, and which his client, had he not unfortunately been prevented in some way from attending, would have supported by his evidence, was that such an expression as a ‘free hand’ could not be limited, fettered, and rendered unmeaning, by any verbiage which might follow it. He would go further and say that the correspondence showed that whatever he might have said in his evidence, Mr. Forsyte had in fact never contemplated repudiating liability on any of the work ordered or executed by his architect. The defendant had certainly never contemplated such a contingency, or, as was demonstrated by his letters, he would never have proceeded with the work — a work of extreme delicacy, carried out with great care and efficiency, to meet and satisfy the fastidious taste of a connoisseur, a rich man, a man of property. He felt strongly on this point, and feeling strongly he used, perhaps, rather strong words when he said that this action was of a most unjustifiable, unexpected, indeed — unprecedented character. If his Lordship had had the opportunity that he himself had made it his duty to take, to go over this very fine house and see the great delicacy and beauty of the decorations executed by his client — an artist in his most honourable profession — he felt convinced that not for one moment would his Lordship tolerate this, he would use no stronger word than daring attempt to evade legitimate responsibility.
Taking the text of Soames’ letters, he lightly touched on ‘Boileau v. The Blasted Cement Company, Limited.’ “It is doubtful,” he said, “what that authority has decided; in any case I would submit that it is just as much in my favour as in my friend’s.” He then argued the ‘nice point’ closely. With all due deference he submitted that Mr. Forsyte’s expression nullified itself. His client not being a rich man, the matter was a serious one for him; he was a very talented architect, whose professional reputation was undoubtedly somewhat at stake. He concluded with a perhaps too personal appeal to the Judge, as a lover of the arts, to show himself the protector of artists, from what was occasionally — he said occasionally — the too iron hand of capital. “What,” he said, “will be the position of the artistic professions, if men of property like this Mr. Forsyte refuse, and are allowed to refuse, to carry out the obligations of the commissions which they have given.” He would now call his client, in case he should at the last moment have found himself able to be present.
The name Philip Baynes Bosinney was called three times by the Ushers, and the sound of the calling echoed with strange melancholy throughout the Court and Galleries.
The crying of this name, to which no answer was returned, had upon James a curious effect: it was like calling for your lost dog about the streets. And the creepy feeling that it gave him, of a man missing, grated on his sense of comfort and security-on his cosiness. Though he could not have said why, it made him feel uneasy.
He looked now at the clock — a quarter to three! It would be all over in a quarter of an hour. Where could the young fellow be?
It was only when Mr. Justice Bentham delivered judgment that he got over the turn he had received.
Behind the wooden erection, by which he was fenced from more ordinary mortals, the learned Judge leaned forward. The electric light, just turned on above his head, fell on his face, and mellowed it to an orange hue beneath the snowy crown of his wig; the amplitude of his robes grew before the eye; his whole figure, facing the comparative dusk of the Court, radiated like some majestic and sacred body. He cleared his throat, took a sip of water, broke the nib of a quill against the desk, and, folding his bony hands before him, began.
A divorce! Thus close, the word was paralyzing, so utterly at variance with all the principles that had hitherto guided his life. Its lack of compromise appalled him; he felt — like the captain of a ship, going to the side of his vessel, and, with his own hands throwing over the most precious of his bales. This jettisoning of his property with his own hand seemed uncanny to Soames. It would injure him in his profession: He would have to get rid of the house at Robin Hill, on which he had spent so much money, so much anticipation — and at a sacrifice. And she! She would no longer belong to him, not even in name! She would pass out of his life, and he — he should never see her again!
He traversed in the cab the length of a street without getting beyond the thought that he should never see her again!
r her hair; and at this scent the burning sickness of his jealousy seized him again.
Struggling into his fur,the watch was a three-cornered note addressed ‘Soames Forsyte,’ in‘Ierceived under the softness and immobility of this figure something desperate and resolved; something not to be turned away, something dangerous. She tore off her hat, and, putting both hands to her brow, pressed back the bronze mass of her hair.

2.找一篇白名单文章white_list如下：

word
He
under
only

3.将文章上传至hdfs文件系统内

hadoop   fs   -put   The_Man_of_Property.txt   /mapreduce
hadoop   fs   -put   white_list   /mapreduce

4.map端代码如下：

#!usr/bin/python
import sys
def read_local_file(file):
        word_set = set()
        file_in = open (file,'r')
        for line in file_in:
                word = line.strip()
                word_set.add(word)
        return word_set
def mapper_func(file):
        word_set=read_local_file(file)

        for line in sys.stdin:
                ss=line.strip().split()
                for word in ss:
                        word.strip()
                        if word != "" and (word in word_set):
                                print "%s\t%s"%(word,"")

if __name__ == "__main__":
        func = getattr(sys.modules[__name__],sys.argv[1])
        args = None
        if len(sys.argv) > 1:
                args = sys.argv[2:]
        func(*args)

5.reduce端代码如下

#!usr/bin/python
import sys
def reducer_func():
        word="None"
        sum=0
        for line in sys.stdin:
                ss=line.split()
                cur_word=ss[0]
                cnt=int(ss[1])
                if cur_word!=word:
                        if word!="None":
                                print "%s\t%s"%(word,sum)
                        word=cur_word
                        sum=0
                else:
                        sum+=cnt
        print "%s\t%s"%(word,sum)
if __name__ == "__main__":
        func = getattr(sys.modules[__name__],sys.argv[1])
        args = None
        if len(sys.argv) > 1:
                args=sys.argv[2:]
        func(*args)

5.运行shell脚本如下：

HADOOP="/usr/local/src/hadoop-1.2.1/bin/hadoop"
HADOOP_STREAMING="/usr/local/src/hadoop-1.2.1/contrib/streaming/hadoop-streaming-1.2.1.jar"
INPUT_PATH="/mapreduce/The_Man_of_Property.txt"
OUTPUT_PATH="/mapreduce/out"
$HADOOP fs -rmr $OUTPUT_PATH
$HADOOP jar $HADOOP_STREAMING \
        -input "$INPUT_PATH" \
        -output "$OUTPUT_PATH" \
        -mapper "python map.py mapper_func ABC" \
        -reducer "python red.py reducer_func" \
        -file "./map.py"\
        -file "./red.py"\
        -cacheFile "hdfs://master:9000/mapreduce/white_list#ABC" #注意这块ABC为代名词，代表分发文件，后续将分发到各个节点的临时目录作为参数传入map函数

6.运行shell脚本

sh  run.sh