The current information explosion has resulted in an increasing number of applications that need to deal with large volumes of data. While many of the data contains useless redundancy data, especially in mass media, web crawler/analytic fields, wasted many precious resources (power, bandwidth, CPU and storage, etc.). This has resulted in an increased interest in algorithms that process the input data in restricted ways.

But traditional hash algorithms have two problems, first it assumes that the data fits in main memory, it is unreasonable when dealing with massive data such as multimedia data, web crawler/analytic repositories and so on. And second, traditional hash can only indentify the identical data. this brings to light the importance of simhash.

 

Simhash 5 steps: Tokenize, Hash, Weigh Values, Merge, Dimensionality Reduction

  • tokenize

    • tokenize your data, assign weights to each token, weights and tokenize function are depend on your business

  • hash (md5, SHA1)

    • calculate token's hash value and convert it to binary (101011 )

  • weigh values

    • for each hash value, do hash*w, in this way: (101011 ) -> (w,-w,w,-w,w,w)

  • merge

    • add up tokens' values, to merge to 1 hash, for example, merge (4 -4 -4 4 -4 4) and (5 -5 5 -5 5 5) , results to (4+5 -4+-5 -4+5 4+-5 -4+5 4+5),which is (9 -9 1 -1 1)

  • Dimensionality Reduction

    • Finally, signs of elements of V corresponds to the bits of the final fingerprint, for example (9 -9 1 -1 1) -> (1 0 1 0 1), we get 10101 as the fingerprint.

How to use SimHash fingerprints?

Hamming distance can be used to find the similarity between two given data, calculate the Hamming distance between 2 fingerprints.

Based on my experience, for 64 bit SimHash values, with elaborate weight values,  distance of similar data often differ appreciably in magnitude from those unsimilar data.

how to calculate Hamming distance:

  XOR, 只有两个位不同时结果是1 ,否则为0,两个二进制value“异或”后得到1的个数 为海明距离 。

SimHash algorithm, introduced by Charikarand is patented by Google.

simhash 0.1.0 : Python Package Index

[SimHash] the Hash-based Similarity Detection Algorithm的更多相关文章

  1. A Node Influence Based Label Propagation Algorithm for Community detection in networks 文章算法实现的疑问

    这是我最近看到的一篇论文,思路还是很清晰的,就是改进的LPA算法.改进的地方在两个方面: (1)结合K-shell算法计算量了节点重重要度NI(node importance),标签更新顺序则按照NI ...

  2. VIPS: a VIsion based Page Segmentation Algorithm

    VIPS: a VIsion based Page Segmentation Algorithm VIPS: a VIsion based Page Segmentation Algorithm In ...

  3. MBMD(MobileNet-based tracking by detection algorithm)作者答疑

    If you fail to install and run this tracker, please email me (zhangyunhua@mail.dlut.edu.cn) Introduc ...

  4. anomaly detection algorithm

    anomaly detection algorithm 以上就是异常监测算法流程

  5. Floyd判圈算法 Floyd Cycle Detection Algorithm

    2018-01-13 20:55:56 Floyd判圈算法(Floyd Cycle Detection Algorithm),又称龟兔赛跑算法(Tortoise and Hare Algorithm) ...

  6. Floyd's Cycle Detection Algorithm

    Floyd's Cycle Detection Algorithm http://www.siafoo.net/algorithm/10 改进版: http://www.siafoo.net/algo ...

  7. 从时序异常检测(Time series anomaly detection algorithm)算法原理讨论到时序异常检测应用的思考

    1. 主要观点总结 0x1:什么场景下应用时序算法有效 历史数据可以被用来预测未来数据,对于一些周期性或者趋势性较强的时间序列领域问题,时序分解和时序预测算法可以发挥较好的作用,例如: 四季与天气的关 ...

  8. 个性探测综述阅读笔记——Recent trends in deep learning based personality detection

    目录 abstract 1. introduction 1.1 个性衡量方法 1.2 应用前景 1.3 伦理道德 2. Related works 3. Baseline methods 3.1 文本 ...

  9. 论文阅读笔记五十二:CornerNet-Lite: Efficient Keypoint Based Object Detection(CVPR2019)

    论文原址:https://arxiv.org/pdf/1904.08900.pdf github:https://github.com/princeton-vl/CornerNet-Lite 摘要 基 ...

随机推荐

  1. C. Serval and Parenthesis Sequence 【括号匹配】 Codeforces Round #551 (Div. 2)

    冲鸭,去刷题:http://codeforces.com/contest/1153/problem/C C. Serval and Parenthesis Sequence time limit pe ...

  2. centos6.5添加阿里docker加速器

    1. 配置阿里docker加速器 vi /etc/sysconfig/docker 在文件末尾追加下面两行 other_args="--registry-mirror=https://pl8 ...

  3. Visual Studio 2012 编译错误【error C4996: 'scanf': This function or variable may be unsafe. 】的解决方案

    在VS 2012 中编译 C 语言项目,如果使用了 scanf 函数,编译时便会提示如下错误: error C4996: 'scanf': This function or variable may ...

  4. CentOS中用Nexus搭建maven私服,为Hadoop编译提供本地镜像

    系统: CentOS release 6.6 (Final) Nexus:nexus-2.8.1-bundle.tar.gz,下载地址:https://sonatype-download.global ...

  5. [NOIp2015]运输计划 (二分 $+$ 树上差分)

    #\(\mathcal{\color{red}{Description}}\) \(Link\) 在一棵带有边权的树上,可以选择使一条边权为零.然后对于所有\(M\)条链,使其链长最大值最小. #\( ...

  6. cgroup测试存储设备IOPS分配

    1 使用:创建树并且attach子系统 首先要创建文件系统的挂载点作为树的根 mkdir /cgroup/name mkdir /cgroup/cpu_and_mem Mount这个挂载点到一个或者多 ...

  7. 浅谈User Information List

    [User Information List]用于查看一个site collection所有可以访问的用户信息.一个site collection只有一个User Information List表. ...

  8. CPP/类/成员函数访问权限

  9. jQuery----操作类样式(依托开关灯案例)

    在网页开发中,元素的样式可以在style标签中定义,但是有很多案例需要添加类样式或者删除类样式,可以获取元素调用css()方法改变元素样式,但是这种方法很繁杂,本文利用开关灯案例,小结使用jquery ...

  10. 学习笔记—MapReduce

    MapReduce是什么 MapReduce是一种分布式计算编程框架,是Hadoop主要组成部分之一,可以让用户专注于编写核心逻辑代码,最后以高可靠.高容错的方式在大型集群上并行处理大量数据. Map ...