The current information explosion has resulted in an increasing number of applications that need to deal with large volumes of data. While many of the data contains useless redundancy data, especially in mass media, web crawler/analytic fields, wasted many precious resources (power, bandwidth, CPU and storage, etc.). This has resulted in an increased interest in algorithms that process the input data in restricted ways.

But traditional hash algorithms have two problems, first it assumes that the data fits in main memory, it is unreasonable when dealing with massive data such as multimedia data, web crawler/analytic repositories and so on. And second, traditional hash can only indentify the identical data. this brings to light the importance of simhash.

 

Simhash 5 steps: Tokenize, Hash, Weigh Values, Merge, Dimensionality Reduction

  • tokenize

    • tokenize your data, assign weights to each token, weights and tokenize function are depend on your business

  • hash (md5, SHA1)

    • calculate token's hash value and convert it to binary (101011 )

  • weigh values

    • for each hash value, do hash*w, in this way: (101011 ) -> (w,-w,w,-w,w,w)

  • merge

    • add up tokens' values, to merge to 1 hash, for example, merge (4 -4 -4 4 -4 4) and (5 -5 5 -5 5 5) , results to (4+5 -4+-5 -4+5 4+-5 -4+5 4+5),which is (9 -9 1 -1 1)

  • Dimensionality Reduction

    • Finally, signs of elements of V corresponds to the bits of the final fingerprint, for example (9 -9 1 -1 1) -> (1 0 1 0 1), we get 10101 as the fingerprint.

How to use SimHash fingerprints?

Hamming distance can be used to find the similarity between two given data, calculate the Hamming distance between 2 fingerprints.

Based on my experience, for 64 bit SimHash values, with elaborate weight values,  distance of similar data often differ appreciably in magnitude from those unsimilar data.

how to calculate Hamming distance:

  XOR, 只有两个位不同时结果是1 ,否则为0,两个二进制value“异或”后得到1的个数 为海明距离 。

SimHash algorithm, introduced by Charikarand is patented by Google.

simhash 0.1.0 : Python Package Index

[SimHash] the Hash-based Similarity Detection Algorithm的更多相关文章

  1. A Node Influence Based Label Propagation Algorithm for Community detection in networks 文章算法实现的疑问

    这是我最近看到的一篇论文,思路还是很清晰的,就是改进的LPA算法.改进的地方在两个方面: (1)结合K-shell算法计算量了节点重重要度NI(node importance),标签更新顺序则按照NI ...

  2. VIPS: a VIsion based Page Segmentation Algorithm

    VIPS: a VIsion based Page Segmentation Algorithm VIPS: a VIsion based Page Segmentation Algorithm In ...

  3. MBMD(MobileNet-based tracking by detection algorithm)作者答疑

    If you fail to install and run this tracker, please email me (zhangyunhua@mail.dlut.edu.cn) Introduc ...

  4. anomaly detection algorithm

    anomaly detection algorithm 以上就是异常监测算法流程

  5. Floyd判圈算法 Floyd Cycle Detection Algorithm

    2018-01-13 20:55:56 Floyd判圈算法(Floyd Cycle Detection Algorithm),又称龟兔赛跑算法(Tortoise and Hare Algorithm) ...

  6. Floyd's Cycle Detection Algorithm

    Floyd's Cycle Detection Algorithm http://www.siafoo.net/algorithm/10 改进版: http://www.siafoo.net/algo ...

  7. 从时序异常检测(Time series anomaly detection algorithm)算法原理讨论到时序异常检测应用的思考

    1. 主要观点总结 0x1:什么场景下应用时序算法有效 历史数据可以被用来预测未来数据,对于一些周期性或者趋势性较强的时间序列领域问题,时序分解和时序预测算法可以发挥较好的作用,例如: 四季与天气的关 ...

  8. 个性探测综述阅读笔记——Recent trends in deep learning based personality detection

    目录 abstract 1. introduction 1.1 个性衡量方法 1.2 应用前景 1.3 伦理道德 2. Related works 3. Baseline methods 3.1 文本 ...

  9. 论文阅读笔记五十二:CornerNet-Lite: Efficient Keypoint Based Object Detection(CVPR2019)

    论文原址:https://arxiv.org/pdf/1904.08900.pdf github:https://github.com/princeton-vl/CornerNet-Lite 摘要 基 ...

随机推荐

  1. Docker实战(五)之端口映射与容器互联

    除了网络访问外,Docker还提供了两个很方便的功能来满足服务访问的基本需求:一个是允许映射容器内应用的服务端口到本地宿主主机;另一个是互联机制实现多个容器间通过容器名来快速访问. 1.端口映射实现访 ...

  2. Linux服务器之间免密同步文件、重启R服务

    机器:ML-01/ML-02/ML-03 需求: 1.在ML-01上自动将文件同步至ML-02/ML-03 2.在ML-01上通过脚本重启ML-02/ML-03上的R服务 说明:以下示例中,ML-02 ...

  3. 【转】Android 4.0 Launcher2源码分析——启动过程分析

    Android的应用程序的入口定义在AndroidManifest.xml文件中可以找出:[html] <manifest xmlns:android="http://schemas. ...

  4. PAT乙级1014

    1014 福尔摩斯的约会 (20 分)   大侦探福尔摩斯接到一张奇怪的字条:我们约会吧! 3485djDkxh4hhGE 2984akDfkkkkggEdsb s&hgsfdk d& ...

  5. 各国货币json文件

    [ {"countryname":"","name":"请选择","currency":" ...

  6. TMS Xdata Server

    Xdata 在TMS中扮演的桥的角色,一年前仔细看过TMS 的源码,当时对流程很清晰,随着时间慢慢的过去,现在该忘记的都忘记了.所以用此文章来记录自己对Xdata还剩下的一点点的记忆... 光有xda ...

  7. MySQL数据表命令

    显示表的相关信息: show table status like "表名": show table status like "表名" \G       格式化, ...

  8. 【CQOI2017】小Q的棋盘

    题面 题解 根据题意,不回头是最好的(显然法) \(dfs\)找到最长链,设长度为\(\mathrm{L}\),然后分类讨论: 如果\(\mathrm{L} > m\),答案就是\(m + 1\ ...

  9. pymysql模块使用教程

    一.操作数据库模板 pymysql是Python中操作mysql的模块,(使用方法几乎和MySQLdb相同,但是在Python3中,mysqldb这个库已经不能继续使用了) 下载安装方法: 方法一. ...

  10. 【Java源码解析】Thread

    简介 线程本质上也是进程.线程机制提供了在同一程序内共享内存地址空间运行的一组线程.对于内核来讲,它就是进程,只是该进程和其他一下进程共享某些资源,比如地址空间.在Java语言里,Thread类封装了 ...