Find Duplicate File in System
Given a list of directory info including directory path, and all the files with contents in this directory, you need to find out all the groups of duplicate files in the file system in terms of their paths.
A group of duplicate files consists of at least two files that have exactly the same content.
A single directory info string in the input list has the following format:
"root/d1/d2/.../dm f1.txt(f1_content) f2.txt(f2_content) ... fn.txt(fn_content)"
It means there are n files (f1.txt
, f2.txt
... fn.txt
with content f1_content
, f2_content
... fn_content
, respectively) in directory root/d1/d2/.../dm
. Note that n >= 1 and m >= 0. If m = 0, it means the directory is just the root directory.
The output is a list of group of duplicate file paths. For each group, it contains all the file paths of the files that have the same content. A file path is a string that has the following format:
"directory_path/file_name.txt"
Example 1:
Input:
["root/a 1.txt(abcd) 2.txt(efgh)", "root/c 3.txt(abcd)", "root/c/d 4.txt(efgh)", "root 4.txt(efgh)"]
Output:
[["root/a/2.txt","root/c/d/4.txt","root/4.txt"],["root/a/1.txt","root/c/3.txt"]]
class Solution {
public static List<List<String>> findDuplicate(String[] paths) {
Map<String, List<String>> map = new HashMap<>();
for(String path : paths) {
String[] tokens = path.split(" ");
for(int i = ; i < tokens.length; i++) {
String file = tokens[i].substring(, tokens[i].indexOf('('));
String content = tokens[i].substring(tokens[i].indexOf('(') + , tokens[i].indexOf(')'));
map.putIfAbsent(content, new ArrayList<>());
map.get(content).add(tokens[] + "/" + file);
}
}
return map.values().stream().filter(e -> e.size() > ).collect(Collectors.toList());
}
}
Follow Up questions:
- Imagine you are given a real file system, how will you search files? DFS or BFS ?
The answer depends on the tree structure. If the branching factor (n) and depth (d) are high, then BFS will take up a lot of memory O(d^n). For DFS, the space complexity is generally the height of the tree - O(d).
- If the file content is very large (GB level), how will you modify your solution?
- If you can only read the file by 1kb each time, how will you modify your solution?
- What is the time complexity of your modified solution? What is the most time consuming part and memory consuming part of it? How to optimize?
- How to make sure the duplicated files you find are not false positive?
Question 1:
core idea: DFS
Reason: if depth of directory is not too deeper, which is suitable to use DFS, comparing with BFS.
Question-2:
If the file content is very large (GB level), how will you modify your solution?
Answer:
core idea: make use of meta data, like file size before really reading large content.
Two steps:
- DFS to map each size to a set of paths that have that size: Map<Integer, Set>
- For each size, if there are more than 2 files there, compute hashCode of every file by MD5, if any files with the same size have the same hash, then they are identical files: Map<String, Set>, mapping each hash to the Set of filepaths+filenames. This hash id's are very very big, so we use the Java library BigInteger.
To optimize Step-2. In GFS, it stores large file in multiple "chunks" (one chunk is 64KB). we have meta data, including the file size, file name and index of different chunks along with each chunk's checkSum(the xor for the content). For step-2, we just compare each file's checkSum.
Disadvantage: there might be flase positive duplicates, because two different files might share the same checkSum.
Question-3:
If you can only read the file by 1kb each time, how will you modify your solution?
Answer:
- makeHashQuick Function is quick but memory hungry, might likely to run with java -Xmx2G or the likely to increase heap space if RAM avaliable.
- we might need to play with the size defined by "buffSize" to make memory efficient.
Question-4:
What is the time complexity of your modified solution? What is the most time-consuming part and memory consuming part of it? How to optimize?
Answer:
- hashing part is the most time-consuming and memory consuming.
- optimize as above mentioned, but also introduce false positive issue.
Question-5:
How to make sure the duplicated files you find are not false positive?
Answer:
Question-2-Answer-1 will avoid it.
We need to compare the content chunk by chunk when we find two "duplicates" using checkSum.
In preparing for my Dropbox interview, I came across this problem and really wanted to find the ideas behind the follow up questions (as these were the questions that the interviewer was most interested in, not the code itself). Since this is the only post with the follow up discussion, i'll comment here! @yujun gave a great solution above and I just wanted to add a bit more to help future interviewees.
To find duplicate files, given input of String array is quite easy. Loop through each String and keep a HashMap of Strings to Set/Collection of Strings: mapping the contents of each file to a set of paths with filename concatenated.
For me, instead of given a list of paths, I was given a Directory and asked to return List of List of duplicate files for all under it. I chose to represent a Directory like:
class Directory{
List<Directory> subDirectories;
List<File> files;
}
Given a directory, you are asked how you can find duplicate files given very large files. The idea here is that you cannot store contents in memory, so you need to store the file contents in disk. So you can hash each file content and store the hash as a metadata field for each file. Then as you perform your search, store the hash instead the file's contents in memory. So the idea is you can do a DFS through the root directory and create a HashMap<String, Set<String>>
mapping each hash to the Set of filepaths + filenames that correspond to that hash's content.
(Note: You can choose BFS / DFS to traverse the Path. I chose DFS as it is more memory efficient and quicker to code up.)
Follow Up: This is great, but it requires you to compute the hash for every single file once, which can be expensive for large files. Is there anyway you can avoid computing the hash for a file?
One approach is to also maintain a metadata field for each file's size on disk. Then you can take a 2 pass approach:
- DFS to map each size to a set of paths that have that size
- For each size, if there are more than 2 files there, compute hash of every file, if any files with the same size have the same hash, then they are identical files.
This way, you only compute hashes if you have multiple files with the same size. So when you do a DFS, you can create a HashMap<Integer, Set<String>>
, mapping each file's size to the list of file paths that have that size. Loop through each String in each set, get its hash, check if it exists in your set, if so, add it to your List<String> res
otherwise add it into the set. In between each key (switching file sizes), you can add your res to your List<List<String>>
.
Just want to share my humble opinions for discussion:
If anyone has a better solution, I would appreciate it if you'd like to correct and enlighten me:-)
Question 2:
In real-world file system, we usually store large file in multiple "chunks" (in GFS, one chunk is 64 MB),so we have meta data recording the file size,file name and index of different chunks along with each chunk's checkSum (the xor for the content).
So when we upload a file, we record the meta data as mentioned above.
When we need to check for duplicates, we could simply check the meta data:
1.Check if files are of the same size;
2.if step 1 passes, compare the first chunk's checkSum
3.if step 2 passes, check the second checkSum
...
and so on.
There might be false positive duplicates, because two different files might share the same checkSum.
Question 3:
In the way mentioned above, we could read the meta data instead of the entire file, and compare the information KB by KB.
Question 5:
Using checkSum, we could quickly and accurately find out the non-duplicated files. But to totally avoid getting the false positive, we need to compare the content chunk by chunk when we find two "duplicates" using checkSum.
Find Duplicate File in System的更多相关文章
- LC 609. Find Duplicate File in System
Given a list of directory info including directory path, and all the files with contents in this dir ...
- [LeetCode] Find Duplicate File in System 在系统中寻找重复文件
Given a list of directory info including directory path, and all the files with contents in this dir ...
- [Swift]LeetCode609. 在系统中查找重复文件 | Find Duplicate File in System
Given a list of directory info including directory path, and all the files with contents in this dir ...
- LeetCode Find Duplicate File in System
原题链接在这里:https://leetcode.com/problems/find-duplicate-file-in-system/description/ 题目: Given a list of ...
- [leetcode-609-Find Duplicate File in System]
https://discuss.leetcode.com/topic/91430/c-clean-solution-answers-to-follow-upGiven a list of direct ...
- 609. Find Duplicate File in System
Given a list of directory info including directory path, and all the files with contents in this dir ...
- 【leetcode】609. Find Duplicate File in System
题目如下: Given a list of directory info including directory path, and all the files with contents in th ...
- 【LeetCode】609. Find Duplicate File in System 解题报告(Python & C++)
作者: 负雪明烛 id: fuxuemingzhu 个人博客: http://fuxuemingzhu.cn/ 目录 题目描述 题目大意 解题方法 日期 题目地址:https://leetcode.c ...
- HDU 3269 P2P File Sharing System(模拟)(2009 Asia Ningbo Regional Contest)
Problem Description Peer-to-peer(P2P) computing technology has been widely used on the Internet to e ...
随机推荐
- Delphi 实现Ping命令
Delphi 实现Ping命令 unit FtPing; interface uses Windows, SysUtils, Classes, Controls, Winsock, StdCtrls ...
- Unable to copy file, Access to the path is denied
Unable to copy file, Access to the path is denied http://stackoverflow.com/questions/7130136/unable- ...
- springboot注释层分解图
- luogu 2294 [HNOI2005]狡猾的商人 差分约束
一个差分约束模型,只需判一下有没有负环即可. #include <bits/stdc++.h> #define N 103 #define M 2004 #define setIO(s) ...
- [Luogu] 列队
https://www.luogu.org/problemnew/show/P3960 如果 x = 1,相当于维护一条链,每次取出第 k 个数放在序列末尾假设有 n + m + q 个位置,每个位置 ...
- 洛谷P2789 直线交点数 [数论,递归]
题目传送门 题目描述 平面上有N条直线,且无三线共点,那么这些直线能有多少不同的交点数? 输入格式 一个正整数N 输出格式 一个整数表示方案总数 输入输出样例 输入 #1 4 输出 #1 5 说明/提 ...
- Java异常Error和Exception
简述 程序运行时,发生了不被期望的结果,阻止了程序按照预期正常执行,这就是异常.世界上没有不出错的程序,只有正确处理好意外情况,才能保证程序的可靠性. Java 语言在设计之初就提供了相对完善的异常处 ...
- UVA 12501 Bulky process of bulk reduction ——(线段树成段更新)
和普通的线段树不同的是,查询x~y的话,给出的答案是第一个值的一倍加上第二个值的两倍一直到第n个值的n倍. 思路的话,就是关于query和pushup的方法.用一个新的变量sum记录一下这个区间里面按 ...
- HDFS CheckPoint && SavePoint
HDFS CheckPoint && SavePoint 标签(空格分隔): Hadoop HDFS CheckPoint HDFS 将文件系统的元数据信息存放在 fsimage 和一 ...
- 事件处理机制与Handler消息传递机制
一.基于监听的事件处理机制 基于监听的时间处理机制模型: 事件监听机制中由事件源,事件,事件监听器三类对象组成 处理流程如下: Step 1:为某个事件源(组件)设置一个监听器,用于监听用户操作 St ...