Building a cli tool to remove duplicates in a drive using python

Hi guys,

Today you're going to learn about one of real world use cases of Python apart from building website and Datascience pipelines, you will learn how to remove all duplicates files on your drives or folder no matter of their names using Python.

Why ?

In many situations you find yourself having duplicates files on your drive in separate folders, and sometimes with different names thus when it comes to tracking and checking them manually it can be tedious work, thus why we need Python to do this since its pretty good at automating boring stuffs like this one.

How will Python remove duplicates ?

With Python we are going to write a python script that will recursively scan all files in multiple folder levels and keep track of all the duplicates and remove them as it finish scanning.

Now the next the next question you might have is how you're going to compare if two files are identical with Python ?

If we were to read the whole file and then compare it to the rest of the files recursively through the given directory it will take a very long time then how do we do it?

The answer is hashing, with hashing we can generate a given short string of letters and numbers to act as the identity or representation of a whole whether its video , images or music files, and then if we find any duplicate hash of the file, it directly means that its a duplicate file.

Alt Text

There a variety of hashing algorithms out there such as

  • md5
  • sha1
  • sha224, sha256, sha384 and sha512

Lets do some coding

Hashing in Python is pretty straight forward we are going to use hashlib library which comes by default  with Python standard library.

To see all supported hashlib supported algorithms do this;

>>> import hashlib 
>>> hashlib.algorithms_available
{'sha3_512', 'shake_256', 'sha224', 'sha512_256', 'whirlpool', 'sha384', 'blake2b', 'sha512', 'sha512_224', 'ripemd160', 'blake2s', 'sha3_256', 'md4', 'sha3_224', 'sha1', 'shake_128', 'sha3_384', 'sm3', 'md5', 'sha256', 'md5-sha1'}

Example of Usage

Below is an example of how we hash stuff using hashlib,  we are going to hash of a string in Python using md5 hashing algorithms

>>> import hashlib
>>> example_text = "Duplython is amazing".encode('utf-8')
>>> hashlib.md5(example_text).hexdigest()
'73a14f46eadcc04f4e04bec8eb66f2ab'

It’s straight forward,  you just need to import hashlib and then use md5 method to create hash and finally use hexdigest to generate string of the hash.

The above example has shown us how to hash a string but as we look in relation to the project we are about to build we are more concerned with files rather than strings, another question arises;

How do we hash files?

Hashing files is similar to hashing string with just minor differences, during the hashing file we first need to open the file in binary and then generate a hash of the file binary value.

Hashing File

Let's say you have simple text document on your project directory with name learn.txt, This is how we will do it.

>>> import hashlib
>>> file = open('learn.txt', 'rb').read()
>>> hashlib.md5(file).hexdigest()
'0534cf6d5816c4f1ace48fff75f616c9'

As you can see above even If you try to generate the hashes for a second time, The value of the generated hash doesn't change as long as its the same file.

The challenge arises when we try to read a very large file, It gonna take a while loading it therefore instead of waiting for the whole file to memory we can keep computing the hashes as we read the file.

Computing hashes while reading the file requires us to read the file in blocks of a given size and keep updating the hashes as we keep reading the file until the complete hashing the whole file.

Doing this way could save us a lot of waiting time that we could use on waiting for the whole file to be ready.

Example of Usage

>>> import hashlib
>>> block_size = 1024
>>> hash = hashlib.md5()
>>> with open('learn.txt', 'rb') as file:
...     block = file.read(block_size)
...     while len(block)>0:
...             hash.update(block)
...             block = file.read(block_size)
...     print(hash)
... 
0534cf6d5816c4f1ace48fff75f616c9

As you can see hash has not changed it still the same, therefore we are ready to go to building our python too to do the job.

But wait for hashing is just one step we need a way to actually removes the duplicates, we gonna use built python module OS in deleting duplicates.

We gonna use Python OS remove( ) method to remove the duplicates on our drive.

Let’s try deleting learn.txt with os module

Example of Usage (os module):

>>> import os
>>> os.listdir()
['Desktop-File-cleaner', '.git', 'learn.txt', 'app.py', 'README.md']
>>> os.remove('learn.txt')
>>> os.listdir()
['Desktop-File-cleaner', '.git', 'app.py', 'README.md']

Well, that’s simple you just call remove ( ) with a parameter of the name of the file you wanna remove done. now let’s go build our application.

Building our cleaning Tool

importing necessary libraries

import time
import os
from hashlib import sha256

I'm a huge fan of Object-oriented programming and on this article, we gonna build our tool as a single class, below is just as exoskeleton class for our code.

import time
import os
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')
        
    def main(self)->None:
      self.welcome()

if __name__ == '__main__':
    App = Duplython()
    App.main()

That’s just initial cover for our Python Program of which when we ran it will just print the welcoming method to the screen

Output :

$ python3 app.py
******************************************************************
****************        DUPLYTHON      ****************************
********************************************************************

----------------        WELCOME        ----------------------------
​
Cleaning .................

We now have to create a simple function to generate hash of a file with a given path using the hashing knowledge we have learned above.

import time
import os
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')
        
   def generate_hash(self, Filename:str)->str:
        Filehash = sha256()
        try:
            with open(Filename, 'rb') as File:
                fileblock = File.read(self.block_size)
                while len(fileblock)>0:
                    Filehash.update(fileblock)
                    fileblock = File.read(self.block_size)
                Filehash = Filehash.hexdigest()
            return Filehash
        except:
            return False
        
    def main(self)->None:
      self.welcome()

if __name__ == '__main__':
    App = Duplython()
    App.main()

Now Let's implement our program Logic

Now after we made a function to generate hash per a given path of the file, Now let's implement Where by we will be comparing those hashes and removing any found duplicate.

I have made a simple function called clean( ) just to that as shown below.

import time
import os
from hashlib import sha256
​
class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0
​
    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')
        
   def generate_hash(self, Filename:str)->str:
        Filehash = sha256()
        try:
            with open(Filename, 'rb') as File:
                fileblock = File.read(self.block_size)
                while len(fileblock)>0:
                    Filehash.update(fileblock)
                    fileblock = File.read(self.block_size)
                Filehash = Filehash.hexdigest()
            return Filehash
        except:
            return False
​
    def clean(self)->None:
        all_dirs = [path[0] for path in os.walk('.')]
        for path in all_dirs:
            os.chdir(path)
            All_Files =[file for file in os.listdir() if os.path.isfile(file)]
            for file in All_Files:
                filehash = self.generate_hash(file)
                if not filehash in self.File_hashes:
                    if filehash:                       
                        self.File_hashes.append(filehash)
                        #print(file)
                else:
                    byte_saved = os.path.getsize(file); self.count_cleaned+=1
                    self.Total_bytes_saved+=byte_saved
                    os.remove(file); filename = file.split('/')[-1]
                    print(filename, '.. cleaned ')
            os.chdir(self.home_dir)
        
    def main(self)->None:
      self.welcome();self.clean()
​
if __name__ == '__main__':
    App = Duplython()
    App.main()

Now Our program is nearly complete, we now have to add a simple method to print the summary of the cleaning process.
to the command line and how many bytes of memory has been saved.

I have implemented the method cleaning_summary () just to do that, print out the summary of the cleaning process to the screen which completes our python tool as shown below

import time
import os
import shutil
from hashlib import sha256

class Duplython:
    def __init__(self):
        self.home_dir = os.getcwd(); self.File_hashes = []
        self.Cleaned_dirs = []; self.Total_bytes_saved = 0
        self.block_size = 65536; self.count_cleaned = 0

    def welcome(self)->None:
        print('******************************************************************')
        print('****************        DUPLYTHON      ****************************')
        print('********************************************************************\n\n')
        print('----------------        WELCOME        ----------------------------')
        time.sleep(3)
        print('\nCleaning .................')

    def generate_hash(self, Filename:str)->str:
        Filehash = sha256()
        try:
            with open(Filename, 'rb') as File:
                fileblock = File.read(self.block_size)
                while len(fileblock)>0:
                    Filehash.update(fileblock)
                    fileblock = File.read(self.block_size)
                Filehash = Filehash.hexdigest()
            return Filehash
        except:
            return False

    def clean(self)->None:
        all_dirs = [path[0] for path in os.walk('.')]
        for path in all_dirs:
            os.chdir(path)
            All_Files =[file for file in os.listdir() if os.path.isfile(file)]
            for file in All_Files:
                filehash = self.generate_hash(file)
                if not filehash in self.File_hashes:
                    if filehash:                       
                        self.File_hashes.append(filehash)
                        #print(file)
                else:
                    byte_saved = os.path.getsize(file); self.count_cleaned+=1
                    self.Total_bytes_saved+=byte_saved
                    os.remove(file); filename = file.split('/')[-1]
                    print(filename, '.. cleaned ')
            os.chdir(self.home_dir)
    
    def cleaning_summary(self)->None:
        mb_saved = self.Total_bytes_saved/1048576
        mb_saved = round(mb_saved, 2)
        print('\n\n--------------FINISHED CLEANING ------------')
        print('File cleaned  : ', self.count_cleaned)
        print('Total Space saved : ', mb_saved, 'MB')
        print('-----------------------------------------------')
        
    def main(self)->None:
        self.welcome();self.clean();self.cleaning_summary()

if __name__ == '__main__':
    App = Duplython()
    App.main()

Our app is complete, now to run the application is now complete now go run the application on the specific folder you want to clean and it will iterate recursively over a given folder to find all the files and remove the duplicate one .

Example output :

$ python3 app.py 
******************************************************************
****************        DUPLYTHON      ****************************
********************************************************************
​
​
----------------        WELCOME        ----------------------------
​
Cleaning .................
0(copy).jpeg .. cleaned 
0 (1)(copy).jpeg .. cleaned 
0 (2)(copy).jpeg .. cleaned 

​
--------------FINISHED CLEANING ------------
File cleaned  :  3
Total Space saved :  0.38 MB
-----------------------------------------------

Hope you find this post interesting, would love to have you your Feedback

Follow me on twitter

Kalebu Jordan
I am a Mechatronics engineer and professional Python Developer passionate about innovation, opensource and technical writing, Here I will be writing mostly on Python, DevOps, AI/ML and security
Dar es Salaam, Tanzania