Dealing With Duplicates In Your Bash History

Your Bash history can be a valuable tool when you mess up and need to retrace your steps to find out what went wrong. It’s in your best interest to keep it from getting cluttered.

Some of my readers are fairly new to Linux, so you may not know that you have a Bash history. Well, you probably do. It’s helpfully called .bash_history and it’s a hidden file in your home directory. You can open it with a text editor if you want.

See, every time you write a command into the terminal (unless you take steps to avoid it) that command is stored in a hidden file called .bash_history. The period at the start of the file name means that it’s a hidden file and you’ll need to enable hidden file viewing in your file manager. (It’s quite frequently F4 or under the ‘View’ menu.)

If you open your terminal (press CTRL + ALT + T) and push the up arrow on your keyboard, you’ll see the last command you used in your terminal. Well, it’s a bit more complicated than that, but you’ll see the bottom-most entry in your .bash_history.

If you use your terminal a lot, you’ll have duplicates. It’s just a fact of life. Many commands will be run multiple times, such as performing updates. These clutter up your history and just waste space. You don’t have to keep the duplicates, or even allow duplicates.

De-dupe .bash_history

First, let’s disable duplicates and delete them if they do somehow occur. To do this, you’re going to edit your .bashrc file. As you can see by the . at the start of the file name, the file is hidden by default. You can use the GUI file manager and favorite text editor. Enable viewing hidden files, and just add the following line to the very bottom of that file.

export HISTCONTROL=ignoreboth:erasedups 

Save it, of course.

If you want, you can do it right in the terminal. Open the terminal with the directions above and enter the following command:

nano ~/.bashrc

Notice how there’s no use of ‘sudo’? That’s because you already own these files and you don’t need elevated permissions to edit them. 

Anyhow, use the down arrow and then paste in the text above, just as though you were editing it in your favorite text editor. Then, go ahead and save it by pressing CTRL + X, then Y, and then ENTER.

There… Now Bash should ignore duplicates and erase any new duplicates that it finds.

All well and good?

Maybe…

I mean, it’s not like you just installed your OS today and have a sparkling clean bash history. No, if you’re like me, you’ve probably got a bunch of clutter in there and all sorts of duplicated content. So, you’re going to want to clear those already existing duplicates out as well.

Seeing as you’re already in the terminal, you can use the following command that I once found online and saved to my notes:

tac ~/.bash_history | awk '!a[$0]++' | tac > t; mv t ~/.bash_history

That right there will go ahead and remove any duplicates that it finds in your bash history. You might want to make a backup of the file before trying that command – but I’ve used it so many times that I have complete faith in it. So far, it has only removed duplicates and not broken anything. (Pay most of your attention to the use of awk in the command if you’re wondering how it works.)

And, there you have it. You should have a nice .bash_history that isn’t full of duplicates. I’ve been known to open the file manually and run through it to remove stuff I’m never going to use again, commands that I’ve gotten wrong, or commands that are no longer of any use. 

Addendum

I should also mention that I have been using the (mostly) same history file for a long time. You save it and import it on new computers that will run the same operating system. Personally, I use my bash history fairly often and I like the consistency. I also do the same with my aliases, but that’s a topic for another day.

By the way, if you don’t want to have your command stored in the history, the easiest way to avoid it (on a command-by-command basis) is just to put a space in front of it. The system doesn’t care that there’s a leading space, except to use it as a cue to not store the command in your history. If you’re entering sensitive data into your terminal, you might want to consider that. If it’s just a junk command that you’re never going to see again, you might want to consider that.

Finally, thanks for reading. I appreciate the audience and I enjoy getting my notes online. The new domain is going well and we’ll soon be getting some of the older content transferred. It’s just a bunch of tedious work. I’d ask for help, but there’s not a whole lot anyone can do to help. I’ll just have to find the time to do it. Don’t forget the newsletter!