Find And Remove Duplicate Files With fdupes

Today’s article has you cleaning up your storage space as you learn to find and remove duplicate files with fdupes. This isn’t something you need to do often and it’s something that could theoretically break your system. If you’re going to remove duplicate files, it’s a good idea to exercise some caution.

I’ve previously shared another way to remove duplicate files:

Find And Remove Duplicate Files With rdfind

I suppose it’s pretty obvious as to why one might want to remove duplicate files. You do so to keep your storage tidy and you do so to make space when space is limited. There are all sorts of ways to make free space and removing duplicate files is just one of them.

The tool we’ll be using this time around is known as ‘fdupes’ and the man page describes it like this:

fdupes – finds duplicate files in a given set of directories

It’s an easy enough application to install and this article shouldn’t be all that long. You’ll find that fdupes is available in your default repositories, or there’s a good chance that it is in there. Of course, this means it’s easy enough to install.

Installing fdupes:

You can install fdupes with a GUI and your software manager, but you can just as easily install it via the terminal. We’ll cover the latter, as it’s the most universal (and, I find, quickest) method. Of course, you’ll need an open terminal. In most cases you can just press CTRL + ALT + T and your default terminal should open.

With your terminal now open, let’s go ahead and install fdupes:

Debian/Ubuntu/derivatives:

SUSE/OpenSUSE/derivatives:

RHEL/Fedora/Rocky/derivatives:

Arch/Manjaro/derivatives:

Gentoo/Calculate/derivatives:

And More! (Just search your default repositories for ‘fdupes’ and it’ll almost certainly be there.)

As you can see, you’ll find that fdupes is available for pretty much every Linux system out there. Not only is it available, it’s already packaged for you and easy enough to install via the terminal. On top of that, it doesn’t take much space to install fdupes, a mere 110 kB or so.

NOTE: I do not have anything against GUI tools. While I have terminals open at all times, I do the majority of my computer interaction in a browser – specifically a GUI browser. I suggest and write about the terminal because it’s more universal. It’s also often faster, assuming you can type or at least cut and paste than it is to go mucking about with a GUI software installer.

Anyhow, it’s nice and easy to remove duplicate files with fdupes. This article is going to show you how – and the article shouldn’t even be that long! It’s pretty simple.

Remove Duplicate Files With fdupes:

I’m going to assume that you left your terminal open after installing fdupes. If you didn’t, you’ll need to open it again. The only way to run the fdupes application is in the terminal. So, even if you installed it with a GUI, it’s a CLI tool and you’ll need the command line to use it.

The basic syntax is pretty easy, and not entirely unlike rdfind. For example, if you want to find duplicate files, you simply run this command:

So, if you wanted to find duplicates in your home directory, you’d run this:

Don’t worry, you’re safe running that command. That won’t delete anything at all. That fdupes command will simply show you the duplicate files that it found.

If you want to run the fdupes command recursively, that is to check all the folders within the directory, you’d run the command like this:

If you want to calculate the size of the files that would be removed when removing the duplicates, the command is just this:

A summary is also available with this command:

You can also search multiple directories for duplicates. That’d be something like this:

Of course, you can run those commands together to get quite a bit of customization. They’re all reasonably harmless and will simply point out the duplicates as well as some meta information. You can then remove the duplicates by hand if you want.

You can also tell fdupes to remove the duplicates that it found. You’d never want to run this command without knowing what exactly is going to be removed, so don’t do that. Always check to make sure you’re not removing anything of value before automatically removing duplicates.

Fortunately, there’s a bit of a stopgap. You can run the following command and fdupes will ask for confirmation before removing the file:

If you want to go whole-hog and remove every duplicate found, and remove the files without any confirmation, you can run this command:

Of course, that’s just the basics. If you want to know more about fdupes, simply check the man page (man fdupes) for more information.

Closure:

Hmm… I think I need to clean or replace this keyboard. The colon key is sticking on me. It’s a bit of a pain in the butt.

Anyhow, if you’ve ever wanted to remove duplicate files with fdupes, you now have directions to do so. This being Linux, you have all sorts of options when it comes to removing duplicate files, though I again urge caution when doing so. If you were to run this on the root directory, you’d likely find a lot of duplicates, and removing them might break your system. So, be careful with these tools, as they’re pretty powerful.

I’m not yet out of ideas for articles but it’d be great if folks might suggest something they’d like to read about. You never know, it might be something I know about. As it is, I have to search this site before writing an article, or else I’d end up with even more duplicate articles. I don’t want that and you don’t want that. It’s not all that easy to keep up this pace, writing a new article every other day. I’ve managed so far, but I’m eventually going to miss a day or two. It’s going to happen.

As always…

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Meta: Getting Indexed In Bing

Today’s article will be largely a meta article, not about Linux but about getting indexed in Bing. For whatever reason, people sometimes have issues getting indexed in Bing and I was among those people.

To cut to the chase, I’ll simply say that this is no longer a problem for this site. We’re now indexed by Bing and the site shows up in their search results. In this article, I’ll tell you how. It may take me a few words to get to that point, but I’ll explain how we managed to get indexed in Bing.

So, first, let’s just say:

Welcome Bing Users!

As you may know, there’s a search engine called Bing. The search engine is owned by Microsoft and has been around since 2009. This isn’t Microsoft’s first search engine, but this incarnation has been around for a while now.

While we’re Linux users here, the Bing search engine has been known to give out some pretty solid results. Even still, Bing is far less popular than Google and they have about 3% of the web search market share.

Considering how their default browser (Edge, available for Linux) defaults to Bing as the search engine, it’s rather amazing how little people care about their search. I’ve used Bing extensively and found it gave me adequate results, especially when I used personalized search and searched from a logged-in account. Yes, they happily gave me Linux-related search results.

Bing also provides the search engine results for Yahoo. Yes, Yahoo still exists. Yes, people still use Yahoo. I am not one of those people, but I appreciate what Yahoo did for the web back in the day. So, while they’re surely a dinosaur, they’re still around and still useful for a few people.

The problem was, that I couldn’t get this site indexed by Bing. By that, Bing would happily crawl the site but they wouldn’t list me in their index. Even searching for the exact domain name showed zero results for this site. It’s a trivial amount of traffic, but I want everyone to be able to find this site and the information here.

Getting Indexed In Bing:

This site published the first article on March 6th, 2021. There is a new article published every other day, many of them are even acceptable! No matter what I did, Bing would not index Linux-Tips.

I did everything correctly. There’s no blackhat SEO, no overly weighted meta tags, no keyword stuffing, no hidden text, and no paying for links in link farms. All the links to this site (and there are thousands of them) are organic.

I even used IndexNow. I’ve used IndexNow from just about day one. That is a method of notifying Bing that new content has been published. They got this notification almost every other day. It does appear to lag sometimes, but that wouldn’t make any major difference. Bing knew the site existed and they refused to index the site. Not only that, Bing knew there was constant new content created and still refused to index the site.

I mostly ignored this. I signed up for Bing’s Webmaster Tools and authenticated the site on day one. They were even granted access to Google’s Search Console, meaning they had even more information about the site. Still, they refused to index the site.

This went on for years. Linux-Tips is more than two years old.

The Rest Of The Story:

Back in 2021, I found a form that I could fill out. I was frustrated and in a rush, so I simply gave them a link to the home page and said something like, “I am not indexed.” I hit the submit button and nothing happened. The site was already full of a lot of content, so I assumed they could figure that out on their own.

They said they’d respond, but they never did. Speaking of which, they still haven’t responded via email to anything I’ve sent them. But, no… No, they didn’t respond. I guess I don’t blame them. I left a vague message.

Earlier this month, I decided I was going to see what I could do to get to the bottom of this. I searched and searched the internet, trying to figure out why the site wasn’t indexed in Bing. One of the things I learned was that sometimes Bing will mistakenly block a site and that block can be removed. I’m not sure if that’s what happened in this case, but it seems the most likely.

There’s Another Form!

See, I found another form by following various Bing help files. This is the link you need to know about:

Bing Webmaster’s Support

For whatever reason, that link isn’t loading properly for me at this point. None of the Bing tools are working for me at this moment in time. I’m unsure if that’s me, if that’s them, or if they moved the form since I looked at it yesterday.

NOTE/UPDATE: The link does work if I use a VPN. Something weird is going on with their site. My presumption is that they’ll fix it. This is above my pay grade.

You’ll have to log in, of course. If you already have a Webmaster Tools account it will populate some of the fields for you. If you don’t already have said account, you’ll probably need to make one. I already had the account. I’m reasonably sure that you can’t do much of anything without an account.

If the above link doesn’t work for you, search around for the support form and that’s the form you need. You should be able to use a drop-down menu to select indexing as the problem and you should be able to specify your site.

This is what I did…

I made my comment as brief as possible while as technical as required. I explained all those things I explained earlier in this article. The information I gave them was brief, factual, and detailed. I had to try something!

The Results:

After I submitted my information on the form, making sure to select the correct options from the drop-down menu, I got an email confirming that they’d received my request. It was a canned email, an autoresponder message, and it went straight into the junk folder without me seeing it.

Amusingly, I use an outlook.com email address for this and it still went straight to spam. I never got another message. This is what that message said:

Thank you for contacting Bing Webmaster Support Team.

This email is confirmation that we have received your request for https://linux-tips.us/ and created the following Request REQ00063891 . Your ticket is being assigned to a Global Support Webmaster Engineer who will be contacting you about the next steps to resolve your issue. We will get back to you in 10 days.

Take care and stay safe!

Sincerely,
Microsoft Bing

I haven’t heard a word from them since – and that’s okay by me.

Sure enough, on the 17th of this month, I started getting traffic (in my server logs) from Bing users. When I logged into the Bing Webmaster tools, all the data was populated, showing hundreds of indexed pages. Everything was as it should be and even IndexNow appeared to be working properly.

For once, read the conclusion – as it contains more details.

Conclusion:

If you’re having trouble getting indexed in Bing, and you meet their guidelines – including having many articles full of unique and formatted content (along with the rest of the guidelines, like meta tags and the like) – then simply root around on their site until you find the hidden contact form and send them an email. 

In that email, be direct and informative – but don’t waste their time. Let them know the good things you’ve done and the good things you’re doing. Show them that you’ve followed the rules and that your site deserves to be indexed on its merits. After all, the worst thing they can do is just ignore you and not index your site. 

But, of course, make sure those things are true. Read their guidelines and become familiar with what they expect. If you’re using WordPress, it’s trivial to follow the rules. Grab one of the many SEO plugins and set it up correctly. They all have adequate help files. I am not willing to spend the money hiring an SEO expert. The site is expensive enough.

Truly, I’m not an SEO expert. Heck, I’m barely qualified to be a WordPress admin, and some folks would say I’m not even qualified to do that. At the end of the day, if you’re not being indexed by Bing, just send them an email. It worked for me and I figured I’d share the results with other people who may have their sites and indexing issues.

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Set A Timeout Value In cURL

Today we’re going to discuss a topic you probably won’t ever need but is worth knowing, we’re going to set a timeout value in cURL. We’re telling the cURL application to quit trying if it takes too long. I suppose it is something worth knowing, so we might as well learn it.

How often do you need this? Well, that depends on you and your workflow. Me? Well, let’s just say that it’s in my notes. I’m not sure that I’ve ever actually used it productively, but it is in my notes. Now? Well, now it’s in your notes! Or, at least it’s here and searchable should you ever actually need to set a timeout value in cURL.

So, what is cURL? I’ve written about it before (some links to follow) but it’s a tool to transfer a URL. That’s exactly what the man page says. Specifically, it says:

If you want to see what the HTML looks like for this site, you can run this:

(That’s not particularly helpful, but you can do it.)

I mentioned that I’d written about cURL before and it may be of some benefit to read these articles (or at least skim them) if you’re unfamiliar with the cURL application.

Let’s Have a Limited Look at Linux’s cURL Application
How To: Make ‘curl’ Ignore Certificate Errors
How To: Add A New Line With CURL

You can see a couple of useful applications of cURL:

Weather In The Terminal? We can do that!
How To: Find Your IP Address Through Your Terminal

See? So, cURL has some use – even for a regular desktop user. If any of those things take too long, you can set a timeout value for cURL, which is what this article is all about.

Set A Timeout Value In cURL:

cURL is a terminal-based tool. Sure, some GUI applications use it in the background, but it’s a terminal tool. As such, you are going to need a terminal available. You should be able to press CTRL + ALT + T to access a terminal. If not, open one from your application menu.

With your terminal open, the syntax for setting one of the timeout values in cURL is pretty basic and easy to understand. Try this:

The time_limit value is in seconds. If you wanted to load the content of this site’s home page and set a timeout value of 10 seconds, you’d run this command:

(Again, not very useful.)

But, that timeout value is just for time-to-first-byte. So, the server will need to respond within 10 seconds else the cURL process will shut down.

There’s another timeout value for cURL. You can set the overall time limit, that is the entire process (including transferring of data) must be completed within that timeframe. If it isn’t, the cURL process will shut itself down. The syntax for that time of timeout value would be like so:

So, if you wanted to make sure the entire transfer of data was done in under 60 seconds, your command would look like this:

(Again, not very useful – but it should certainly take less than 60 seconds!)

I suppose you might find some of this useful if you’re cURLing files more weighty than a web page. You can cURL actual files and write that data to your terminal’s standard output. That’s what cURL does, after all. So, you might find a use for this command.

Closure:

Well, this wasn’t a very long article. It doesn’t cover a great deal and probably won’t be useful to 99 out of 100 people. That’s okay. Not all of my articles are meant for the 99% and sometimes you just gotta write what you feel like writing. This is what I felt like writing. It probably won’t do well for search engine results and that’s okay. Someday, somebody will want this information, type it into Google, and find this site. Or another one just like it, I suppose…

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Change Your Timezone In Linux

Today’s article is an easy one about how you can change your timezone which isn’t something you’ll likely need to do all that often. It’s not all that complicated, though it can look like it is. It can be a bit tedious, but that’s just at first blush. I’ll show you how to make it a bit easier.

NOTE: This is actually a duplicate. I wrote it some time ago and realized it was a duplicate, or reasonably close to another article. I decided to save it and publish a duplicate article when an ’emergency’ came up. Well, we have an ’emergency’. I had too many adult beverages before it was time to write a new article. So, you get this one.

Now, the title is obviously not correct. You’re not going to change your timezone, you’re going to change the timezone settings your computer is using. Alas, headlines aren’t to be all that long and are allowed to make some assumptions. If you want to change your timezone, you’ll have to move.

This article is only useful to you if you use systemd.

My regular readers may have noticed a giant outage. Linux-Tips.us was unreachable for a good part of the 6th. My initial assumptions were that we’d been hacked, that is that WordPress had become compromised. That’s a reasonable assumption.

It turns out that we’d been moved to a new data center. We’re now located in New York. I use a CDN that relies on an IP address. There are a bunch of DNS records behind the scenes. Those records had to be updated. That wasn’t a major task, but troubleshooting the problem was the challenge.

Everything turned out okay and I won’t even miss a scheduled publication. Of course, I’m writing this at 04:45, but you will have your article today. So far, I haven’t missed a publication date. I’m quite amazed by this.

Speaking of today’s article, we’re going to be learning how to change your timezone. We’ll be doing this in the terminal and the tool we’ll be using is known as timedatectl. If your distro uses systemd, you have this installed. When you check the man page (with man timedatectl) you’ll see that timedatectl defines itself as:

timedatectl – Control the system time and date

As you can see, this is probably the right tool for the job – assuming the job is changing your timezone. Seeing as that’s what the title says we’ll be doing, we might as well do that.

Change Your Timezone:

As mentioned in the introduction portion of this article, you’ll be doing the work in the terminal. It’s often easy to open a terminal with a keyboard shortcut. Frequently, you can open your default terminal by just pressing CTRL + ALT + T. Otherwise, look in your application menu. That’d be a good spot to look for a way to open your terminal.

With your terminal open,  you can enter the following command to see what you have for your current timezone settings:

Before you can change your timezone you have to know what time zones are available. There are a whole lot of them, meaning you can be scrolling for quite a while. Go ahead and enter this command:

(You can press CTRL + C to get out of that.)

So, what we’re going to do is narrow down the results shown to you. You’ll need to know the region you’re in, such as Europe, Africa, America, etc for this next part and we’re going to use a pipe and grep (which we’ve used in the past). For me, I’d want to set my timezone to New York in America and my command would look like:

You’ll note that this is case-sensitive. Listed in the output would be the text I’m after, the text needed for the command that lets you change your timezone. It looks like this:

With that information available, I can now complete the command to change my timezone to that of New York. That command is easy enough. It just looks like:

Or, when using my location as the example, the command looks like this:

It’s not too daunting a task to change your timezone. If you entered the wrong timezone during installation, you can trivially change it to the correct timezone.

I should mention that your computer will use NTP to keep your system’s clock set. This is also stored in your system’s hardware. The CMOS battery keeps that and other settings stored while the power is disconnected. This battery can go bad.

If that battery does go bad, you may find yourself setting the date and time of your system every time you start your computer. NTP should then kick in and keep your system’s time updated. For reasons deeper than this blog will go, your system depends on time (specifically Unix Time) in all sorts of applications.

Closure:

Well, there was some drama. The site was down for an extended period and this article wasn’t written until the wee hours of the morning. However… However, I didn’t miss a publication date. Also, I just realized I haven’t done a meta article in a while.

With some great help from the hosting company upstream, everything was resolved and you got an article about how to change your timezone in Linux. All’s well that ends well.

While there are backups, my heart sank when I thought that the site had been hacked. Cleaning that mess would have been tedious. Fortunately, that wasn’t required. No data was lost and we can move on knowing that it’s just a footnote in the site’s history.

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Access Recovery Mode On Ubuntu

Today’s article shouldn’t be too long, but covers something important, which is how to access recovery mode on Ubuntu. If your Linux system has problems, you may need to access the recovery mode in order to fix it. This shouldn’t be a very long article, and I don’t think it’s all that complicated. Others may view this as complicated, but it seems pretty basic to me.

It isn’t all that complicated a process, but it’s something you should memorize, just in case you have no other way to access a search engine to learn this information. This is one of those skills you really ought to know by rote. It is one of those things about Linux that you probably should have memorized. 

What is Recovery Mode?

When you boot to recovery mode, you’ll be given a choice of booting to any of your currently installed kernels. The system then boots with the minimal amount of resources and mounts the root partition as read only.

You might be interested in a recent article on the subject of multiple kernels:

Show Installed Kernels In Ubuntu

Recovery mode lets you access the system to troubleshoot problems. It probably isn’t a necessary step for most issues, but it’s a necessary step that you will want to take if your desktop isn’t otherwise loading. If you’re unable to get to the GUI desktop, you should try recovery mode.

You can also try previous kernels by doing this. Recovery mode is just one option you’ll have from the GRUB menu. You get to pick your other kernels and even enter a text mode to work on your system via the CLI. For most folks that have had the kind of difficulties that stop the GUI from loading, trying recovery mode is a good step to take.

This may very well work in other operating systems. I’m being specific to Ubuntu because that’s what I have pictures for. Well, technically I’m using Lubuntu in a VM to get the pictures, but this should work for all sorts of distros. Do me a favor and leave a comment if you know this works in other distros. It’s way too late in the day and I don’t have time to check.

Access Recovery Mode On Ubuntu:

Well, you don’t need a terminal for this exercise, but you do need to reboot your computer. I suppose we can do this from a terminal, if you really want. You can just as easily use the GUI, or you can close things down gracefully and then run the following command in the terminal:

BEFORE YOU DO THAT…

We need to know if you are using EUFI/EFI boot or if you’re using the legacy BIOS mode to boot your computer. Fortunately, I’ve already covered this!

How To: Tell If You Are You Using UEFI or BIOS

Now, when your computer reboots, it’s going to perform a quick POST (Power On Self Test). That’s the point when you can decide to just let it boot or decide to enter a temporary boot menu. These shouldn’t be confused. If you’re interested in accessing your temporary boot menu, read this article:

How do I ‘Boot to USB’? (Or CD/DVD, if Such is Available)

As soon as POST ends and the OS starts to load, there’s a very brief window of time when you can press a key and access the GRUB menu. Trust me, this works. You may have to try multiple times before you get the hang of it, but this 100% works.

If you have a working keyboard, this works. You just need to time it right. You can try pressing and holding the right key as soon as you reboot, or you can try tapping the key over and over again. Should you have issues getting it right, check your settings for “Fast Boot” and temporarily disable that to give you more time.

If you’re using UEFI, the key you’ll press is the ESC key.
If you’re using BIOS, the key you’ll press is the SHIFT key.

Assuming you’ve done this right, you’re now at the GRUB menu…

The GRUB Menu:

GRUB stands for “Grand Unified Bootloader” and is what you’ll be seeing (and using) on a stock Ubuntu configuration. There are other bootloaders out there, but we’re assuming you’ve not changed from the default.

If you’ve done the above steps properly, you’ll see a GRUB screen. If you haven’t done this correctly, you’ll see the splash screen as the operating system starts to load. When you see the splash screen, you might as well just reboot so that you can try this all over again.

The first screen you’ll see will look like this:

the grub boot screen
I’ve highlighted the choice you want to select with your arrow keys and enter button.

As indicated, you want to select the advanced options for Ubuntu. Of course, that won’t be the case if you’re using a different distro. You’ll want to pick the advanced options for that distro.

Finally, you need to select the correct option to access the recovery mode on Ubuntu. It’s really obvious, but here’s a handy picture:

grub option to pick the recovery mode for ubuntu
You can see that you can also select the recovery mode from an older kernel.

Once you’ve done that, and with any luck, your computer will boot to the recovery mode, and you’ll have learned how to access recovery mode on Ubuntu. You can see the option to boot to an older kernel is there as well. That’s another useful tool to have in your Linux toolbox.

Closure:

Well, if everything goes perfect (and it seldom does), you won’t need to access recovery mode on Ubuntu. It’s just not something you’ll need to do. But, if things don’t go perfect – and we tend to tweak, modify, and break things as Linux users, you do have a recovery mode, and it’s not too hard to access it. Of course, what you do once you’ve entered recovery mode is up to you and will depend on your problems, but you now know how to access it.

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Subscribe To Our Newsletter
Get notified when new articles are published! It's free and I won't send you any spam.
.
Linux Tips
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.