Category: Command Line

Monitor Progress With RSYNC

When you’re moving a lot of files around you might want to know the progress, so here’s how you monitor progress with rsync. This will likely be an intentionally short article, an easy article, as it’s another weekend article – this time about rsync and how you can monitor the progress.

The rsync application is a tool for transferring and synchronizing files between two places. It’s a very handy application and rsync has a whole lot of features. It’s possible to write a half-dozen rsync articles and not even scratch the surface. We’ve had rsync available since the mid-90s and you can visit the rsync project page.

You’ll almost certainly not need to install anything to use rsync. It’s pretty much available with every distro though I’ll mention ahead of time that this is a terminal-based application. By the way, if you’re not happy with your terminal, you can change to a new default terminal easily enough.

There are frontends for rsync, such as luckyBackup, but we’ll just be using a plain old terminal for this article. If you play with luckyBackup, you’ll discover that you can use it to spit out the actual rsync command you’ve configured and use it in the terminal without any additional overhead.

RSYNC:

If it’s not obvious, the tool we’ll be using is rsync. Specifically, we’re going to monitor the progress as rsync synchronizes your data from one place to another. It’s going to be a very easy article, I promise. Well, I think it’s easy. I suppose I could be wrong and you find it more difficult.

You shouldn’t need to install anything to follow along. You can first verify that you have rsync installed and available. Use this command:

If you don’t have rsync available immediately, I’m sure you can install it from one of your default repositories. If it’s not installed by default, let me know. I’m curious…

Anyhow, you can check the man page (with man rsync) to learn more about this application. It’s a rather long and complicated man page, but rsync is described as:

rsync – a fast, versatile, remote (and local) file-copying tool

And that’s what we want to do. We want to monitor the progress of file transfers. So, that’s what we’re going to do.

Except, I’m also going to show you how to make a simple backup of your data as a bit of bonus information. Let’s get on with it!

Monitor Progress With RSYNC:

Now, I did tell you that you’ll need an open terminal for this. That’s easily done. More often than not, you can open your default terminal by pressing CTRL + ALT + T. Otherwise, you can find it in your application menu.

With your terminal now open, your standard command would be something like this following rsync command:

To monitor progress with rsync, you’d add the --progress flag, like so:

That’s all well and good, but let’s pretend you want to make a backup of your home directory with rsync and that you want to monitor the progress. Well, you can happily use rsync with directories. This is going to be pretty easy!

The -r flag means ‘recursive’. That means it will include the files in the directory you specified, find the additional directories within that directory, and include those files. It will keep digging until there are no more directories unless you decide to specify a maximum depth.

Let’s further assume you have a USB device plugged in and you want to store your home directory on that USB. Let’s also assume that the USB’s called ‘backup’. The command for that would just be something very close to this command:

That might look something like this:

showing the progress of rsync
As you see, it’s easy to monitor the progress of rsync. Yes, that’s Windows ME. Don’t judge me!

That command will move all the files from your home directory (as we know that ~/ is a short way to say your /home/user directory) to your USB device. You’ll have then backed up your personal data. The second time you run it, it will sync those files – only replacing those files that have changed since the last time you ran the command.

You can easily use that command (modified for your needs, of course) to make a backup of your home directory. This is the most important data on your system. The rest is just system files and those are easily restored with a fresh installation.

While it would be possible to use rsync to backup the entire drive, it’d try to do stupid things like backup the /media directory. You’d need an advanced command for that and that’s beyond the scope of this article.

As it stands, the backup bit is extra. This article is just about how you can monitor the progress with rsync. It’s a pretty trivial thing, but something you can learn about quickly.

Closure:

Yeah, it’s a pretty short and easy article. At least I hope it’s easy. I did make a promise, after all! This one should be easy enough for anyone because it’s not all that hard to learn how to monitor progress with rsync. Also, with a reasonably large file, you can do things like check transfer rates, be it over a network or from one drive to another. There’s all sorts of fun!

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Monitor Your Linux System With NMON

There are times when you want to see what’s going on with your devices and you can monitor your Linux system with nmon. If you’ve never heard of nmon, that’s okay. You’ll find that nmon is a handy application that lets you monitor all sorts of aspects of your system. You’ll also find that nmon is an application that’s used in the terminal, so be prepared for that.

This nmon application is available for most Linux systems and I’ll let you know how to install nmon in this article. You’ll find that nmon stands for ‘Nigel’s Monitor’ and has been around since the early IBM AIX days. It has since been made available for Linux.

You can read about nmon on Wikipedia.
You can also visit the nmon project page.

If you check the man page, you’ll see nmon described as:

nmon – systems administrator, tuner, benchmark tool.

We won’t be doing much of that. We’ll just be using nmon to monitor the system. You can then take that information and use it to administrate, tune, and benchmark the system. We’ll be using nmon to monitor CPU, RAM, your network usage, and things of that nature.

You can also learn more about nmon from the man page because I’ll only be covering how to monitor your Linux system with nmon. There’s almost always so much more! It’s almost a shame that I try to limit many of my articles.

Installing nmon:

You can use your usual GUI tools to find and install nmon. It’ll be available for most distros. It’s also possible to install nmon in the terminal. We’ll cover that, so just press CTRL + ALT + T to open up your terminal.

With your terminal now open, we’re ready to install nmon.

Debian/Ubuntu/etc:

Arch/Mandrake:

RHEL/CentOS:

SUSE/OpenSUSE:

Gentoo:

I think that covers the vast majority of distros out there. If you don’t see your distro, or if I got one wrong, please let me know in the comments. I don’t always get this section right. I wasn’t even planning on writing this article today. But… Here we are!

Monitor Your Linux System With NMON:

You should have a terminal open already, from when you were installing nmon. This is going to sound strange, but you’re new to nmon – or you probably wouldn’t be reading this article. Because you’re new, open up a second terminal.

In the first terminal, enter this:

In the second terminal, enter this:

Now, do you see why I had you open two terminals?

In the first terminal, look at your settings and options. In your second terminal, apply them. See, once you apply them you can no longer trivially refer to the material in the first terminal.

You can apply the monitoring options in any order you’d like and in any combination that you like. The nmon application makes it easy. You press c to monitor the CPU, m to monitor memory, n to monitor the network, and things like that.

So, pick what you want to monitor from the first terminal and press the appropriate key in the second terminal. It might look a bit like this:

nmon monitoring options
There are quite a few options to pick from and it’s straightforward.

Then, your monitoring terminal could look a bit like this:

monitoring system processes with nmon
See? That’s monitoring CPU, memory, the network, and top processes – all in one screen.

Of course, once you know how nmon works and what you’d like to monitor in your Linux system, you won’t need to have the first terminal open. There are a lot of options, but you’ll find they’re easy to remember. You can also use ? instead of a separate terminal window. I find the two-terminal method to be easier for me because I don’t run nmon often enough to recollect the myriad options.

I suppose you want to know how to get out of nmon without closing the terminal window and leaving nmon running in the background. Well, that’s easy. Just press CTRL + C to quit nmon and return to your regular terminal prompt.

Closure: 

I didn’t plan on writing this article today, but I’d cracked open nmon for one reason or another and decided that I’d write about nmon when I was thinking about it. It’s another one of those articles that scratches my own itch, but it’s also something worth sharing.

It’s a pretty easy task to monitor your Linux system with nmon. I figure anyone can do it and I tried to make it easier for new nmon users by suggesting they use two distinct terminal windows. It seems like a reasonable way to get started.

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Repair Your Filesystem With FSCK

Over the years, we’ve covered the fsck command fairly extensively but I’ve never really written an article about how you repair your filesystem with fsck. This may very well be the last time we have an fsck article on this site. It seems like I’ve covered everything you need to know.

This article is about using fsck when you’re booted into your operating system. You can’t run fsck against a mounted drive. So, if you want to check and repair your root filesystem, the following article may be of assistance:

Repair Your Linux Filesystem With a Live USB or DVD

That’s a fairly popular article, as it ranks well in search engines. I’ve also recently authored this article:

Enable fsck On System Start

This article will be fairly simple and reasonably short. We’ll see if that turns out to be true, ’cause I wrote this before I finished the article! We shall see!

The Tools:

You’ll only need a couple of tools to repair your filesystem with fsck. First and foremost, you’ll need a terminal. I’ll assume you have a terminal. All the tools in this article will require an open terminal. You can usually press CTRL + ALT + T to open your default terminal.

You will also need…

lsblk:

The lsblk application is the first tool we’ll be using. Using lsblk is how we’ll identify your partitions. You almost certainly won’t need to install anything as this is one of those tools installed by default. You can verify that lsblk is installed by running:

If you check the man page, you’ll see that this is indeed the tool for the job.

lsblk – list block devices

See? We want to list block devices – that will show us the partitions and various filesystems.

umount:

The next tool we’ll use is the umount command. You can’t run fsck against a mounted filesystem, so we’ll need first to unmount the devices before we can repair them with fsck. You can verify that umount is installed with this command:

Again, we’ll check the man page to ensure that this is the correct tool for the job.

umount – unmount filesystems

See? I wouldn’t steer you wrong. This is the tool we need to unmount filesystems before we use fsck, the next tool, to repair filesystem errors.

fsck:

This is the tool we’ll be using. We’re going to repair your filesystem with fsck, assuming your filesystem needs to be repaired. This must be a Linux filesystem, meaning Ext2, Ext3, Ext4, etc… You’ll need other tools for filesystems otherwise formatted. Once again, you can verify that fsck is installed with the following command:

Again, if we check the man page, you’ll see that this is the right tool for the job.

fsck – check and repair a Linux filesystem

I’d say that these are the best tools we can use for today’s task.

Repair Your Filesystem With FSCK:

I gave instructions above that told you how you can usually open your terminal. That keyboard combination is not true with every distro and I don’t know why. It seems to me that it should be a standard. If you’re using such a distro, you can open your terminal from the application menu.

With your terminal open, we first need to identify your filesystems. You can do that with the following command:

You’ll get an output similar to this one:

You can also run:

That has the added benefit of showing limited information and which filesystems are identified as Linux filesystems. I use lsblk above because I use that often and want a consistent set of directions across the site. However, an example output from the above command would contain information like this:

You can run that command if you prefer. You may have to scroll up to see all of the information. In the output I shared above, you’ll notice that it didn’t include any information about /dev/sda* filesystems. As those are mounted and need to be mounted (in my case) it doesn’t matter.

At this point, you need to identify the filesystem you want to check. If you use the first command, you’ll need to remember to add /dev to the front. So, it’s not sdb2, it’s /dev/sdb2. If you use the second command, it gives you that information.

Now, we’re just going to automatically check for problems and repair them with fsck, but first you need to unmount the filesystem. To do that, you just run the following command:

Then, you can just run fsck against the filesystem’s device ID, like so:

For example, I could run:

With the -p flag enabled, it will automatically check the filesystem for errors and repair them – unless they’re really serious. If they’re serious errors, it will ask you what to do.

NOTES:

Unfortunately, I don’t have any filesystems that need to be repaired. Linux is rather robust and our filesystems are usually fairly healthy. If you suffer from random reboots without properly shutting down, you might have a different experience.

When you’re done with this, you can remount the filesystems you checked for errors. Frankly, that’s a whole other article. If you open your file manager, the GUI one, you can often mount filesystems right from there. You can also unmount them, but we did that right there in the terminal.

Again, you can’t unmount your root partition or any other important partitions. If you use separate partitions like /home or /dev, you’ll need to use a live USB to repair those mounted filesystems. This command will only work with Linux filesystems. It is not going to work with FAT, EXFAT, NTFS, etc., it will only work with Linux filesystems.

That might be a good reason to run the 2nd command ( sudo fsck -l) where you’ll get an output like this:

Look at /dev/sda1, where it properly identifies the type as an “EFI System”. That means that it’s not a supported filesystem and fsck will do you no good when you target it. I figure you’re smart enough to know the differences and what you have used, so lsblk is easier – and more consistent as we use that command frequently here on Linux-Tips.us.

Closure:

Well, it turns out that I was mistaken. I thought this was going to be a quick and easy article. I think it’s easy enough, but it wasn’t very quick. It has been a mixed bag lately as I play around with the formatting of various articles. This often makes them longer, but I’m okay with that. Plus, I type quickly!

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Unzip .gz Files With gunzip

You can just as easily do this in a GUI, but we’ll be using the Linux terminal to unzip .gz files with gunzip. Why? Because we can! You never know when you’ll be limited to a terminal and need to extract the files found in a .gz file.

If you do some searching around the web, you’ll see that .gz files are made with gzip. If you do some more digging, you’ll learn that gzip is both a file type and the name of the application. The .gz files are referred to as gzip files.

While you may encounter just plain .gz files, you’ll often find them as ‘tar.gz’ files, meaning they’ve been prepared for tape archives. You can read about that here:

Let’s Decompress A File (tar.gz) In The Terminal

You won’t need to install anything for this article. The tools will be installed on any major distro – and even on most minor distros. Gzip has been around for more than 30 years, at the time of this writing, and is widely used – even though there are many other compression tools out there. The .gz format is not patent encumbered, nor is it proprietary.

We’ll be using gunzip to unzip .gz files. If you check the man page, you’ll see:

gzip, gunzip, zcat – compress or expand files

As you can see, that means it is the right tool for the job. You can just as easily use ‘file-roller’ in your GUI to unzip .gz files. You have choices!

It’s currently a Sunday evening and my last few articles have been quite long. I think we’ll keep this one short. I might as well…

Unzip .gz Files With gunzip:

As I mentioned earlier, you can do this with a GUI application. We’ll be unzipping .gz files in the terminal. Just press CTRL + ALT + T to get your terminal open and we can begin.

The first thing you need is a .gz file.

Download example.gz here: Link

With that freshly downloaded, you can use cd to get to your ~/Downloads directory (or whichever directory you used):

Now, you can simply extract the contents, like so:

If you want to extract multiple .gz files at one time, try this:

If you want to extract all the .gz files in a folder, try this command:

There you have it. That’s all you need to know if you want to unzip .gz files with gunzip. It’s not even a little complicated. Anyone can figure this one out.

Closure:

I wasn’t kidding when I said I’d keep this one short. There’s no reason to make it all that long and it’s a weekend. It’s also a holiday weekend and you got a lovely article on Saturday. This one is just a simple task, for those who might want to unzip ..gz files with gunzip – in the Linux terminal. It’s pretty easy!

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Enable fsck On System Start

If you ever want to check your filesystem’s health in Linux, you might have wanted to enable fsck on system start. This is some maintenance work that we should probably consider performing regularly. Verifying the health of our data is not a bad thing and it doesn’t even take that long.

When you run fsck, you’re not checking the hardware’s integrity. That’s something done with S.M.A.R.T. and is something else you might want to check regularly. This is checking the integrity of our data, the health of your partitions, and that sort of thing.

I’m inept and didn’t remember to write this ahead of time, so this was written after eating way too much and a few glasses of wine. You have been warned.

Anyhow, I’ve written about fsck before:

Repair Your Linux Filesystem With a Live USB or DVD
How To: Check A Disk For Errors
A Couple More Ways To Find Your File System Type

If you get a minute, those articles might interest you.

What Is fsck:

You won’t need to install anything in this article. If you hunt around the ‘net, you’ll find that fsck somehow stands for ‘file system consistency check’. Please don’t blame me, I didn’t name anything. You’ll find that fsck is installed by default on most anything UNIX-like, including the BSD family and MacOS.

You can run fsck manually against unmounted partitions. In this case, we’ll make that easier by running fsck during the boot process. We’ll be telling fsck to run during the startup process. This means it’ll run automatically when you reboot.

If you check the man page for fsck, you’ll see it’s described like so:

fsck – check and repair a Linux filesystem

So, this is the correct tool for the job. If you look further, you’ll also come across this bit of information:

fsck is used to check and optionally repair one or more Linux filesystems.

You can verify that fsck is installed with this command:

The output from that command will tell you the version and the bit about ‘util-linux’ will let you know why I can assume that fsck is installed by default.

We’ll also be using the following tool…

What is tune2fs:

The tune2fs application is the tool we’ll be using to schedule the fsck command. The syntax is quite simple and you’ll be a professional in no time. If you were to check the man page, you’d see it described like this:

tune2fs – adjust tunable file system parameters on ext2/ext3/ext4 filesystems

NOTE: It’s important to realize that this is only going to work if you’re using one of those three formats. If you’re not using an ext* formatted partition, this isn’t going to work for you. There are other tools. This one is not for you.

The tune2fs application is almost certainly installed by default. If you check the man page it says it’s an invalid option – but it does tell you the version. Which means you can ensure that it is installed. However, you can just as easily verify that tune2fs is installed with this command:

While that won’t give you a version number, you don’t need to know the version. This is one of those applications that are robustly finished, for lack of a better term. It’ll work with your system.

lsblk:

We will also be using lsblk. We’ve used this command many times. Several other commands will get you the same information, but lsblk is a nice and easy example. If you check the man page, you’ll see that lsblk is described as:

lsblk – list block devices

Which, as you can see, is exactly the right tool for the job. Block devices are, in this case, drives and partitions. We’re interested in the partitions for this exercise. It’s a basic Linux utility and will be installed by default. You can verify that lsblk is installed by default with this command:

If any of these tools aren’t installed by default, please leave a comment. I don’t think I can recall a server or desktop distro that doesn’t have these basic tools installed by default. You shouldn’t need any additional tools for this exercise.

Enable fsck On System Start:

If it’s not obvious, fsck is a tool usually called from the terminal. I’m sure a GUI frontend exists somewhere, but that’s not covered in this article. So, press CTRL + ALT + T and let’s get started learning how to enable fsck when you start your system up.

First, let me cover another method. If you want to run fsck on the next boot and against your root partition, you can skip all of this and just run this command:

I think I’d be doing you a disservice if I didn’t mention that. The next time you reboot the device, it will run fsck on the root partition. It will clean itself up afterward and only run it that once, assuming all goes to plan.

What we’ll be doing is using tune2fs to schedule running fsck. We’ll specify the disk name and the frequency at which fsck is run. You’ll have options with the tune2fs method. I’ll explain them as well as I can.

Enable fsck With tune2fs:

With that open terminal, we need first to find the name of the partition you want to check. For that, we’ll use the lsblk command. The syntax is easy.

An example output from that command would be:

We are interested in the name section, which is sda* and we’ll be adding /dev to that name.

In this instance, we can see that there’s an efi partition and that efi isn’t an ext* formatted partition. So, that leaves us with sda2. As you learned earlier in this article, we could just run fsck against the root (the / partition) easily enough. So, in this article, we’re going to refer to sdb1 and you’ll apply it to your personal needs.

The syntax to enable fsck on system start is quite simple. I’ll give you an example:

So, let’s say we want to run fsck on sdb1 the next time we reboot. The command would look like this:

If I want to run this command every tenth time the system reboots, I can do that easily. I’d run this modified command:

You can schedule fsck easily this way, and apply it to different partitions without much additional effort. If you want to set it to check every 25 times, you can do that. You can just set it and forget it and then you’d have a slower boot time every 25 system starts because the system would be checking the drive(s) for errors.

When you’re done, you can disable this command easily enough. Just change the frequency to zero. If I were to disable fsck from running at system startup, I’d run this command:

That’s all there is to it. I’m sure you can manage. Hopefully, those directions are clear enough. If not, you can always ask questions!

Closure:

Well, I didn’t expect I’d write a long article tonight. I expected it to be a short and easy article. You can see how that turned out. Though, if you ever need to enable fsck on system start, you now know how you can go about doing so.

Best of all, assuming I’ve done my job well enough, you’ll be able to do so easily and without mistakes. However, mistakes with just this won’t be too worrisome. I suppose the worst case is that you check your filesystem’s health more often than you want to. Even then, that’s easily corrected. 

You might want to write down how to cancel it or remember this page so that you can return to remind you how to disable it. It’s not complicated and might even be stored in your bash history file so that you can reference it easily.

Either way, I hope you all had a happy Thanksgiving – if you celebrate. If you do not celebrate, I hope you had a happy Thursday! As for me and mine, it was a pretty good day. I have no complaints, other than there wasn’t a giant snowstorm to make things interesting.

Thanks for reading! If you want to help, or if the site has helped you, you can donate, register to help, write an article, or buy inexpensive hosting to start your site. If you scroll down, you can sign up for the newsletter, vote for the article, and comment.

Linux Tips
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.