Tuesday, October 22, 2013
Thursday, September 19, 2013
I converted root filesystem to lvm since the root partition was huge and I needed more flexibility in managing the partitions. Besides, lvm would also enable for easy backups with lvm snapshots.
I had a sizable swap partition of 2GB which I used to transfer my root files to and rebooted to it, prior to the conversion.
Please know what you are doing prior and make sure to create backups.
- This is what the partitions looked like prior to the change:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 228G 1.4G 215G 1% / /dev/sda1 99M 17M 77M 19% /boot none 1014M 0 1014M 0% /dev/shm # fdisk -l Disk /dev/sda: 250.0 GB, 250000000000 bytes 255 heads, 63 sectors/track, 30394 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 83 Linux /dev/sda2 14 267 2040255 82 Linux swap /dev/sda3 268 30394 241995127+ 83 Linux
- With the current setup, I have a better sized partition, and can further create/resize partitions to add or remove at will:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-lvroot0 24G 1.3G 22G 6% / /dev/sda1 99M 19M 75M 21% /boot none 1014M 0 1014M 0% /dev/shm # fdisk -l Disk /dev/sda: 250.0 GB, 250000000000 bytes 255 heads, 63 sectors/track, 30394 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 83 Linux /dev/sda2 14 267 2040255 83 Linux swap /dev/sda3 268 30394 241995127+ 8e Linux LVM
- I had a sizable swap partition of 2GB, so I turned it off.
- Formated the swap partition as ext3 and copied over the root files:
- Once the files were transfered. I then edited the "/mnt/etc/fstab" file to reflect the new root.
/dev/sda2 / ext3 defaults 1 1
- Add a grub section to use "/dev/sda2" as the new root.
title CentOS (2.6.9-55.0.9.ELsmp.sda2) root (hd0,0) kernel /vmlinuz-2.6.9-55.0.9.ELsmp ro root=/dev/sda2 initrd /initrd-2.6.9-55.0.9.ELsmp.img
- After reboot the original root (/dev/sda3) is no longer mounted and is free to work with.
- Change the partition type from Linux to LVM (8e).
- Inform the Operating System for the partition table changes.
- Initialize LVM
- Make the new partition into a Physical Volume.
- Create a new volume group.
- Create a logical volume to hold the new root (/).
- Make a filesystem in the logical volume and copy the root files onto it.
- Additionally, since the new root is now lvm, the initrd image file should have support for it else kernel will not be able to find root on reboot and end up with a kernel panic.Note: At this stage, instead of creating a new image, I update the kernel so that the new image supports lvm. LVM support can also be checked via extracting the initrd image file and checking for lvm binary in the bin folder.If image does not have the lvm binary, create and move it to the /boot folder.
- Edit "/mnt/etc/fstab" on the new root so that / is mounted on /dev/vg0/lvroot0.
/dev/vg0/lvroot0 / ext3 defaults 1 1
- Add grub section to use "/dev/vg0/lvroot0" to boot to.
title CentOS (2.6.9-55.0.9.ELsmp.lvroot0) root (hd0,0) kernel /vmlinuz-2.6.9-55.0.9.ELsmp ro root=/dev/vg0/lvroot0 initrd /initrd-2.6.9-55.0.9.ELsmp.lv.img
- Reboot, and enjoy the flexibility of lvm!!
Wednesday, September 11, 2013
wget utility is the best option to download files from internet. wget can pretty much handle all complex download situations including large file downloads, recursive downloads, non-interactive downloads, multiple file downloads etc.,
In this article let us review how to use wgetfor various download scenarios using 15 awesome wget examples.
1. Download Single File with wget
The following example downloads a single file from internet and stores in the current directory.
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
While downloading it will show a progress bar with the following information:
- %age of download completion (for e.g. 31% as shown below)
- Total amount of bytes downloaded so far (for e.g. 1,213,592 bytes as shown below)
- Current download speed (for e.g. 68.2K/s as shown below)
- Remaining time to download (for e.g. eta 34 seconds as shown below)
Download in progress:
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 Saving to: `strx25-0.9.2.1.tar.bz2.1' 31% [=================> 1,213,592 68.2K/s eta 34s
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 Saving to: `strx25-0.9.2.1.tar.bz2' 100%[======================>] 3,852,374 76.8K/s in 55s 2009-09-25 11:15:30 (68.7 KB/s) - `strx25-0.9.2.1.tar.bz2' saved [3852374/3852374]
2. Download and Store With a Different File name Using wget -O
By default wget will pick the filename from the last word after last forward slash, which may not be appropriate always.
Wrong: Following example will download and store the file with name: download_script.php?src_id=7701
$ wget http://www.vim.org/scripts/download_script.php?src_id=7701
Even though the downloaded file is in zip format, it will get stored in the file as shown below.
$ ls download_script.php?src_id=7701
Correct: To correct this issue, we can specify the output file name using the -O option as:
$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701
3. Specify Download Speed / Download Rate Using wget –limit-rate
While executing the wget, by default it will try to occupy full possible bandwidth. This might not be acceptable when you are downloading huge files on production servers. So, to avoid that we can limit the download speed using the –limit-rate as shown below.
In the following example, the download speed is limited to 200k
$ wget --limit-rate=200k http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
4. Continue the Incomplete Download Using wget -c
Restart a download which got stopped in the middle using wget -c option as shown below.
$ wget -c http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
This is very helpful when you have initiated a very big file download which got interrupted in the middle. Instead of starting the whole download again, you can start the download from where it got interrupted using option -c
Note: If a download is stopped in middle, when you restart the download again without the option -c, wget will append .1 to the filename automatically as a file with the previous name already exist. If a file with .1 already exist, it will download the file with .2 at the end.
5. Download in the Background Using wget -b
For a huge download, put the download in background using wget option -b as shown below.
$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 Continuing in background, pid 1984. Output will be written to `wget-log'.
It will initiate the download and gives back the shell prompt to you. You can always check the status of the download using tail -f as shown below.
$ tail -f wget-log Saving to: `strx25-0.9.2.1.tar.bz2.4' 0K .......... .......... .......... .......... .......... 1% 65.5K 57s 50K .......... .......... .......... .......... .......... 2% 85.9K 49s 100K .......... .......... .......... .......... .......... 3% 83.3K 47s 150K .......... .......... .......... .......... .......... 5% 86.6K 45s 200K .......... .......... .......... .......... .......... 6% 33.9K 56s 250K .......... .......... .......... .......... .......... 7% 182M 46s 300K .......... .......... .......... .......... .......... 9% 57.9K 47s
Also, make sure to review our previous multitail article on how to use tail command effectively to view multiple files.
6. Mask User Agent and Display wget like Browser Using wget –user-agent
Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.
$ wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:18.104.22.168) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD
7. Test Download URL Using wget –spider
When you are going to do scheduled download, you should check whether download will happen fine or not at scheduled time. To do so, copy the line exactly from the schedule, and then add –spider option to check.
$ wget --spider DOWNLOAD-URL
If the URL given is correct, it will say
$ wget --spider download-url Spider mode enabled. Check if remote file exists. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving.
This ensures that the downloading will get success at the scheduled time. But when you had give a wrong URL, you will get the following error.
$ wget --spider download-url Spider mode enabled. Check if remote file exists. HTTP request sent, awaiting response... 404 Not Found Remote file does not exist -- broken link!!!
You can use the spider option under following scenarios:
- Check before scheduling a download.
- Monitoring whether a website is available or not at certain intervals.
- Check a list of pages from your bookmark, and find out which pages are still exists.
8. Increase Total Number of Retry Attempts Using wget –tries
If the internet connection has problem, and if the download file is large there is a chance of failures in the download. By default wget retries 20 times to make the download successful.
If needed, you can increase retry attempts using –tries option as shown below.
$ wget --tries=75 DOWNLOAD-URL
9. Download Multiple Files / URLs Using Wget -i
First, store all the download files or URLs in a text file as:
$ cat > download-file-list.txt URL1 URL2 URL3 URL4
Next, give the download-file-list.txt as argument to wget using -i option as shown below.
$ wget -i download-file-list.txt
10. Download a Full Website Using wget –mirror
Following is the command line which you want to execute when you want to download a full website and made available for local viewing.
$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL
- –mirror : turn on options suitable for mirroring.
- -p : download all files that are necessary to properly display a given HTML page.
- –convert-links : after the download, convert the links in document for local viewing.
- -P ./LOCAL-DIR : save all the files and directories to the specified directory.
11. Reject Certain File Types while Downloading Using wget –reject
You have found a website which is useful, but don’t want to download the images you can specify the following.
$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED
12. Log messages to a log file instead of stderr Using wget -o
When you wanted the log to be redirected to a log file instead of the terminal.
$ wget -o download.log DOWNLOAD-URL
13. Quit Downloading When it Exceeds Certain Size Using wget -Q
When you want to stop download when it crosses 5 MB you can use the following wget command line.
$ wget -Q5m -i FILE-WHICH-HAS-URLS
Note: This quota will not get effect when you do a download a single URL. That is irrespective of the quota size everything will get downloaded when you specify a single file. This quota is applicable only for recursive downloads.
14. Download Only Certain File Types Using wget -r -A
You can use this under following situations:
- Download all images from a website
- Download all videos from a website
- Download all PDF files from a website
$ wget -r -A.pdf http://url-to-webpage-with-pdfs/
15. FTP Download With wget
You can use wget to perform FTP download as shown below.
Anonymous FTP download using Wget
$ wget ftp-url
FTP download using wget with username and password authentication.
$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL
If you liked this article, please bookmark it with delicious or Stumble.
Thursday, September 5, 2013
Tuesday, September 3, 2013
Some times when you connect to the webhost via FileZilla you may encounter this error message.
"421 Sorry, cleartext sessions are not accepted on this server."
Just add a ftpes:// before the domain name
host : domain.com
username : xxx
pass : yyy
port : 21
host : ftpes://domain.com
username : xxx
pass : yyy
port : 21
Go To : Service Configuration
Go To : FTP Server Configuration
Check that :
TLS Encryption Support : set to “Optional”
Try that, it will work.
Saturday, August 31, 2013
About Virtual Hosts
Virtual Hosts are used to run more than one domain off of a single IP address. This is especially useful to people who need to run several sites off of one virtual private server. The sites display different information to the visitors, depending on with which the users accessed the site.There is no limit to the number of virtual hosts that can be added to a VPS.
Set UpThe steps in this tutorial require the user to have root privileges.Furthermore, if I reference the user in a step, I’ll use the name www. You can implement whatever username suits you.
Additionally, you need to have apache already installed and running on your virtual server
If this is not the case, you can download it with this command:
sudo yum install httpd
Step One— Create a New Directory
The first step in creating a virtual host is to a create a directory where we will keep the new website’s information.
This location will be your Document Root in the Apache virtual configuration file later on. By adding a -p to the line of code, the command automatically generates all the parents for the new directory.
sudo mkdir -p /var/www/example.com/public_html
You will need to designate an actual DNS approved domain, or an IP address, to test that a virtual host is working. In this tutorial we will use example.com as a placeholder for a correct domain name.
However, should you want to use an unapproved domain name to test the process you will find information on how to make it work on your local computer in Step Six.
Step Two—Grant Permissions
We need to grant ownership of the directory to the user, instead of just keeping it on the root system.
sudo chown -R www:www /var/www/example.com/public_html
Additionally, it is important to make sure that everyone will be able to read our new files.
sudo chmod 755 /var/www
Now you are all done with permissions.
Step Three— Create the Page
We need to create a new file called index.html within our configurations directory.
sudo vi /var/www/example.com/public_html/index.html
We can add some text to the file so we will have something to look at when the IP redirects to the virtual host.
Success: You Have Set Up a Virtual Host
Save and Exit
Step Four—Turn on Virtual Hosts
The next step is to enter into the apache configuration file itself.
sudo vi /etc/httpd/conf/httpd.conf
There are a few lines to look for.
Make sure that your text matches what you see below.
#Listen 22.214.171.124:80 Listen 80
Scroll down to the very bottom of the document to the section called Virtual Hosts.
NameVirtualHost *:80 # # NOTE: NameVirtualHost cannot be used without a port specifier # (e.g. :80) if mod_ssl is being used, due to the nature of the # SSL protocol. # # # VirtualHost example: # Almost any Apache directive may go into a VirtualHost container. # The first VirtualHost section is used for requests without a known # server name. #
ServerAdmin email@example.com DocumentRoot /var/www/example.com/public_html ServerName www.example.com ServerAlias example.com ErrorLog /var/www/example.com/error.log CustomLog /var/www/example.com/requests.log
The most important lines to focus on are the lines that say NameVirtualHost, Virtual Host, Document Root, and Server Name. Let’s take these one at a time.
-Uncomment (remove the number sign) NameVirtualHost without making any changes. The star means that any IP address going through port 80 will be a virtual host. As your system probably only has one IP address this is not an issue—however, if you prefer, you can replace the star with your IP address.
-You can leave the rest of the number marks in place until you reach the line
-Document Root is key! For this section, write in the extension of the new directory created in Step One. If the document root is incorrect or absent you will not be able to set up the virtual host.
-Server Name is another important piece of information, containing the virtual host’s domain name (eg. www.example.com). Make sure that you spell the domain out in full; we will put in any alternate possibilities in the next line.
-ServerAlias is a new line in the config file that is not there by default. Adding it will allow you to list a few variants of the domain name, for example without the www in the front.
The rest of the lines in this section are not required to set up a virtual host. However, it is still helpful to know what they do.
-Server admin asks for the webmaster’s email.
-The Error Logs and Custom Logs keep track of any issues with the server. The error log covers issues that arise while maintaining the server, and the custom log tracks server requests. You can set up a custom location for these processes.
-Make sure that
Step Five—Restart Apache
We’ve made a lot of the changes to the configuration. However, they will not take effect until Apache is restarted.
First stop all apache processes:
sudo apachectl -k stop
Then start up apache once again.
sudo /etc/init.d/httpd start
You may see the following error:
Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
The message is just a warning, and you will be able to access your virtual host without any further issues.
Optional Step Six—Setting Up the Local Hosts
If you have pointed your domain name to your virtual private server’s IP address you can skip this step—you do not need to set up local hosts. Your virtual hosts should work. However, if want to try out your new virtual hosts without having to connect to an actual domain name, you can set up local hosts on your computer alone.
For this step, make sure you are on the computer itself, not your droplet.
To proceed with this step you need to know your computer’s administrative password, otherwise you will be required to use an actual domain name to test the virtual hosts.
If you are on a Mac or Linux, access the root user (
su) on the computer and open up your hosts file:
If you are on a Windows Computer, you can find the directions to alter the host file on the Microsoft site
You can add the local hosts details to this file, as seen in the example below. As long as that line is there, directing your browser toward, say, example.com will give you all the virtual host details for the corresponding IP address.
# Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost #Virtual Hosts 126.96.36.1999 www.example.com
However, it may be a good idea to delete these made up addresses out of the local hosts folder when you are done to avoid any future confusion.
Step Seven—RESULTS: See Your Virtual Host in Action
Once you have finished setting up your virtual host, you can see how it looks online. Type your ip address into the browser (ie. http://188.8.131.529)
It should look somewhat similar to my handy screenshot
Adding More Virtual Hosts
To create additional virtual hosts, you can just repeat the process above, being careful to set up a new document root with the appropriate new domain name each time. Then just copy and paste the new Virtual Host information into the Apache Config file, as shown below
ServerAdmin firstname.lastname@example.org DocumentRoot /var/www/example.com/public_html ServerName www.example.com ServerAlias example.com ErrorLog /etc/var/www/example.com/error.log CustomLog /var/www/example.com/requests.log ServerAdmin email@example.com DocumentRoot /var/www/example.org/public_html ServerName www.example.org ServerAlias example.org ErrorLog /var/www/example.org/error.log CustomLog /var/www/example.orgrequests.log