It is possible to make clone of the EBS volume other than from the AWS console (and then move it between AWS servers, zones or accounts).
Via AWS console, attach EBS volume to the temporary server.
First step - fill with zeros all free space on the disk (because for removed files data still stored on the disk). This will prepare drive for better data compression.
$ mount /dev/xvdf /mnt/disk1
$ dd if=/dev/zero of=/mnt/disk1/tmp/zero bs=10M
$ rm /mnt/disk1/tmp/zero
$ umount /mnt/disk1
Copy and compress raw drive data:
$ cat /dev/xvdf | bzip2 >image.bz2
*This may take some minutes.
Then you can copy this file to another server (via scp, ftp, ...)
And finally you can restore this image on the EBS-volume of the same size as the original.
$ bzcat image.bz2 >/dev/xvdf
Amazon Server
Blog about Amazon services, servers and how have your own network in the cloud.
Sunday, January 20, 2013
Saturday, January 19, 2013
Cleanup Amazon Linux
Amazon Linux is pretty good and complete.
But sometimes it is necessary to clean and minimize the default system.
It is possible, but be especially careful when cleaning the system!
Disk usage after creation:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 932M 6.9G 12% /
and after "sudo yum update":
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 978M 6.9G 13% /
$ sudo du -sh /*
7.7M bin
25M boot
5.0M etc
28K home
58M lib
22M lib64
4.0K local
16K lost+found
4.0K media
4.0K mnt
56M opt
44K root
11M sbin
4.0K selinux
4.0K srv
12K tmp
534M usr
115M var
Remove previous kernels
Size of /boot/ directory depends on amount of previous versions of kernel in your system. Sometimes "yum update" upgrades the linux kernel and size of /boot directory incrementally increases.
It is possible to remove old kernels from the file system (System.map-, config-, initramfs-, vmlinuz- files) and from the "/boot/grub/menu.lst" file
Remove unnecessary packages
For example I don't need aws-tools and java on some of my servers (java exists by default because aws-tools depends on it).
$ yum remove java
or remove only aws packages:
$ yum remove aws-apitools-common
$ yum remove aws-amitools-ec2
$ yum remove aws-cfn-bootstrap
!Don't forget to remove "/etc/profile.d/aws-apitools-common.sh" file
Remove yum cache
Just delete "/var/cache/yum/x86_64/latest"
You can then recover this cache via "sudo yum makecache" command
Remove unnecessary locales
I don't need any national languages, and everything except the English. Therefore I purge non-en* files in these directories:
/usr/share/locale
/usr/share/doc
/usr/share/man
/usr/share/vim/vim72/doc
/usr/share/vim/vim72/lang
/usr/share/vim/vim72/tutor
After that I have about double free space:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 643M 7.2G 9% /
But sometimes it is necessary to clean and minimize the default system.
It is possible, but be especially careful when cleaning the system!
Disk usage after creation:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 932M 6.9G 12% /
and after "sudo yum update":
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 978M 6.9G 13% /
$ sudo du -sh /*
7.7M bin
25M boot
5.0M etc
28K home
58M lib
22M lib64
4.0K local
16K lost+found
4.0K media
4.0K mnt
56M opt
44K root
11M sbin
4.0K selinux
4.0K srv
12K tmp
534M usr
115M var
Remove previous kernels
Size of /boot/ directory depends on amount of previous versions of kernel in your system. Sometimes "yum update" upgrades the linux kernel and size of /boot directory incrementally increases.
It is possible to remove old kernels from the file system (System.map-, config-, initramfs-, vmlinuz- files) and from the "/boot/grub/menu.lst" file
Remove unnecessary packages
For example I don't need aws-tools and java on some of my servers (java exists by default because aws-tools depends on it).
$ yum remove java
or remove only aws packages:
$ yum remove aws-apitools-common
$ yum remove aws-amitools-ec2
$ yum remove aws-cfn-bootstrap
!Don't forget to remove "/etc/profile.d/aws-apitools-common.sh" file
Remove yum cache
Just delete "/var/cache/yum/x86_64/latest"
You can then recover this cache via "sudo yum makecache" command
Remove unnecessary locales
I don't need any national languages, and everything except the English. Therefore I purge non-en* files in these directories:
/usr/share/locale
/usr/share/doc
/usr/share/man
/usr/share/vim/vim72/doc
/usr/share/vim/vim72/lang
/usr/share/vim/vim72/tutor
After that I have about double free space:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 643M 7.2G 9% /
Thursday, January 17, 2013
Access Amazon Linux via Far Manager
The best tool I ever use in my life - Far Manager.
It looks like ancient Norton Commander or Dos Navigator, but it is modern, perfect and very stable program. And I use it everywhere, including ssh (scp) access to my amazon servers.
Just download Far Manager from its official site, then download NetBox plugin from here and unpack it to Far's plugin directory (C:\Program Files\Far Manager\Plugins\NetBox).
Then you can create ssh session and access remote server as local drive.
It looks like ancient Norton Commander or Dos Navigator, but it is modern, perfect and very stable program. And I use it everywhere, including ssh (scp) access to my amazon servers.
Just download Far Manager from its official site, then download NetBox plugin from here and unpack it to Far's plugin directory (C:\Program Files\Far Manager\Plugins\NetBox).
Then you can create ssh session and access remote server as local drive.
Sunday, January 13, 2013
Merge log files from multiple servers
If you have some similar servers (e.g. balanced EC2-instances) and need to analyze log files from them - it is possible to automatize downloading and merging all these files via shell script like this:
#!/bin/bash
LOG_DIR=log-`date +%Y.%m.%d_%H.%M.%S`
mkdir -p $LOG_DIR
scp -C ec2-user@node1.server.com:/log/file.log $LOG_DIR/file1.log
scp -C ec2-user@node2.server.com:/log/file.log $LOG_DIR/file2.log
scp -C ec2-user@node3.server.com:/log/file.log $LOG_DIR/file3.log
cat $LOG_DIR/*.log | sort >$LOG_DIR/files.log
Also it is a good idea to use ssh-agent to keep private keys.
Amazon World Wide Cloud
I love this system.
Amazon is excellent hosting for virtual servers: what's more, it really helps to build worldwide solutions.
It is very simple to raise virtual server instance in a preferred location, be it Virginia, Ireland, Tokyo or another place. From the web console it takes about 5-7 clicks.
In order to be closer to your customers (to deliver content or to execute requests faster), you can create your own distributed network based on EC2-instances, or just use Amazon CloudFront.
Amazon is excellent hosting for virtual servers: what's more, it really helps to build worldwide solutions.
It is very simple to raise virtual server instance in a preferred location, be it Virginia, Ireland, Tokyo or another place. From the web console it takes about 5-7 clicks.
In order to be closer to your customers (to deliver content or to execute requests faster), you can create your own distributed network based on EC2-instances, or just use Amazon CloudFront.
Saturday, January 12, 2013
Recover broken Amazon EC2 instance
Sometimes shit happens, accidentally.
Especially when you lost ssh access to your virtual server (e.g. due to errors in ~/.ssh/authorized_keys, /etc/sudoers, /etc/group, /etc/shadow or another important config file).
When you catch errors like:
No supported authentication methods available
or
sudo
sudo: >>> /etc/sudoers: syntax error near line 1 <<<
sudo: parse error in /etc/sudoers near line 1
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin
Don't worry, inside the Amazon cloud you can repair almost all.
Follow these simple steps to solve the problem:
Especially when you lost ssh access to your virtual server (e.g. due to errors in ~/.ssh/authorized_keys, /etc/sudoers, /etc/group, /etc/shadow or another important config file).
When you catch errors like:
No supported authentication methods available
or
sudo
sudo: >>> /etc/sudoers: syntax error near line 1 <<<
sudo: parse error in /etc/sudoers near line 1
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin
Don't worry, inside the Amazon cloud you can repair almost all.
Follow these simple steps to solve the problem:
- Via AWS console:
- Launch new temporary EC2 micro-instance in the same availability zone.
- Stop broken instance.
- Detach root EBS volume from the broken instance.
- Attach this EBS to the temporary instance (as /dev/sdf).
- Via SSH console from the temporary instance:
- Mount this EBS:
sudo mount /dev/xvdf /mnt - Fix all problems on mounted file system.
- Unmount this EBS:
sudo umount /mnt - Via AWS console:
- Detach this EBS from the temporary instance.
- Attach this EBS to the broken instance (as /dev/sda1).
- Start broken instance.
- Check that broken instance now is healthy (available via ssh and everything is functioning normally).
- Then you can stop temporary instance (or terminate it).
Disk drive for the Amazon virtual server
Amazon virtual server uses ELB (Elastic Block Store) storage as disk drive.
It's like a HDD (Hard Disk Drive) within the physical server.
You can create ELB of any size (1GB..1TB) from the Amazon Web Console.
Then you can attach ELB to EC2 (via Web Console),
format:
sudo mkfs.ext4 /dev/xvdf
mount:
sudo mount /dev/xvdf /mnt/disk1
and use:
sudo mkdir --mode=777 /mnt/disk1/tmp
I guess that internally ELB is just a dynamically expanding file, movable between servers.
And I've expanded ELB to the entire allocated size:
dd if=/dev/zero of=/mnt/disk1/tmp/zero bs=1M
921+0 records in
920+0 records out
965447680 bytes (965 MB) copied, 29.1309 s, 33.1 MB/s
Then ELB is very fast (speed depends on the file size and state of system caches):
(1.0 MB) copied, 0.0018899 s, 555 MB/s
(10 MB) copied, 0.0122498 s, 856 MB/s
(105 MB) copied, 0.442308 s, 237 MB/s
(262 MB) copied, 4.84185 s, 54.1 MB/s
You can create ELB of any size (1GB..1TB) from the Amazon Web Console.
Then you can attach ELB to EC2 (via Web Console),
format:
sudo mkfs.ext4 /dev/xvdf
mount:
sudo mount /dev/xvdf /mnt/disk1
and use:
sudo mkdir --mode=777 /mnt/disk1/tmp
I guess that internally ELB is just a dynamically expanding file, movable between servers.
And I've expanded ELB to the entire allocated size:
dd if=/dev/zero of=/mnt/disk1/tmp/zero bs=1M
921+0 records in
920+0 records out
965447680 bytes (965 MB) copied, 29.1309 s, 33.1 MB/s
Then ELB is very fast (speed depends on the file size and state of system caches):
(1.0 MB) copied, 0.0018899 s, 555 MB/s
(10 MB) copied, 0.0122498 s, 856 MB/s
(105 MB) copied, 0.442308 s, 237 MB/s
(262 MB) copied, 4.84185 s, 54.1 MB/s
Init
I move my articles related to Amazon AWS from my main blog to this place.
"I'm going to write series of articles about the Amazon Web Services: cloud services, servers, storage.
For me it was quite simple and very usable service. And of course I will use it in my future projects."
Subscribe to:
Posts (Atom)