Copy EC2 EBS

It is possible to make clone of the EBS volume other than from the AWS console (and then move it between AWS servers, zones or accounts).

Via AWS console, attach EBS volume to the temporary server.

First step - fill  with zeros all free space on the disk (because for removed files data still stored on the disk). This will prepare drive for better data compression.

$ mount /dev/xvdf /mnt/disk1
$ dd if=/dev/zero of=/mnt/disk1/tmp/zero bs=10M
$ rm /mnt/disk1/tmp/zero
$ umount /mnt/disk1

Copy and compress raw drive data:

$ cat /dev/xvdf | bzip2 >image.bz2
*This may take some minutes.

Then you can copy this file to another server (via scp, ftp, ...)
And finally you can restore this image on the EBS-volume of the same size as the original.

$ bzcat image.bz2 >/dev/xvdf

Cleanup Amazon Linux

Amazon Linux is pretty good and complete.
But sometimes it is necessary to clean and minimize the default system.
It is possible, but be especially careful when cleaning the system!

Disk usage after creation:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  932M  6.9G  12% /

and after "sudo yum update":

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  978M  6.9G  13% /

$ sudo du -sh /*
7.7M    bin
25M     boot
5.0M    etc
28K     home
58M     lib
22M     lib64
4.0K    local
16K     lost+found
4.0K    media
4.0K    mnt
56M     opt
44K     root
11M     sbin
4.0K    selinux
4.0K    srv
12K     tmp
534M    usr
115M    var


Remove previous kernels
Size of /boot/ directory depends on amount of previous versions of kernel in your system. Sometimes "yum update" upgrades the linux kernel and size of /boot directory incrementally increases.
It is possible to remove old kernels from the file system (System.map-, config-, initramfs-, vmlinuz- files) and from the "/boot/grub/menu.lst" file

Remove unnecessary packages
For  example I don't need aws-tools and java on some of my servers (java exists by default because aws-tools depends on it).

$ yum remove java


or remove only aws packages:

$ yum remove aws-apitools-common
$ yum remove aws-amitools-ec2
$ yum remove aws-cfn-bootstrap


!Don't forget to remove "/etc/profile.d/aws-apitools-common.sh" file

Remove yum cache

Just delete "/var/cache/yum/x86_64/latest"
You can then recover this cache via "sudo yum makecache" command



Remove unnecessary locales
I don't need any national languages​​, and everything except the English. Therefore I purge non-en* files in these directories:

/usr/share/locale
/usr/share/doc
/usr/share/man

/usr/share/vim/vim72/doc
/usr/share/vim/vim72/lang
/usr/share/vim/vim72/tutor


After that I have about double free space:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  643M  7.2G   9% /

Access Amazon Linux via Far Manager

The best tool I ever use in my life - Far Manager.
It looks like ancient Norton Commander or Dos Navigator, but it is modern, perfect and very stable program. And I use it everywhere, including ssh (scp) access to my amazon servers.

Just download Far Manager from its official site, then download NetBox plugin from here and  unpack it to Far's plugin directory (C:\Program Files\Far Manager\Plugins\NetBox).

Then you can create ssh session and access remote server as local drive.

Merge log files from multiple servers

If you have some similar servers (e.g. balanced EC2-instances) and need to analyze log files from them - it is possible to automatize downloading and merging all these files via shell script like this:

#!/bin/bash

LOG_DIR=log-`date +%Y.%m.%d_%H.%M.%S`

mkdir -p $LOG_DIR

scp -C ec2-user@node1.server.com:/log/file.log $LOG_DIR/file1.log
scp -C ec2-user@node2.server.com:/log/file.log $LOG_DIR/file2.log
scp -C ec2-user@node3.server.com:/log/file.log $LOG_DIR/file3.log

cat $LOG_DIR/*.log | sort >$LOG_DIR/files.log

Also it is a good idea to use ssh-agent to keep private keys.

Amazon World Wide Cloud

I love this system.

Amazon is excellent hosting for virtual servers: what's more, it really helps to build worldwide solutions.
It is very simple to raise virtual server instance in a preferred location, be it Virginia, Ireland, Tokyo or another place. From the web console it takes about 5-7 clicks.

In order to be closer to your customers (to deliver content or to execute requests faster), you can create your own distributed network based on EC2-instances, or just use Amazon CloudFront.

Recover broken Amazon EC2 instance

Sometimes shit happens, accidentally.
Especially  when you lost ssh access to your virtual server (e.g. due to errors in  ~/.ssh/authorized_keys, /etc/sudoers, /etc/group, /etc/shadow or another important config file).

When you catch errors like:

No supported authentication methods available

or

sudo
sudo: >>> /etc/sudoers: syntax error near line 1 <<<
sudo: parse error in /etc/sudoers near line 1
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin


Don't worry, inside the Amazon cloud you can repair almost all.

Follow these simple steps to solve the problem:

1. Via AWS console:
1. Launch new temporary EC2 micro-instance in the same availability zone.
2. Stop broken instance.
3. Detach root EBS volume from the broken instance.
4. Attach this EBS to the temporary instance (as /dev/sdf).
2. Via SSH console from the temporary instance:
1. Mount this EBS:
   sudo mount /dev/xvdf /mnt
2. Fix all problems on mounted file system.
3. Unmount this EBS:
   sudo umount /mnt
3. Via AWS console:
1. Detach this EBS from the temporary instance.
2. Attach this EBS to the broken instance (as /dev/sda1).
3. Start broken instance.
4. Check that broken instance now is healthy (available via ssh and everything is functioning normally).
5. Then you can stop temporary instance (or terminate it).

I hope this article will make someone little bit happy)

Disk drive for the Amazon virtual server

Amazon virtual server uses ELB (Elastic Block Store) storage as disk drive.

It's like a HDD (Hard Disk Drive) within the physical server.

You can create ELB of any size (1GB..1TB) from the Amazon Web Console.
Then you can attach ELB to EC2 (via Web Console),
format:
sudo mkfs.ext4 /dev/xvdf

mount:
sudo mount /dev/xvdf /mnt/disk1

and use:
sudo mkdir --mode=777 /mnt/disk1/tmp

I guess that internally ELB is just a dynamically expanding file, movable between servers.
And I've expanded ELB to the entire allocated size:

dd if=/dev/zero of=/mnt/disk1/tmp/zero bs=1M

921+0 records in
920+0 records out
965447680 bytes (965 MB) copied, 29.1309 s, 33.1 MB/s

Then ELB is very fast (speed depends on the file size and state of system caches):

(1.0 MB) copied, 0.0018899 s, 555 MB/s
(10 MB) copied, 0.0122498 s, 856 MB/s
(105 MB) copied, 0.442308 s, 237 MB/s
(262 MB) copied, 4.84185 s, 54.1 MB/s

Init

I move my articles related to Amazon AWS from my main blog to this place.

"I'm going to write series of articles about the Amazon Web Services: cloud services, servers, storage.
For me it was quite simple and very usable service. And of course I will use it in my future projects."