Category Archives: System Administration

Amazon Load Balancer setting SSL with Certificate Chain

First of all, let’s assume you have the following files with you:

  • yourdomain.key Your domain’s private Key
  • yourdomain.crt Your domain’s public Key
  • sf_bundle.crt The Certificate Chain

Step 1 – Preparing the files

Create a PEM-encoded version of your private key

openssl rsa -in yourdomain.key -outform PEM -out yourdomain.pem

Step 2 – Setting the certificate on Amazon

On your Amazon account go to Load Balancers > Your Load Balencer > Listeners

Load Balencer Protocol: HTTPS
Load Balencer Port: 443
Instance Protocol: HTTP
Instance Port: 80
Cipher: ELBSample-OpenSSLDefaultCipherPolicy
Certificate Name:
Private Key: <past yourdomain.pem file here>
Public Key Certificate: <past yourdomain.crt file here>
Certificate Chain: <past sf_bundle.crt file here>

Note: This means every request to the Load Balancer will be made on HTTPS. The traffic from the Load Balancer to the destiny instance will be regular HTTP. This way you don’t have to setup any certificate on your instance’s Apache/Nginx web server.

Step 3 – Test

If everything went as expected you should be able to open

Now, use a SSL check tool to see if everything is OK:

You should see something like this:



Give access to /var/log/messages to non-root user

Sometimes while debugging our projects, our developers need to check /var/log/messages for some messages that are being syslogged instead of going to the application logs.

To prevent accidental changes on our servers, developers don’t have root access (that is limited to sys admins) and from time to time sys admins get requests to check /var/log/messages to verify if there are some messages there that could help identify a problem that is occurring.

Using ACLs we were able to give access to /var/log/messages to the ‘users’ group (where all developers belong).

The steps required were:

  • Ensure the filesystems are mounted with acl option.

/etc/fstab was like this before changing:

LABEL=/ / ext4 defaults,noatime 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda3 none swap sw,comment=cloudconfig 0 0

We need to add acl option after noatime to be:

LABEL=/ / ext4 defaults,noatime,acl 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda3 none swap sw,comment=cloudconfig 0 0

After doing this change we just need to remount the / filesystem

mount -o remount /
  • Give access to the file using ACL for users group
/usr/bin/setfacl -m g:users:r /var/log/messages
  • Add a postrotate script to /etc/logrotate.conf to ensure we give the same ACL access after /var/log/messages is rotated. We just need to add the following to /etc/logrotate.conf:
# Give access to /var/log/messages to 'users' group after rotate
/usr/bin/setfacl -m g:users:r /var/log/messages

Amazon EC2 – Move a EC2 non-AMI instance to other EC2 region

Amazon EC2 is spread across several regions in the globe (US/Virginia, US/Oregon, US/North California, Europe/Ireland, Asia/Singapore, Asia/Tokyo, Asia Pacific/Sidney, South America/S. Paulo). Some of these regions are very recent (Tokyo, Sidney and also S. Paulo).

To support our Brazil operations we have a set of servers setup on Amazon. Some of them were created prior to the launch of S. Paulo region on Amazon or were created from a Amazon Marketplace image that was not available on S. Paulo region.

We’re in the process of optimizing our infrastructure and it makes sense to move Brazil dedicated servers to S. Paulo region. One of them we had to move from US/Virginia to S.Paulo was a EC2 server based on Ubuntu 12.0.4 LTS (different than the standard image for Amazon EC2, Amazon Machine Image Linux).

Amazon recently released a new feature for EC2 that is key to this process. The new feature is the ability to copy EBS snapshots between regions.

So, to move a Ubuntu 12.0.4 LTS EC2 instance from US/Virginia to South America/S. Paulo you need to:

    -Stop the machine on US/Virginia – only to make sure no changes are done to the filesystem
    -Create a snapshot of the EBS volume attached to the machine on U/Virginia – Go to the Volumes page, identify the volume that is attached to the machine and do the snapshot. If there are multiple volumes attached to the EC2 instance, create a snapshot for each volume. Important: Take note of the size of EC2 instance root volume. It will be required later.
    -On the Snapshot list, for each snapshot that you have created, press “Copy” on the toolbar and select South America/S. Paulo as the destination. This operation may take some time depending on the snapshots size. 10 Gb took less than 30 minutes in our case.
    -Now selecting South America/S.Paulo in the EC2 management console, press “Launch Instance” to create a new EC2 instance and select “Classic Wizard”
    -Choose Ubuntu 12.0.4.x LTS as the type of instance you need (this was the type of our original EC2 instance, it should work with other non-AMI instances).
    -When selecting the root volume type and size, make sure you select the same size as the root volume size of the original instance. Select the key pairs and the availability zone as you wish. Take note of the availability zone where the EC2 machine was launched.
    -When the snapshot that you copied from the original region (root volume) is available on the new region, create a new volume for the snapshot. Make sure you select the same availability zone where the new EC2 instance was created.
    -Stop the new EC2 machine.
    -Go to the volumes list, check the volume that is currently attached to the new instance (this is a completely new volume without the data from the original machine that you want to migrate) and choose “Force detach”
    -Now select the volume you created from the snapshot of the EC2 original machine root volume and attach to the newly created EC2 instance. Choose the same device as the previous volume that was attached (normally /dev/sda1)
    -Create volumes (in the correct availability zone) for the remaining snapshots you want to carry forward from the original machine (in our case, we didn’t have any) and attach to the new machine,.
    -Start the machine on South America/S. Paulo.

Then, don’t forget to update your DNS records and monitoring alert applications (e.g. Nagios) to point to the new IP address of the machine.

Some tips on using .htaccess

Imagine you have a website running on and you need to redirect all the traffic to other website that is located on

The site running on has the same structure as your site:

  • -> http://
  •<7a> -> http://

    This can be done defining the following rules on the .htaccess file in the root folder of

    RewriteEngine on

    ## Redirect the requests to into
    RewriteRule ^(.*)$$1 [P]

    The [P] flag is for Proxy which means request will be proxied to the new destination but the user will keep seeing the same URL.

    Now, imagine that all the images, css and javascript on the HTML files inside are being referenced from the root folder like:

    <link rel="stylesheet" type="text/css" media="screen" href="/css/conventional.css" />
    <script type="text/javascript" src="/js/jquery.js"></script>

    This means that our redirect using .htaccess inside the /site1/ directory would not work. To solve this, we need to place a .htaccess file in the root directory of the domain (the directory that serves but what if is a website itself or has other /site*/ websites that you don’t want to redirect the requests into

    This can be done by using a RewriteCond to trigger the RewriteRule. And our RewriteCond can be based on the HTTP_REFERER of the request that will let us know if the request is coming from or not):

    RewriteEngine on

    ## Requests for css, images or js (/css or /images or /js) that are coming from
    ## will be redirected to
    RewriteCond %{HTTP_REFERER} [NC]
    RewriteRule ^(css|images|js)/(.*)$$1/$2 [P]

    If you keep struggling with .htacess files and are not understanding why rules are not being matched and why it is not working, you can enable logging by placing the following in the virtual host configuration for the web site.

    RewriteLog "/var/www/"
    RewriteLogLevel 5

    Note: This requires having MOD_REWRITE installed and enabled on Apache.

  • Prevent wrong rm -rf

    I writted the following function to prevent wrong rm -rf:

    function rm() {
    for arg in “$@”
    if [[ $arg == -*r*f* ]]
    if [ $rf_detected -eq 1 ]
    echo “-rf detected. Action stoped.”
    /bin/rm “$@”

    To use just put it at the end of .bashrc, this way if you use rm -rf you will get a message, so you have to think twice and to really remove use /bin/rm -rf …

    Be careful about the use of this function, for example if some crontabs have inside them source ~/.bashrc they will use this function too, so at them you need to use /bin/rm.

    Another way (not so good) to avoid this to add at this function a check of $PPID if it belongs to to bash you use, but this can be tricky too, see this example:

    eoliveira@local:~$ echo $PPID
    eoliveira@local:~$ cat /proc/1870/cmdline

    eoliveira@local:~$ echo $PPID
    eoliveira@local:~$ cat /proc/4119/cmdline

    root@blogs:/var/www# echo $PPID
    root@blogs:/var/www# cat /proc/27808/cmdline
    sudo-i-u root

    You can do in another way and check if the thing running is a crontab.

    Crontab, prevent duplicate process

    Sometimes you want to use crontab to run some scripts from time to time, but you are not sure how much it will take. And you want to avoid run more than one process of the script at the same time.

    This will do the trick:

    */MINUTES    *       *       *       *       if [ `ps aux | grep NAME | grep -v “grep” | wc -l` -eq 0 ]; then  /…/NAME  ; fi

    How can I check if DKIM is configured

    To check if dkim key is present in a certain domain you can use this command line tool:

    dig txt <full_key_name> @<nameserver>


    For instance:

    dig txt

    ; <<>> DiG 9.7.3 <<>> txt
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57040
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

    ;    IN    TXT

    ;; ANSWER SECTION: 3555 IN    TXT    “k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC3/ss2gtuJ8ZBXe/Byoiobcxjt ilMXW+ZJigAu0SCED7mHPBFMdiaxvElyC3BdLjzwR2PbpO+0yNdd1OJlSbVJ4wFS +DHgJLZJVMZIhTwDRbWa+AwPAKCVh+fPPRXM0ho1VO6SzlBUvJrNmxMIhXgIbumV 0CIZEHtbCYE/bhny8wIDAQAB”

    ;; Query time: 43 msec
    ;; SERVER:
    ;; WHEN: Fri Nov  4 20:47:15 2011
    ;; MSG SIZE  rcvd: 285