Move to AWS

[Mobile version(QRCode)] Total views:65,839 / Applauds for blog:1
Welcome to my page. I am an adminisrator of this site.
If you are this db's user, please contact with me by private mail. If not, please contact with me by email or twitter or facebook
Access record[Graph / PV Info.(Past 1 day / Past 1 week) / Access from outside (Yesterday / Past 1 week) / Vistors's list]
ProfilePmail(Mail)
Inbox   /Send   /Sent
Reviews(List   /Limit)
Poll   /Agree:Got   /Sent
Fan
Works/Music
Blog
[Write]
Links
My Play List
<=Newer article MRT
=>Older article Only the person who wrote can read

1.
2017/01/24 "AWS > Move to AWS"
[Show only this article / Modify / Delete / Send trackback / Add to the shared category]

1. Background
  1. Prerequisite
    1. Your PC
    2. Create AWS account by sign up
    3. Create user account using IAM
  2. Decide the region
    1. Check mandatory service for you
    2. Compare price
    3. Decide default region
  3. Cost estimation before you start using AWS
  4. Set up AWS CLI env on your PC
  5. Create network (VPC)
  6. Move the authority to Rout53
  7. SSL Certificate
  8. ec2 instance
  9. EFS
  10. Request to unlock the limitation of SES
  11. Change mail to use SES
  12. Change the code which is using REMOTE_ADDR
  13. Timezone of Amazon Linux
  14. Stop unnecessary process

1. Background

I decided the service of sakuhindb move to AWS.
I will show the procedure for you to reference.
  1. Prerequisite


    1. Your PC

Mac or Linux environment to use linux command
    2. Create AWS account by sign up

https://aws.amazon.com/free/
    3. Create user account using IAM

After you have succeeded in signing in web console, create user group and user.
  2. Decide the region

https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
    1. Check mandatory service for you

North virginia has all service and price is low.
So check the region's status which you are considering to serve the service.
In my case, it is tokyo and following services are not available now.

ServiceNecessary
Amazon Cloud DirectoryN
Amazon Elastic File System (EFS)Y
Amazon Kinesis AnalyticsN
Amazon Kinesis FirehoseN
Amazon LightsailN
Amazon Machine LearningN
Amazon Mobile AnalyticsN
Amazon PinpointN
Amazon Polly
Amazon QuickSightN
Amazon RekognitionN
Amazon Simple Email Service (SES)N
Amazon WorkDocsN
Amazon WorkMailN
AWS Application Discovery ServiceN
AWS CodeDeployN
AWS Managed ServicesN
AWS SnowballN
AWS Snowball EdgeN

    2. Compare price

EC2 Price
RegionN. VirginiaSingaporeTokyo
t2.micro$0.012$0.015$0.016
RDS Price 
RegionN. VirginiaSingaporeTokyo
db.t2.micro$0.017$0.026$0.026

ELB
RegionN. VirginiaSingaporeTokyo
ELB$0.025$0.028$0.027

    3. Decide default region

In this case, I needed EFS, so I chose N. Virginia.
  3. Cost estimation before you start using AWS

EFS50GB x 0.3=$15

  4. Set up AWS CLI env on your PC

brew install awscli;
brew install boto;
brew install jq;
aws configure;

# API key information can be gotten from IAM console
# Input region name seeing http://docs.aws.amazon.com/general/latest/gr/rande.html
  5. Create network (VPC)

ansible-playbook playbook/network-security/vpc.yml -i hosts/localhost --extra-vars=@extra-vars/base.yml

vpc.yml
# ansible-playbook playbook/network-security/vpc.yml -i hosts/localhost --extra-vars=@extra-vars/base.yml
---
- hosts: 127.0.0.1
  gather_facts: no
  connection: local

  tasks:
    - name: "make vpc for Prod"
      ec2_vpc:
        state: present
        cidr_block: 10.0.0.0/21
        resource_tags: { "Environment":"Prod",  "Name":"Prod {{ pj }}"  }
        subnets:
          - cidr: 10.0.0.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Public", "Name" : "Prod-Public" }
          - cidr: 10.0.1.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Private", "Name" : "Prod-Private" }
          - cidr: 10.0.2.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Secure", "Name":"Prod-Secure" }
          - cidr: 10.0.3.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Admin", "Name":"Prod-Admin" }
          - cidr: 10.0.4.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Public2", "Name":"Prod-Public2" }
          - cidr: 10.0.5.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Private2", "Name":"Prod-Private2" }
          - cidr: 10.0.6.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Secure2", "Name":"Prod-Secure2" }
          - cidr: 10.0.7.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Prod", "Tier" : "Admin2", "Name":"Prod-Admin2" }
        internet_gateway: True
        route_tables:
          - subnets:
              - 10.0.0.0/24
              - 10.0.1.0/24
              - 10.0.2.0/24
              - 10.0.3.0/24
              - 10.0.4.0/24
              - 10.0.5.0/24
              - 10.0.6.0/24
              - 10.0.7.0/24
            routes:
              - dest: 0.0.0.0/0
                gw: igw
        region: "{{ aws_default_region }}"

    - name: "make vpc for Stg"
      ec2_vpc:
        state: present
        cidr_block: 10.0.8.0/21
        resource_tags: { "Environment":"Stg", "Name":"Stg {{ pj }}" }
        subnets:
          - cidr: 10.0.8.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Public", "Name":"Stg-Public" }
          - cidr: 10.0.9.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Private", "Name":"Stg-Private" }
          - cidr: 10.0.10.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Secure", "Name":"Stg-Secure" }
          - cidr: 10.0.11.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Admin", "Name":"Stg-Admin" }
          - cidr: 10.0.12.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Public2", "Name":"Stg-Public2" }
          - cidr: 10.0.13.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Private2", "Name":"Stg-Private2" }
          - cidr: 10.0.14.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Secure2", "Name":"Stg-Secure2" }
          - cidr: 10.0.15.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Stg", "Tier" : "Admin2", "Name":"Stg-Admin2" }
        internet_gateway: True
        route_tables:
          - subnets:
              - 10.0.8.0/24
              - 10.0.9.0/24
              - 10.0.10.0/24
              - 10.0.11.0/24
              - 10.0.12.0/24
              - 10.0.13.0/24
              - 10.0.14.0/24
              - 10.0.15.0/24
            routes:
              - dest: 0.0.0.0/0
                gw: igw
        region: "{{ aws_default_region }}"

    - name: "make vpc for Dev"
      ec2_vpc:
        state: present
        cidr_block: 10.0.16.0/21
        resource_tags: { "Environment":"Dev", "Name":"Dev {{ pj }}" }
        subnets:
          - cidr: 10.0.16.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Public", "Name":"Dev-Public" }
          - cidr: 10.0.17.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Private", "Name":"Dev-Private"  }
          - cidr: 10.0.18.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Secure", "Name":"Dev-Secure" }
          - cidr: 10.0.19.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Admin", "Name":"Dev-Admin" }
          - cidr: 10.0.20.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Public2", "Name":"Dev-Public2" }
          - cidr: 10.0.21.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Private2", "Name":"Dev-Private2" }
          - cidr: 10.0.22.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Secure2", "Name":"Dev-Secure2" }
          - cidr: 10.0.23.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Dev", "Tier" : "Admin2", "Name":"Dev-Admin2" }
        internet_gateway: True
        route_tables:
          - subnets:
              - 10.0.16.0/24
              - 10.0.17.0/24
              - 10.0.18.0/24
              - 10.0.19.0/24
              - 10.0.20.0/24
              - 10.0.21.0/24
              - 10.0.22.0/24
              - 10.0.23.0/24
            routes:
              - dest: 0.0.0.0/0
                gw: igw
        region: "{{ aws_default_region }}"

    - name: "make vpc for Admin"
      ec2_vpc:
        state: present
        cidr_block: 10.0.24.0/21
        resource_tags: { "Environment":"Admin", "Name":"Admin {{ pj }}" }
        subnets:
          - cidr: 10.0.24.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Public", "Name":"Admin-Public"  }
          - cidr: 10.0.25.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Private", "Name":"Admin-Private" }
          - cidr: 10.0.26.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Secure", "Name":"Admin-Secure" }
          - cidr: 10.0.27.0/24
            az: "{{ dca }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Admin", "Name":"Admin-Admin" }
          - cidr: 10.0.28.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Public2", "Name":"Admin-Public2" }
          - cidr: 10.0.29.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Private2", "Name":"Admin-Private2" }
          - cidr: 10.0.30.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Secure2", "Name":"Admin-Secure2" }
          - cidr: 10.0.31.0/24
            az: "{{ dcb }}"
            resource_tags: { "Environment":"Admin", "Tier" : "Admin2", "Name":"Admin-Admin2" }
        internet_gateway: True
        route_tables:
          - subnets:
              - 10.0.24.0/24
              - 10.0.25.0/24
              - 10.0.26.0/24
              - 10.0.27.0/24
              - 10.0.28.0/24
              - 10.0.29.0/24
              - 10.0.30.0/24
              - 10.0.31.0/24
            routes:
              - dest: 0.0.0.0/0
                gw: igw
        region: "{{ aws_default_region }}"

hosts/localhost
[localhost]
127.0.0.1 ansible_python_interpreter=/usr/local/bin/python

extra-vars/base.yml
pj: 1stclass
aws_default_rds_type: db.t2.micro
aws_default_region: us-east-1 # N. Virgnia
aws_prefered_zone: us-east-1d
dca: us-east-1d
dcb: us-east-1e
aws_ec2_key: ec2-key-pair
aws_ami_id: prod-web

internal_cid: ["202.55.66.114/32", # new-analyzer
"182.19.143.105/32" # SG
]

dev_internal_cid: ["202.55.66.114/32", # new-analyzer
"182.19.143.105/32" # SG
]

jenkins_integrated_cid: ["202.55.66.114/32" # new-analyzer
]

  6. Move the authority to Rout53

Before you delegate DNS's authority from existing DNS to Route 53, create DNS record in Route 53 beforehand.
Create zone from web console for sakuhindb.com.
After it, execute following ansible.

playbook/network-security/route53.yml
# ansible-playbook playbook/network-security/route53.yml -i hosts/localhost --extra-vars=@extra-vars/base.yml
---
- hosts: 127.0.0.1
  gather_facts: no
  connection: local
  tasks:
    - route53:
        command: create
        zone: sakuhindb.com
        record: sakuhindb.com
        type: A
        ttl: 300
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        record: blog-sc.sakuhindb.com
        type: A
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        record: chat.sakuhindb.com
        type: A
        value: 157.7.136.190
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: common.sakuhindb.com
        value: 157.7.136.235
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.image.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.image-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.music.sakuhindb.com
        value: 158.199.143.203

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.product-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.video.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en.video-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: en2.sakuhindb.com
        value: 157.7.136.190
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: image.sakuhindb.com
        value: 157.7.136.235
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: image-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: img.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.blog-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.chat.sakuhindb.com
        value: 157.7.136.190
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.image.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.image-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.product-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: m.video.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: music.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: product-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: rss.sakuhindb.com
        value: 158.199.143.203

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: video.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: video-sc.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: www.sakuhindb.com
        value: 158.199.143.203
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: A
        record: www2.sakuhindb.com
        value: 157.7.136.190
        wait: yes

    - route53:
        command: create
        zone: sakuhindb.com
        type: MX
        record: sakuhindb.com
        value: mail.accessup.org
        wait: yes

After everything becomes O.K, go to your domain name registrer's web page and change NS server assignment to AWS's naming server.
  7. SSL Certificate

You have to have an mail account like admin@$DOMAINAME to show the ownership of the domain.
for 
sakuhindb.com
video.sakuhindb.com
en.image.sakuhidb.com
, you have to create SSL record for sakuhindb.com, *.sakuhindb.com and *.image.sakuhindb.com
  8. ec2 instance

Only initial time, create an instance from web screen.
Before creating web instance, create IAM role for webinstance, which will be used when you create a web instance.
  9. EFS

You have to open 2049 port

sudo yum -y install nfs-utils
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 file-system-id.efs.aws-region.amazonaws.com:/ efs-mount-point;

#Example
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-4e65c607.efs.us-east-1.amazonaws.com:/ /www

[ec2-user@aws-sakuhindb ~]$ df -k
Filesystem                                       1K-blocks    Used        Available Use% Mounted on
devtmpfs                                            498764      60           498704   1% /dev
tmpfs                                               509640       0           509640   0% /dev/shm
/dev/xvda1                                        30830568 1231096         29499224   5% /
fs-4e65c607.efs.us-east-1.amazonaws.com:/ 9007199254740992       0 9007199254740992   0% /www



# File transfer
rsync --rsync-path="rsync" -ave "ssh -c arcfour" /www/* www@prod-sakuhindb:/www
  10. Request to unlock the limitation of SES


  11. Change mail to use SES


  12. Change the code which is using REMOTE_ADDR

sudo yum install mod_extract_forwarded;
  13. Timezone of Amazon Linux


  14. Stop unnecessary process

sudo vi /etc/sysconfig/init
#ACTIVE_CONSOLES=/dev/tty[1-6]
ACTIVE_CONSOLES=/dev/tty1

shutdown -r now

Add comment to this article


[Read other articles]
<=Newer article MRT
=>Older article Only the person who wrote can read


Articles categorized as "AWS by this user"
All articles of this user
Subscribe to RSS
RSS
Term
Category
All
1.Japan
2.Atlassian's products
3.Self
4.Development of this site
5.Japanese comics
6.Japanese anime
7.Weekly hot news of Japanese culture
8.OP/ED/PV
9.Japanese game
10.Ranking
11.Japanese Comics (Manga)
12.Search Engine
13.Japanese drama
14.Japanese otaku culture
15.Programming
16.Ineternet world
17.Movie
18.C/C++
19.BerkeleyDB
20.Apache programming
21.Spam
22.Meteor
23.Marketing
24.Python
25.Scrum
26.JIRA
27.Git
28.CI
29.Jenkins
30.AWS
31.Operation
32.Singapore
33.Cloud
34.Mysql Cluster
35.DevOps
36.Bitbucket
37.Xamarin
38.Good and new
39.Monitoring
40.JavaScript(node.js)
41.React
42.Phillipines
43.Hiring
44.Python
45.SEO
46.Malaysia
47.Mongodb
48.Perl
49.Docker
50.Life hack
51.Dance
52.QA
53.Mysql
54.Digital Life Hack
55.Project management
56.Django
57.Gatsby
58.Administrator
Sayings from S-Cry-Ed

Rule in this world is speed. Even stupid person can write cool novel if he can spend 20 years for it.

If someone helped me, I will help him in return, which is my rule.

To become stronger, consider what is cowardliest thinking. And rebel against the thinking, which will make you stronger.



I am Japanese but working for some English sites.

Doctor Job Career
Nurse Job Career
Top Page top MetaSeachJP Works