Packer build image on AWS - Part I

Pre-reqirement

• Create S3 bucket

  # aws s3api create-bucket --bucket packer-images

• Display S3 bucket content

  # aws s3 ls s3://packer-images --recursive --summarize --human-readable

• Delete S3 bucket

  # aws s3api delete-bucket --bucket packer-images

• Create Role - vmimport
  • Create trust-policy.json

  {
     "Version": "2012-10-17",
     "Statement": [
        {
           "Effect": "Allow",
           "Principal": { "Service": "vmie.amazonaws.com" },
           "Action": "sts:AssumeRole",
           "Condition": {
              "StringEquals":{
                 "sts:Externalid": "vmimport"
              }
           }
        }
     ]
  }

  • Create a role named vmimport and give VM import/Export access.
  Ensure that your full path of trust-policy.json file, and that prefix file:///your/full/path/trust-policy.json

  # aws iam create-role --role-name vmimport --assume-role-policy-document "file:///tmp/packer/trust-policy.json"

  • Check Role stting is correct

  aws iam get-role --role-name vmimport

  • Create Policy = role-policy.json

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:GetBucketLocation",
                  "s3:GetObject",
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::packer-images",
                  "arn:aws:s3:::packer-images/*"
              ]
          },
          {
              "Effect": "Allow",
              "Action": [
                  "ec2:ModifySnapshotAttribute",
                  "ec2:CopySnapshot",
                  "ec2:RegisterImage",
                  "ec2:Describe*"
              ],
              "Resource": "*"
          }
      ]
  }

  • Create a policy and attach policy to the role.
  Ensure that your full path of role-policy.json file. And that prefix file:///your/full/path/role-policy.json

  aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document "file:///tmp/packer/role-policy.json"

  • Check vmimport policy setting is correct.

  aws iam get-role-policy --role-name vmimport --policy-name vmimport-policy

Reference:

• 使用 VM Import/Export 將 VM 匯入為映像
• amazon-import: If the import job fails Packer wait until timeout instead of failing immediately
• How To Configure VMimport Role
• How to set up for and use Packer's Amazon Import post-processor
• Import Machine Images Into AWS With Packer

Install boto3 on MAC

System Information

# sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

Check pip version

# pip --version
pip 19.1.1 from /Library/Python/2.7/site-packages/pip-19.1.1-py2.7.egg/pip (python 2.7)

Install boto3

• Pre-requirement Package

if you see an error like below

ERROR: Could not install packages due to an EnvironmentError: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/man'

and then you have one parameter missing needs add --user

# sudo pip install matplotlib --user

• Install boto3

if you see an error like below

ERROR: Cannot uninstall 'six'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

and then you should add the parameter --ignore-installed six

# sudo pip install boto3 --ignore-installed six

Now you can use boto3 call AWS API

Reference:

• Quickstart
• Python: Cannot uninstall 'six' while pip install tld

Ansible 判斷空字串

Ansible Tips

• 判斷空字串

- name: Do something with my_var.
  shell: "check do_something_with {{ my_var }}"
  when: my_var != ''

• 舊方法

  when: my_var != ''

  使用 ansible-lint 會噴警告

  [602] Don't compare to empty string
  /roles/projects/tasks/main.yml:195
          - my_var.stdout != ""

• 新方法

  when: my_var | length > 0

  除了上面講的新方法，還可以用另外一招，就是使用 .ansible-lint，將上述警告的代碼 602 加入至此檔案，加入後再次執行 ansible-lint 就不會在噴警告了。

Reference:

• ansible-lint issue 450
• ansible-lint issue 457
• ansible-lint usage
• ansible-lint default rule

AWS CLI 設定

System info

• Check Mac OS version

# sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

Install the AWS CLI on macOS Using pip

• Check pip3 version

# pip3 --version

• Install the AWS CLI on macOS Using pip
  1. Download and install the latest version of Python from the download page of Python.org
  2. Download and run the pip3 installation script.

     # curl -O https://bootstrap.pypa.io/get-pip.py
     # python3 get-pip.py --user

  3. Use pip3 to install the AWS CLI.

     # pip3 install awscli --upgrade --user

  4. Verify that the AWS CLI is installed correctly.

     # aws --version
     aws-cli/1.16.203 Python/3.7.3 Darwin/18.6.0 botocore/1.12.193

  5. To upgrade to the latest version, run the installation command again.

     # pip3 install awscli --upgrade --user

• Add the AWS CLI Executable to Your macOS Command Line Path
  1. Find out AWS CLI installation path

     # which aws
     /Users/nobody/Library/Python/3.7/bin/aws

  2. To modify your PATH variable

     # echo 'export PATH=/Users/nobody/Library/Python/3.7/bin:$PATH >> ~/.zshrc

  3. Load the updated profile into your current session.

     # source ~/.zshrc

Configuration and Multiple Profiles for AWS

• The aws configure command is the fastest way to set up your AWS CLI installation

# aws configure --profile "Your Name"
AWS Access Key ID [None]: "Your Access Key ID"
AWS Secret Access Key [None]: "Your Secret Access Key"
Default region name [None]: us-east-1
Default output format [None]: table

• Output format:
  • json
  • text
  • table
• List .aws directory

# tree ~/.aws
/Users/nobody/.aws
├── config
└── credentials

and your can see the config in ~/.aws/config and ~/.aws/credentials

• Setting environment variable to get AWS config

# export AWS_DEFAULT_PROFILE=nobody

or

# echo 'export AWS_DEFAULT_PROFILE=nobody' >> ~/.zshrc
# source ~/.zshrc

• List aws profiles

# aws configure list --profile nobody

• Get Current Region

# aws configure get nobody.region

• Use

Reference:

• Configuring the AWS CLI
• Available Regions
• Supported environment variables
• AWS Regions and Endpoints

Use curl to confirm that images are using gzip

Use curl to confirm that images are using gzip

#!/bin/bash

PCHOME_URL='https://a.ecimg.tw/css/2016/style/images/v201607/mobile/v1.9/mobile_loading.svg'
MOMO_URL='https://image.momoshop.com.tw/ecm/img/de/0/bt_0_042/twcaseal_small1.jpg'


for URLs in ${PCHOME_URL} ${MOMO_URL}
do

    echo "\033[33;44m Check Respones Headers \033[0m"
    curl -H "Cache-Control: no-cache" -H "Accept-Encoding:  gzip, deflate, br" -Is ${URLs}
    echo "\033[33;44m Check content-length \033[0m"
    curl -H "Cache-Control: no-cache" -H "Accept-Encoding:  gzip, deflate, br" -Is ${URLs} | awk '/content-length/ {print $2}'
done

Reference:

• Why You Need to Optimize SVGs
• How SVG Optimization Can Help In Improving Website Speed?
• codepen
• Check GZip Encoding with curl
• Best Practices for Using the Vary Header
• Principles of content negotiation
• Accept-Encoding
• Quality values
• Active Gzip Compression

Git Tips

刪除遠端 branch

查看遠端 branch，可以執行 git branch -r

git branch -r | grep -Ev 'master|develop' | cut -d / -f 2- | xargs -I {} sh -c "git push origin :{}"

當刪除完遠端 branch 後，可以執行 git branch -r 確認一下

刪除本地 branch 的紀錄

git branch -r | grep -Ev 'master|develop' | xargs -I {} sh -c "git branch -d -r {}"

執行完成後，可以執行 git branch -a 確認一下。

查目前預設的 branch

git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@'

or

git ls-remote --symref origin HEAD | awk '/refs/ {print $2}' | awk -F'/' '{print $3}'

Merge 時出現的 error messages

fatal: You have not concluded your merge (MERGE_HEAD exists).
Please, commit your changes before you merge.

有二種解法，一種是保留本地端的修改，另一種是拉 upstream 上的來覆蓋本地端的

• 第一種解法:

  
  git merge --abort
  git reset --merge
  
  

  合併完之後，記得一定要在重新 commit 一次，然後再重新 git fetch
• 第二種解法:

  
  git fetch --all
  git reset --hard origin/master
  git fetch upstream
  git merge upstream/develop
  git push
  
  

抓取遠端 pull requests 下來到 local 端修改

• 要先確認遠端 PR 的 ID

底下的 ID 跟 BRANCHNAME 要做修改，請先查好對應的 ID 及 BRANCH NAME

git fetch origin pull/ID/head:BRANCHNAME

• 接下來就可以直接 checkout 到抓下來的 branch 了

git checkout BRANCHNAME

Reference

• How to list branches in Git?
• 【狀況題】怎麼刪除遠端的分支？
• git remote branch deleted but still appears in 'branch -a'
• git - how to get default branch?
• You have not concluded your merge (MERGE_HEAD exists)
• How can I check out a GitHub pull request with git?

Enable Authy OTP on Ubuntu

Environment:

OS: Ubuntu 16.04.6 LTS

• Step1: Register Twilio Account
  首先要先去 Twilio 註冊帳號，接著請連到 Authy Dashboard，這邊我是使用 Sign In with Twilio
  登入後，請執行
  • 建立 Application
  • 建立 User Account
  • 也可以建立 Collaborators User ( Optional )
  請記住剛剛的 API KEY 安裝時會用到
• Step2: 登入至自已的 Linux 主機，安裝 authy-ssh
  • 登入 Linux 主機後，請切換成 root
  • 下載 auth-ssh 並安裝

  # wget "https://raw.github.com/authy/authy-ssh/master/authy-ssh" -O authy-ssh
  # sudo bash authy-ssh install /usr/local/bin
  # sudo /usr/local/bin/authy-ssh enable `whoami`   
  # sudo service ssh restart

  這邊測試的時候，請打開你的 APP，將你的 TOKEN 打進去就好了，除非你是用簡訊驗証的，才先打 sms，等 Authy 發簡訊給你 TOKEN 後，再輸入即可。

  # authy-ssh test
  Authy Token (type 'sms' to request a SMS token): 

Reference:

• Add 2FA To Your SSH In 30 Seconds
• How To Install Authy And Configure Two-Factor Authentication For SSH