憤Ⓘ鬥Ⓣ屎: 7月 2019

2019年7月28日星期日

Packer build image on AWS - Part II

System Information

# sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

Pre-requirement

Install VirtualBox

# brew cask install virtualbox

Install Packer

# brew install packer

Package	Version	CLI
VirtualBox	6.0.6r130049	VBoxManage -v
Packer	1.4.1	packer --version

Create Packer Directory

Create Packer Directory Structure like below

Packer
├── ISO_Ubuntu_Server_xenial_16.04.6.json
├── http
│   └── preseed.cfg
└── output

Create Ubuntu Pressed.cfg file in http directory

### Localization
d-i debian-installer/language string en d-i debian-installer/locale string en_US.UTF-8
d-i debian-installer/country string TW

### Keyboard selection.
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us
d-i netcfg/choose_interface select auto

### Setting hostname and domain assigned from dhcp server.
d-i netcfg/get_hostname string ubuntu
d-i netcfg/get_domain string unassigned-domain

### Mirror settings
d-i mirror/country string manual
d-i mirror/http/hostname string archive.ubuntu.com
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string

### To create a normal user account.
d-i passwd/user-fullname string Ubuntu User
d-i passwd/username string ubuntu

### Normal user's password, either in clear text
d-i passwd/user-password password ubuntu
d-i passwd/user-password-again password ubuntu

### Create the first user with the specified UID instead of the default.
d-i passwd/user-uid string 1000

### Allow installer use weak password
d-i user-setup/allow-password-weak boolean true

### The user account will be added to some standard initial groups.
d-i passwd/user-default-groups string adm cdrom plugdev lpadmin sambashare sudo ubuntu

### Set to true if you want to encrypt the first user's home directory.
d-i user-setup/encrypt-home boolean false

### Clock and time zone setup
d-i clock-setup/utc boolean true
d-i time/zone string Asia/Taipei

### Partitioning
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic

### This makes partman automatically partition without confirmation.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

### Apt setup
d-i apt-setup/services-select multiselect security
d-i apt-setup/security_host string security.ubuntu.com
d-i apt-setup/security_path string /ubuntu
d-i debian-installer/allow_unauthenticated boolean true

### Package selection
tasksel tasksel/first multiselect none

### Individual additional packages to install
d-i pkgsel/include string openssh-server build-essential net-tools chrony vim wget curl

### Policy for applying updates.
d-i pkgsel/update-policy select none

### When installer has finished and the system's locate database will be updated.
d-i pkgsel/updatedb boolean true

### Makes grub install automatically to the MBR
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i grub-installer/bootdev  string /dev/sda

### Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note

### Running custom commands during the installation
d-i preseed/late_command string chroot /target sh -c "/bin/echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers; \
/bin/echo 'Hello World' >> /tmp/Hello.txt

Create ISO_Ubuntu_Server_xenial_16.04.6.json in Packer directory

{
  "variables":{
  "vm_description": "Ubuntu Server Image",
  "vm_version": "0.0.1",
  "cpus": "2",
  "memory": "2048",
  "disk_size": "40960",
  "vm_name": "ubuntu",
  "iso_url": "http://ftp.ubuntu-tw.org/mirror/ubuntu-releases/16.04.6/ubuntu-16.04.6-server-amd64.iso",
  "iso_checksum": "16afb1375372c57471ea5e29803a89a5a6bd1f6aabea2e5e34ac1ab7eb9786ac",
  "iso_checksum_type": "sha256",
  "ssh_username": "ubuntu",
  "ssh_password": "ubuntu",
  "s3_bucket_name": "packer-images"
  },
  "builders": [
    {
      "type": "virtualbox-iso",
      "output_directory": "builds",
      "format": "ova",
      "guest_os_type": "Ubuntu_64",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_checksum_type": "{{user `iso_checksum_type`}}",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_port": 22,
      "ssh_wait_timeout": "1000s",
      "disk_size": "{{user `disk_size`}}",
      "keep_registered": "true",
      "shutdown_command": "echo {{user `ssh_password`}} | sudo -S shutdown -P now",
      "vboxmanage": [
        ["modifyvm", "{{.Name}}", "--cpus", "{{user `cpus`}}"],
        ["modifyvm", "{{.Name}}", "--memory", "{{user `memory`}}"]
      ],
      "http_directory": "./http/",
      "boot_wait": "10s",
      "boot_command": [
        "<enter><wait><f6><esc><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        " /install/vmlinuz<wait>",
        " noapic<wait>",
        " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg auto<wait>",
        " locale=en_US<wait>",
        " kbd-chooser/method=us<wait>",
        " keyboard-configuration/modelcode=pc105<wait>",
        " keyboard-configuration/layout=US<wait>",
        " keyboard-configuration/variant=US<wait>",
        " netcfg/get_hostname=ubuntu<wait>",
        " fb=false <wait>",
        " debconf/frontend=noninteractive<wait>",
        " console-setup/ask_detect=false<wait>",
        " initrd=/install/initrd.gz -- <wait>",
        "<enter><wait>"
      ]
    }
  ],
  "post-processors": [
    {
      "type": "amazon-import",
      "keep_input_artifact": true,
      "s3_bucket_name": "{{user `s3_bucket_name`}}",
      "ami_name": "ubuntu-16.04.6",
      "license_type": "BYOL",
      "tags": {
        "Description": "Packer Import "
      }
    }
  ]
}

Check `ISO_Ubuntu_Server_xenial_16.04.6.json` is validated via Packer

Check the file is validate via Packer CLI

# packer validate ISO_Ubuntu_Server_xenial_16.04.6.json
Template validated successfully.

Now we can build images

# packer build ISO_Ubuntu_Server_xenial_16.04.6.json

if your want to see more information, please use Debug Mode

# PACKER_LOG=1 packer build ISO_Ubuntu_Server_xenial_16.04.6.json

Check S3 bucket

aws s3 ls s3://packer-images --recursive --summarize --human-readable

Check import task

# aws ec2 describe-import-image-tasks

Check import task status

aws ec2 describe-import-image-tasks --import-task-ids import-ami-xxxxxxxxxxxxxxxx

2019年7月25日星期四

Packer build image on AWS - Part I

Pre-reqirement

Create S3 bucket

# aws s3api create-bucket --bucket packer-images

Display S3 bucket content

# aws s3 ls s3://packer-images --recursive --summarize --human-readable

Delete S3 bucket

# aws s3api delete-bucket --bucket packer-images

Create Role - vmimport

Create trust-policy.json

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": { "Service": "vmie.amazonaws.com" },
         "Action": "sts:AssumeRole",
         "Condition": {
            "StringEquals":{
               "sts:Externalid": "vmimport"
            }
         }
      }
   ]
}

Create a role named vmimport and give VM import/Export access.

Ensure that your full path of trust-policy.json file, and that prefix file:///your/full/path/trust-policy.json

# aws iam create-role --role-name vmimport --assume-role-policy-document "file:///tmp/packer/trust-policy.json"

Check Role stting is correct

aws iam get-role --role-name vmimport

Create Policy = role-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::packer-images",
                "arn:aws:s3:::packer-images/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:ModifySnapshotAttribute",
                "ec2:CopySnapshot",
                "ec2:RegisterImage",
                "ec2:Describe*"
            ],
            "Resource": "*"
        }
    ]
}

Create a policy and attach policy to the role.

Ensure that your full path of role-policy.json file. And that prefix file:///your/full/path/role-policy.json

aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document "file:///tmp/packer/role-policy.json"

Check vmimport policy setting is correct.

aws iam get-role-policy --role-name vmimport --policy-name vmimport-policy

Reference:

2019年7月24日星期三

Install boto3 on MAC

System Information

# sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

Check pip version

# pip --version
pip 19.1.1 from /Library/Python/2.7/site-packages/pip-19.1.1-py2.7.egg/pip (python 2.7)

Install boto3

Pre-requirement Package

if you see an error like below

ERROR: Could not install packages due to an EnvironmentError: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/man'

and then you have one parameter missing needs add --user

# sudo pip install matplotlib --user

Install boto3

if you see an error like below

ERROR: Cannot uninstall 'six'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

and then you should add the parameter --ignore-installed six

# sudo pip install boto3 --ignore-installed six

Now you can use boto3 call AWS API

Reference:

2019年7月22日星期一

Ansible 判斷空字串

Ansible Tips

判斷空字串

- name: Do something with my_var.
  shell: "check do_something_with {{ my_var }}"
  when: my_var != ''

舊方法

when: my_var != ''

使用 ansible-lint 會噴警告

[602] Don't compare to empty string
/roles/projects/tasks/main.yml:195
        - my_var.stdout != ""

新方法
```
when: my_var | length > 0
```
除了上面講的新方法，還可以用另外一招，就是使用 .ansible-lint，將上述警告的代碼 602 加入至此檔案，加入後再次執行 ansible-lint 就不會在噴警告了。

Reference:

AWS CLI 設定

System info

Check Mac OS version

# sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

Install the AWS CLI on macOS Using pip

Check pip3 version

# pip3 --version

Install the AWS CLI on macOS Using pip
1. Download and install the latest version of Python from the download page of Python.org
2. Download and run the pip3 installation script.
```
# curl -O https://bootstrap.pypa.io/get-pip.py
# python3 get-pip.py --user
```
3. Use pip3 to install the AWS CLI.
```
# pip3 install awscli --upgrade --user
```
4. Verify that the AWS CLI is installed correctly.
```
# aws --version
aws-cli/1.16.203 Python/3.7.3 Darwin/18.6.0 botocore/1.12.193
```
5. To upgrade to the latest version, run the installation command again.
```
# pip3 install awscli --upgrade --user
```
Add the AWS CLI Executable to Your macOS Command Line Path
1. Find out AWS CLI installation path
```
# which aws
/Users/nobody/Library/Python/3.7/bin/aws
```
2. To modify your PATH variable
```
# echo 'export PATH=/Users/nobody/Library/Python/3.7/bin:$PATH >> ~/.zshrc
```
3. Load the updated profile into your current session.
```
# source ~/.zshrc
```

Configuration and Multiple Profiles for AWS

The aws configure command is the fastest way to set up your AWS CLI installation

# aws configure --profile "Your Name"
AWS Access Key ID [None]: "Your Access Key ID"
AWS Secret Access Key [None]: "Your Secret Access Key"
Default region name [None]: us-east-1
Default output format [None]: table

Output format:
- json
- text
- table
List .aws directory

# tree ~/.aws
/Users/nobody/.aws
├── config
└── credentials

and your can see the config in ~/.aws/config and ~/.aws/credentials

Setting environment variable to get AWS config

# export AWS_DEFAULT_PROFILE=nobody

# echo 'export AWS_DEFAULT_PROFILE=nobody' >> ~/.zshrc
# source ~/.zshrc

List aws profiles

# aws configure list --profile nobody

Get Current Region

# aws configure get nobody.region

Reference:

2019年7月7日星期日

Use curl to confirm that images are using gzip

#!/bin/bash

PCHOME_URL='https://a.ecimg.tw/css/2016/style/images/v201607/mobile/v1.9/mobile_loading.svg'
MOMO_URL='https://image.momoshop.com.tw/ecm/img/de/0/bt_0_042/twcaseal_small1.jpg'


for URLs in ${PCHOME_URL} ${MOMO_URL}
do

    echo "\033[33;44m Check Respones Headers \033[0m"
    curl -H "Cache-Control: no-cache" -H "Accept-Encoding:  gzip, deflate, br" -Is ${URLs}
    echo "\033[33;44m Check content-length \033[0m"
    curl -H "Cache-Control: no-cache" -H "Accept-Encoding:  gzip, deflate, br" -Is ${URLs} | awk '/content-length/ {print $2}'
done

Reference:

2019年7月4日星期四

Git Tips

刪除遠端 branch

查看遠端 branch，可以執行 git branch -r


git branch -r | grep -Ev 'master|develop' | cut -d / -f 2- | xargs -I {} sh -c "git push origin :{}"

當刪除完遠端 branch 後，可以執行 git branch -r 確認一下

刪除本地 branch 的紀錄


git branch -r | grep -Ev 'master|develop' | xargs -I {} sh -c "git branch -d -r {}"

執行完成後，可以執行 git branch -a 確認一下。

查目前預設的 branch


git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@'


git ls-remote --symref origin HEAD | awk '/refs/ {print $2}' | awk -F'/' '{print $3}'

Merge 時出現的 error messages


fatal: You have not concluded your merge (MERGE_HEAD exists).
Please, commit your changes before you merge.

有二種解法，一種是保留本地端的修改，另一種是拉 upstream 上的來覆蓋本地端的

第一種解法:
```
git merge --abort
git reset --merge
```
合併完之後，記得一定要在重新 commit 一次，然後再重新 git fetch

第二種解法:


git fetch --all
git reset --hard origin/master
git fetch upstream
git merge upstream/develop
git push

抓取遠端 pull requests 下來到 local 端修改

要先確認遠端 PR 的 ID

底下的 ID 跟 BRANCHNAME 要做修改，請先查好對應的 ID 及 BRANCH NAME


git fetch origin pull/ID/head:BRANCHNAME

接下來就可以直接 checkout 到抓下來的 branch 了


git checkout BRANCHNAME

Reference

How to list branches in Git?

【狀況題】怎麼刪除遠端的分支？

git remote branch deleted but still appears in 'branch -a'

git - how to get default branch?

You have not concluded your merge (MERGE_HEAD exists)

How can I check out a GitHub pull request with git?

訂閱：文章 (Atom)

2019年7月28日 星期日

System Information

# sw_vers ProductName: Mac OS X ProductVersion: 10.14.5 BuildVersion: 18F132

Pre-requirement

Install VirtualBox # brew cask install virtualbox Install Packer # brew install packer PackageVersionCLI VirtualBox6.0.6r130049VBoxManage -v Packer1.4.1packer --version

Create Packer Directory

Check ISO_Ubuntu_Server_xenial_16.04.6.json is validated via Packer

2019年7月25日 星期四

Pre-reqirement

2019年7月24日 星期三

System Information

# sw_vers ProductName: Mac OS X ProductVersion: 10.14.5 BuildVersion: 18F132

Check pip version

# pip --version pip 19.1.1 from /Library/Python/2.7/site-packages/pip-19.1.1-py2.7.egg/pip (python 2.7)

Install boto3

2019年7月22日 星期一

Ansible Tips

System info

Check Mac OS version # sw_vers ProductName: Mac OS X ProductVersion: 10.14.5 BuildVersion: 18F132

Install the AWS CLI on macOS Using pip

Configuration and Multiple Profiles for AWS

2019年7月7日 星期日

Use curl to confirm that images are using gzip

2019年7月4日 星期四

刪除遠端 branch

查看遠端 branch，可以執行 git branch -r git branch -r | grep -Ev 'master|develop' | cut -d / -f 2- | xargs -I {} sh -c "git push origin :{}" 當刪除完遠端 branch 後，可以執行 git branch -r 確認一下

刪除本地 branch 的紀錄

git branch -r | grep -Ev 'master|develop' | xargs -I {} sh -c "git branch -d -r {}" 執行完成後，可以執行 git branch -a 確認一下。

查目前預設的 branch

git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@' or git ls-remote --symref origin HEAD | awk '/refs/ {print $2}' | awk -F'/' '{print $3}'

Merge 時出現的 error messages

抓取遠端 pull requests 下來到 local 端修改

要先確認遠端 PR 的 ID 底下的 ID 跟 BRANCHNAME 要做修改，請先查好對應的 ID 及 BRANCH NAME git fetch origin pull/ID/head:BRANCHNAME 接下來就可以直接 checkout 到抓下來的 branch 了 git checkout BRANCHNAME

Reference

2019年7月28日星期日

`# sw_vers ProductName: Mac OS X ProductVersion: 10.14.5 BuildVersion: 18F132`

Install VirtualBox

`# brew cask install virtualbox`

Install Packer

`# brew install packer`

Package Version CLI

VirtualBox 6.0.6r130049 VBoxManage -v

Packer 1.4.1 packer --version

Check `ISO_Ubuntu_Server_xenial_16.04.6.json` is validated via Packer

2019年7月25日星期四

2019年7月24日星期三

`# sw_vers ProductName: Mac OS X ProductVersion: 10.14.5 BuildVersion: 18F132`

`# pip --version pip 19.1.1 from /Library/Python/2.7/site-packages/pip-19.1.1-py2.7.egg/pip (python 2.7)`

2019年7月22日星期一

Check Mac OS version

`# sw_vers ProductName: Mac OS X ProductVersion: 10.14.5 BuildVersion: 18F132`

2019年7月7日星期日

2019年7月4日星期四

查看遠端 branch，可以執行 `git branch -r`

`git branch -r | grep -Ev 'master|develop' | cut -d / -f 2- | xargs -I {} sh -c "git push origin :{}"`

當刪除完遠端 branch 後，可以執行 `git branch -r` 確認一下

`git branch -r | grep -Ev 'master|develop' | xargs -I {} sh -c "git branch -d -r {}"`

執行完成後，可以執行 `git branch -a` 確認一下。

`git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@'`

or

`git ls-remote --symref origin HEAD | awk '/refs/ {print $2}' | awk -F'/' '{print $3}'`

要先確認遠端 PR 的 ID

底下的 `ID` 跟 `BRANCHNAME` 要做修改，請先查好對應的 ID 及 BRANCH NAME

`git fetch origin pull/ID/head:BRANCHNAME`

接下來就可以直接 checkout 到抓下來的 branch 了

`git checkout BRANCHNAME`