憤Ⓘ鬥Ⓣ屎: 8月 2019

2019年8月29日星期四

Packer build image on AWS - Part III

Packer integration with Ansible

Pre-requirement

Don't forget to install ansible

Install pip

# apt-get install python-pip

Install ansible

# pip install ansible==2.8.3

Check ansible version

# ansible --version
ansible 2.8.3

Create Ansible Directory

Packer
├── ISO_Ubuntu_Server_xenial_16.04.6.json
├── ansible
│   ├── all.yml
│   ├── ansible.cfg
│   └── roles
│       └── install-offical-nginx
│           ├── README.md
│           ├── defaults
│           │   └── main.yml
│           ├── files
│           ├── handlers
│           │   └── main.yml
│           ├── meta
│           │   └── main.yml
│           ├── tasks
│           │   └── main.yml
│           ├── templates
│           ├── tests
│           │   ├── inventory
│           │   └── test.yml
│           └── vars
│               └── main.yml
├── http
│   └── preseed.cfg
└── output

13 directories, 12 files

Check ansible.cfg file

[defaults]
inventory      = ./inventory
roles_path      = ./roles
host_key_checking = False
retry_files_enabled = False
gathering = smart

fact_caching = jsonfile
fact_caching_connection = ./cache
fact_caching_timeout = 60

#strategy = mitogen_linear
#strategy_plugins = ./mitogen/ansible_mitogen/plugins/strategy/

callback_whitelist = profile_tasks
[callback_profile_tasks]
task_output_limit = 60


[privilege_escalation]
become=True
become_user=root

Check all.yml file

---
- hosts: all
  become: True
  roles:
  - install-offical-nginx

Check ISO_Ubuntu_Server_xenial_16.04.6.json file

{
  "variables":{
  "vm_description": "Ubuntu Server Image",
  "vm_version": "0.0.1",
  "cpus": "2",
  "memory": "2048",
  "disk_size": "8192",
  "vm_name": "ubuntu",
  "iso_url": "http://ftp.ubuntu-tw.org/mirror/ubuntu-releases/16.04.6/ubuntu-16.04.6-server-amd64.iso",
  "iso_checksum": "16afb1375372c57471ea5e29803a89a5a6bd1f6aabea2e5e34ac1ab7eb9786ac",
  "iso_checksum_type": "sha256",
  "ssh_username": "ubuntu",
  "ssh_password": "ubuntu",
  "s3_bucket_name": "packer-images"
  },
  "provisioners": [
    {
      "type": "ansible",
      "user": "ubuntu",
      "playbook_file": "./ansible/all.yml"
    }
  ],
  "builders": [
    {
      "type": "virtualbox-iso",
      "output_directory": "builds",
      "format": "ova",
      "guest_os_type": "Ubuntu_64",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_checksum_type": "{{user `iso_checksum_type`}}",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_port": 22,
      "ssh_wait_timeout": "2000s",
      "disk_size": "{{user `disk_size`}}",
      "keep_registered": "true",
      "shutdown_command": "echo {{user `ssh_password`}} | sudo -S shutdown -P now",
      "vboxmanage": [
        ["modifyvm", "{{.Name}}", "--cpus", "{{user `cpus`}}"],
        ["modifyvm", "{{.Name}}", "--memory", "{{user `memory`}}"]
      ],
      "http_directory": "./http/",
      "boot_wait": "10s",
      "boot_command": [
        "<enter><wait><f6><esc><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
        " /install/vmlinuz<wait>",
        " noapic<wait>",
        " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg auto<wait>",
        " locale=en_US<wait>",
        " kbd-chooser/method=us<wait>",
        " keyboard-configuration/modelcode=pc105<wait>",
        " keyboard-configuration/layout=US<wait>",
        " keyboard-configuration/variant=US<wait>",
        " netcfg/get_hostname=ubuntu<wait>",
        " fb=false <wait>",
        " debconf/frontend=noninteractive<wait>",
        " console-setup/ask_detect=false<wait>",
        " initrd=/install/initrd.gz -- <wait>",
        "<enter><wait>"
      ]
    }
  ],
  "post-processors": [
    {
      "type": "amazon-import",
      "keep_input_artifact": true,
      "s3_bucket_name": "{{user `s3_bucket_name`}}",
      "ami_name": "ubuntu-16.04.6",
      "license_type": "BYOL",
      "tags": {
        "Description": "Packer Import "
      }
    }
  ]
}

Check `ISO_Ubuntu_Server_xenial_16.04.6.json` is validated via Packer

Check the file is validate via Packer CLI

# packer validate ISO_Ubuntu_Server_xenial_16.04.6.json
Template validated successfully.

You can build right now.

2019年8月28日星期三

Collection Packer Build Error Messages

Q1

Error Messages

Build 'virtualbox-iso' errored: Output directory exists: output-virtualbox-iso
Ans: Use the force flag to delete it prior to building.

Solution

When you will build images, just need to add this parameter -force, like below command

# packer build -force template.json

Reference

Packer Github issue 163

Q2

Error Messages

==> Builds finished but no artifacts were created.
Build 'virtualbox-iso' errored: Error deleting port forwarding rule: VBoxManage error: VBoxManage: error: The machine 'packer-virtualbox-iso-1567040405' is already locked for a session (or being unlocked)
2019/08/29 09:07:01 [INFO] (telemetry) Finalizing.
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports
VBoxManage: error: Context: "LockMachine(a->session, LockType_Write)" at line 531 of file VBoxManageModifyVM.cpp

Solution

Following below parameter pasted in your config

  "builders": [
    {
      (...)
      "post_shutdown_delay": "180s",
      "shutdown_command": "echo {{user `ssh_password`}} | sudo -S shutdown -P now",
      (...)
    }
  ]

Reference

Github Packer issue 5424

2019年8月14日星期三

terraform aws_route_table always change

Terraform Version

Terraform v0.12.6

Q: Terraform aws_route_table always change?

main.tf

resource "aws_route_table" "route_to_nat" {
  vpc_id = "${aws_vpc.vpc.id}"
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_nat_gateway.nat.id}"
  }
  tags = {
    "Name"  = "Route-To-Nat"
  }
}

Error

  # aws_lb_target_group_attachment.test must be replaced
-/+ resource "aws_lb_target_group_attachment" "test" {
      ~ id               = "arn:aws:elasticloadbalancing:us-east-1:xxxxxxxxxxxx:targetgroup/test-lb-tg-tf/583716da0369baf2-20190814081528833600000001" -> (known after apply)
        port             = 80
        target_group_arn = "arn:aws:elasticloadbalancing:us-east-1:xxxxxxxxxxxx:targetgroup/test-lb-tg-tf/583716da0369baf2"
      ~ target_id        = "i-075a770cca5ab74e4" -> (known after apply) # forces replacement
    }

  # aws_route_table.route_to_nat will be updated in-place
  ~ resource "aws_route_table" "route_to_nat" {
        id               = "rtb-06e975d7fb6e235e3"
        owner_id         = "xxxxxxxxxxxx"
        propagating_vgws = []
      ~ route            = [
          - {
              - cidr_block                = "0.0.0.0/0"
              - egress_only_gateway_id    = ""
              - gateway_id                = ""
              - instance_id               = ""
              - ipv6_cidr_block           = ""
              - nat_gateway_id            = "nat-02fb444b359ab0f52"
              - network_interface_id      = ""
              - transit_gateway_id        = ""
              - vpc_peering_connection_id = ""
            },
          + {
              + cidr_block                = "0.0.0.0/0"
              + egress_only_gateway_id    = ""
              + gateway_id                = "nat-02fb444b359ab0f52"
              + instance_id               = ""
              + ipv6_cidr_block           = ""
              + nat_gateway_id            = ""
              + network_interface_id      = ""
              + transit_gateway_id        = ""
              + vpc_peering_connection_id = ""
            },
        ]
        tags             = {
            "Name" = "Route-To-Nat"
        }
        vpc_id           = "vpc-0b5320622318ef4c2"
    }

Solution

Use nat_gateway_id instead of gateway_id.

Data Source: aws route table
- gateway_id - The Internet Gateway ID.
- nat_gateway_id - The NAT Gateway ID.
Resource: aws route table
- gateway_id - (Optional) Identifier of a VPC internet gateway or a virtual private gateway.
- nat_gateway_id - (Optional) Identifier of a VPC NAT gateway.
NOTE on gatewayid and natgatewayid: The AWS API is very forgiving with these two attributes and the awsroutetable resource can be created with a NAT ID specified as a Gateway ID attribute.This will lead to a permanent diff between your configuration and statefile, as the API returns the correct parameters in the returned route table. If you're experiencing constant diffs in your awsroute_table resources, the first thing to check is whether or not you're specifying a NAT ID instead of a Gateway ID, or vice-versa.

Reference:

why terraform aws_instance always recreate

Terraform Version

Terraform v0.12.6

Q: Why terraform aws_instance always recreate?

main.tf

data "aws_ami" "linux" {
  owners = ["self"]

  filter {
    name      = "tag:Description"
    values    = ["${var.instance_ami.base}"]
  }
}

resource "aws_instance" "linux" {
  ami                         = "${data.aws_ami.linux.id}"
  instance_type               = "t2.micro"
  subnet_id                   = "${aws_subnet.public.id}"
  availability_zone           = "${var.availability_zone.us-east-1}"
  security_groups             = ["${aws_security_group.allow_ssh.id}"] 
  associate_public_ip_address = true
  key_name                    = "HelloWorld"

  tags = {
    Name = "Web"
  }
}

Error

Terraform will perform the following actions:

  # aws_instance.linux must be replaced
-/+ resource "aws_instance" "linux" {
        ami                          = "ami-0c7157d76b3a45a34"
      ~ arn                          = "arn:aws:ec2:us-east-1:xxxxxxxxxxxx:instance/i-075a770cca5ab74e4" -> (known after apply)
        associate_public_ip_address  = true
        availability_zone            = "us-east-1a"
      ~ cpu_core_count               = 1 -> (known after apply)
      ~ cpu_threads_per_core         = 1 -> (known after apply)
      - disable_api_termination      = false -> null
      - ebs_optimized                = false -> null
        get_password_data            = false
      + host_id                      = (known after apply)
      ~ id                           = "i-075a770cca5ab74e4" -> (known after apply)
      ~ instance_state               = "running" -> (known after apply)
        instance_type                = "t2.micro"
      ~ ipv6_address_count           = 0 -> (known after apply)
      ~ ipv6_addresses               = [] -> (known after apply)
        key_name                     = "devops"
      - monitoring                   = false -> null
      + network_interface_id         = (known after apply)
      + password_data                = (known after apply)
      + placement_group              = (known after apply)
      ~ primary_network_interface_id = "eni-00fefb01fc9fa12de" -> (known after apply)
      ~ private_dns                  = "ip-172-88-0-166.ec2.internal" -> (known after apply)
      ~ private_ip                   = "172.88.0.166" -> (known after apply)
      ~ public_dns                   = "ec2-xx-xx-xx-xx.compute-1.amazonaws.com" -> (known after apply)
      ~ public_ip                    = "xx.xx.xx.xx" -> (known after apply)
      ~ security_groups              = [ # forces replacement
          + "sg-025186720be2334e9",
        ]
        source_dest_check            = true
        subnet_id                    = "subnet-0007ef0e7a0bf8cc6"
        tags                         = {
            "Name" = "Test"
        }
      ~ tenancy                      = "default" -> (known after apply)
      ~ volume_tags                  = {} -> (known after apply)
      ~ vpc_security_group_ids       = [
          - "sg-025186720be2334e9",
        ] -> (known after apply)

      - credit_specification {
          - cpu_credits = "standard" -> null
        }

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + kms_key_id            = (known after apply)
          + snapshot_id           = (known after apply)
          + volume_id             = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + ephemeral_block_device {
          + device_name  = (known after apply)
          + no_device    = (known after apply)
          + virtual_name = (known after apply)
        }

      + network_interface {
          + delete_on_termination = (known after apply)
          + device_index          = (known after apply)
          + network_interface_id  = (known after apply)
        }

      ~ root_block_device {
          ~ delete_on_termination = false -> (known after apply)
          ~ encrypted             = false -> (known after apply)
          ~ iops                  = 120 -> (known after apply)
          + kms_key_id            = (known after apply)
          ~ volume_id             = "vol-0f77b0b8696ac4379" -> (known after apply)
          ~ volume_size           = 40 -> (known after apply)
          ~ volume_type           = "gp2" -> (known after apply)
        }
    }

Solution

Use vpc_security_group_ids instead of security_groups.

Terraform Data Source: aws_instance
- vpc_security_group_ids - The associated security groups in a non-default VPC.
- security_groups - The associated security groups.
Terraform Resource: aws_instance
- vpc_security_group_ids - (Optional, VPC only) A list of security group IDs to associate with.
- security_groups - (Optional, EC2-Classic and default VPC only) A list of security group names (EC2-Classic) or IDs (default VPC) to associate with.

Reference:

AWS instance is always recreated

2019年8月8日星期四

AWS CLI Tips

Finding Owner AMIs ID

aws ec2 describe-images --owners self --filters "Name=tag:Description,Values=Base Ubuntu Image" --query 'Images[*].{ID:ImageId}' --output 'text'

Reference:

2019年8月5日星期一

Ansible Failed - Failed to lock apt for exclusive operation

Manage Node

System Information

ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

Ansible Verison

ansible 2.8.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/nobody/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Python/2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.10 (default, Feb 22 2019, 21:55:15) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]

Remote Node

System Information

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:    16.04
Codename:   xenial

Playbook

---
- hosts: test
  become: True
  roles:
    - { role: common }

Roles

# tree roles/common
roles/common
├── README.md
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── tasks
│   ├── install_packages.yml
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

8 directories, 9 files

cat roles/common/tasks/install_packages.yml

---
- name: Apt Update
  apt:
    force_apt_get: yes
    update_cache: yes
    cache_valid_time: 3600


- name: Ensure install software-properties-common
  apt:
    name: software-properties-common
    state: present
    force_apt_get: yes

Error Messages

First error message

[WARNING]: Updating cache and auto-installing missing dependency: python-apt

Fixed first error

$ sudo apt-get isntall -y python-apt

Second error error

Failed to lock apt for exclusive operation

Fixed second error

Add parameter to apt task

cache_valid_time: 3600

Reference: - Ansible Failed - apt lock

訂閱：文章 (Atom)

2019年8月29日 星期四

Packer build image on AWS - Part III

Packer integration with Ansible

Pre-requirement

Create Ansible Directory

Check ISO_Ubuntu_Server_xenial_16.04.6.json is validated via Packer

2019年8月28日 星期三

Collection Packer Build Error Messages

Q1

Q2

2019年8月14日 星期三

terraform aws_route_table always change

Terraform Version

Q: Terraform aws_route_table always change?

why terraform aws_instance always recreate

Q: Why terraform aws_instance always recreate?

2019年8月8日 星期四

AWS CLI Tips

Finding Owner AMIs ID

aws ec2 describe-images --owners self --filters "Name=tag:Description,Values=Base Ubuntu Image" --query 'Images[*].{ID:ImageId}' --output 'text' Reference: describe-images Finding an AMI ID Using AWS CLI in AWS Marketplace 尋找 Linux AMI

2019年8月5日 星期一

Ansible Failed - Failed to lock apt for exclusive operation

Manage Node

Remote Node

Playbook

Roles

Error Messages

2019年8月29日星期四

Check `ISO_Ubuntu_Server_xenial_16.04.6.json` is validated via Packer

2019年8月28日星期三

2019年8月14日星期三

2019年8月8日星期四

`aws ec2 describe-images --owners self --filters "Name=tag:Description,Values=Base Ubuntu Image" --query 'Images[*].{ID:ImageId}' --output 'text'`

Reference:

describe-images

Finding an AMI ID Using AWS CLI in AWS Marketplace

尋找 Linux AMI

2019年8月5日星期一