Build an AMI #4

jcf · 2018-04-11T14:38:55Z

The AMI now includes Clojure and common Clojure tooling (i.e. Boot and
Leiningen).

Support for both CodeDeploy and CloudFront are included (although not configured
or enabled out of the box), and packages from the AUR can be pulled down and
cached in a local repo.

Tasks

Test CodeDeploy starts up on EC2
Make sure log output reaches CloudWatch (need to create a log group and the associated config)
~~Make sure CodeDeploy isn't logging to a file~~ Write CodeDeploy logs to journald #7

The AMI now includes Clojure and common Clojure tooling (i.e. Boot and Leiningen). Support for both CodeDeploy and CloudFront are included (although not configured or enabled out of the box), and packages from the AUR can be pulled down and cached in a local repo.

jcf · 2018-04-11T14:40:04Z

I'd consider using something like Ansible to provision these changes in future. Doing everything with a set of Bash scripts is already a little painful, and will only get worse as complexity increases.

jcf · 2018-04-11T14:40:45Z

arch.json

@@ -1,35 +1,63 @@
 {
-    "variables": {},
+    "_comment": "Keys prefixed with an underscore are comments.",


As I don't have any comments I could remove this, but maybe it'll help someone down the line.

jcf · 2018-04-11T14:42:16Z

etc/makepkg.conf

+# Default: put built package and cached source in build directory
+#
+#-- Destination: specify a fixed directory where all packages will be placed
+PKGDEST=/var/cache/pacman/juxt


This file is in here so I can point makepkg at the new Pacman cache dir where local PKGBUILD and anything from AUR will end up.

You can query packages not in 'core' Arch repos like so:

pacman -Sl juxt

jcf · 2018-04-11T14:42:57Z

etc/sudoers

+# %wheel ALL=(ALL) ALL
+
+## Same thing without a password
+%wheel ALL=(ALL) NOPASSWD: ALL


Passwordless sudo so we can install things more easily during deployment. Getting makepkg to work as the rock user when it asks to sudo things is not straightforward.

jcf · 2018-04-11T14:43:16Z

scripts/install-base.sh

+# Rock user
+
+# Group for running pacman commands without needing to enter a password.
+groupadd build


Build group can go as I'm no longer using it.

jcf · 2018-04-11T14:44:15Z

scripts/install-custom.sh

+install_pkgs \
+    jq \
+    pacutils \
+    repose


repose is on the way out and people are moving to repoctl I'm told. Something to look out for in the not too distant future.

jcf · 2018-04-11T14:44:57Z

scripts/install-custom.sh

+# We skip the PGP check because we won't have AladW's GPG key at this point.
+#
+# TODO Pull in GPG key and use it to check package.
+su --preserve-environment -c 'makepkg --skippgpcheck' rock


Skipping PGP checks is me being lazy. It's the one package, but we could import AladW's key and do this right perhaps?

jcf · 2018-04-11T14:45:59Z

scripts/install-custom.sh

+    systemd-cloud-watch
+
+# To install any packages from the AUR can now be done like so:
+#     aursync <pkg0> <pkg1> <pkg2> ...


This comment isn't clear enough. You still have to get Pacman to put the package contents in the right place. This only installs in as much as it builds the package and adds it to your Pacman cache and the JUXT package repo.

jcf · 2018-04-11T14:46:31Z

share/Makefile

+all: $(DIRS)
+
+$(DIRS): force
+	cd $@ && makepkg --printsrcinfo > .SRCINFO


Easier than escaping the redirect in the shell script that does the provisioning.

jcf · 2018-04-11T16:43:31Z

terraform/main.tf

+  user_data         = "${data.template_file.user_data.rendered}"
+
+  # This needs to be created manually.
+  key_name = "rock"


I've created this key in the JUXT AWS account, and can share it with you, @malcolmsparks.

Sent your way.

jcf · 2018-04-11T16:45:42Z

terraform/Makefile

+	terraform plan -out proposed.plan
+
+apply:
+	terraform apply proposed.plan


Plan looks reasonable, but I don't want to apply this anywhere near customer data etc.

@malcolmsparks do you have somewhere I can test this out tomorrow? If I can get an example user data script that configures things and starts up the services I think this work will be more or less done.

terraform fmt terraform plan -out proposed.plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.template_file.user_data: Refreshing state... data.aws_ami.rock: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + aws_instance.rock id: <computed> ami: "ami-cec09ab7" associate_public_ip_address: <computed> availability_zone: "eu-west-1" ebs_block_device.#: <computed> ephemeral_block_device.#: <computed> get_password_data: "false" instance_state: <computed> instance_type: "t2.small" ipv6_address_count: <computed> ipv6_addresses.#: <computed> key_name: "rock" network_interface.#: <computed> network_interface_id: <computed> password_data: <computed> placement_group: <computed> primary_network_interface_id: <computed> private_dns: <computed> private_ip: <computed> public_dns: <computed> public_ip: <computed> root_block_device.#: "1" root_block_device.0.delete_on_termination: "true" root_block_device.0.volume_id: <computed> root_block_device.0.volume_size: "20" root_block_device.0.volume_type: <computed> security_groups.#: <computed> source_dest_check: "true" subnet_id: <computed> tags.%: "2" tags.Name: "rock" tags.Source: "juxt/rock" tenancy: <computed> user_data: "8aed11c63960f4590c9cdd426593ecba98c4ab69" volume_tags.%: <computed> vpc_security_group_ids.#: <computed> + aws_security_group.allow_all id: <computed> arn: <computed> description: "Allow all inbound traffic." egress.#: <computed> ingress.#: "1" ingress.482069346.cidr_blocks.#: "1" ingress.482069346.cidr_blocks.0: "0.0.0.0/0" ingress.482069346.description: "" ingress.482069346.from_port: "0" ingress.482069346.ipv6_cidr_blocks.#: "0" ingress.482069346.protocol: "-1" ingress.482069346.security_groups.#: "0" ingress.482069346.self: "false" ingress.482069346.to_port: "0" name: "allow_all" owner_id: <computed> revoke_rules_on_delete: "false" tags.%: "2" tags.Name: "allow_all" tags.Source: "juxt/rock" vpc_id: <computed> Plan: 2 to add, 0 to change, 0 to destroy. ------------------------------------------------------------------------ This plan was saved to: proposed.plan To perform exactly these actions, run the following command to apply: terraform apply "proposed.plan"

jcf · 2018-04-11T16:46:28Z

terraform/rock.sh

+EOF
+
+systemctl daemon-reload
+systemctl start systemd-cloud-watch


Only logging for now. I've got to look at CodeDeploy once I've got journald forwarding nicely.

jcf · 2018-04-11T16:47:33Z

terraform/main.tf

+    name = "name"
+
+    # Maybe we should rename the AMI to `juxt-rock`?
+    values = ["juxt-arch-*"]


Maybe I should rename the AMIs to juxt-rock-*?

I used Arch because I was thinking about splitting the AMIs in two; one for a more-or-less empty Arch AMI with up-to-date everything and a custom repo for AUR packages, and one for the Clojure stuff we want installed…

I've completed the rename in 30be5f7.

AMIs will now be named using the format "juxt-rock-{git-sha}-{timestamp}".

As an alternative to the somewhat broken Go client this script is being used in production and as such we know it works. It isn't in any accessible version control so I've vendored it inside the package.

This prevents potential conflicts and permission issues.

jcf · 2018-04-12T15:09:01Z

I've added the Bash script used elsewhere in production to push journald logs over to CloudWatch, and packaged it up.

It's installed in the AMI I'm building presently (ami-bafcd9c3), and has an associated systemd unit file that you can activate like so:

systemctl start journald-cloud-watch-script

Internally that runs the provided Bash script, which can be found in /usr/bin/tail-journald.

I haven't had time to test out the script so can't say for sure it will work if you provision an EC2 instance via the included Terraform configuration, which can be done like so:

cd terraform
make plan
# Review the output to make sure it's correct.
make apply

Note, Terraform state is only stored locally, although pushing state to S3 is relatively straightforward should you wish to do so. I've destroyed everything via make destroy.

HTH!

jcf · 2018-04-12T15:14:52Z

rock.json

+            "source_ami": "ami-0b8ec472",
+            "instance_type": "t2.small",
+            "ssh_username": "root",
+            "ami_name": "juxt-rock-{{user `commit_ref`}}-{{timestamp}}"


To make the AMI public you need to add "ami_groups": ["all"].

jcf · 2018-04-12T15:16:43Z

scripts/install-custom.sh

+# Install
+install_pkgs \
+    codedeploy-agent \
+    journald-cloud-watch-script \


This is the Bash script that will forward logs as an alternative to the Go project that doesn't seem to work.

jcf · 2018-04-12T15:17:29Z

share/journald-cloud-watch-script/tail-journald

+
+INSTANCE_ID="${INSTANCE_ID:-$(curl -s http://169.254.169.254/latest/meta-data/instance-id)}"
+AWS_REGION="${AWS_REGION:-$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)}"
+LOG_GROUP_NAME="${LOG_GROUP_NAME:-rock}"


These env vars could be set in an environment file via user data, perhaps?

jcf · 2018-04-12T15:18:24Z

terraform/rock.sh

+# Looks like there's a but that results in a lot of log output from this library.
+#
+# Fix is in this PR: https://github.com/advantageous/systemd-cloud-watch/pull/16
+# systemctl start systemd-cloud-watch


This is the user data file that will be executed inside the EC2 instance when you provision with Terraform.

Starting services, adding configuration files goes here.

jcf self-assigned this Apr 11, 2018

jcf commented Apr 11, 2018

View reviewed changes

Add Terraform config to test out the AMI

d0764bb

jcf commented Apr 11, 2018

View reviewed changes

jcf added 4 commits April 11, 2018 17:49

Remove use of build group

e71a6fe

Replace mention of Arch in names with Rock

30be5f7

AMIs will now be named using the format "juxt-rock-{git-sha}-{timestamp}".

Use new AMI name and owner when Terraform-ing

f18b917

Deploy an EC2 instance using Rock

cc3a3b1

jcf mentioned this pull request Apr 12, 2018

Maybe swap out log forwarder #6

Open

jcf added 2 commits April 12, 2018 16:07

Add script to forward logs to CloudWatch

370df02

As an alternative to the somewhat broken Go client this script is being used in production and as such we know it works. It isn't in any accessible version control so I've vendored it inside the package.

Clean up src/pkg dirs before building a Packer image

e7239ad

This prevents potential conflicts and permission issues.

jcf changed the title ~~Install Clojure dependencies via a private local repo~~ Build an AMI Apr 12, 2018

jcf commented Apr 12, 2018

View reviewed changes

malcolmsparks merged commit af50d58 into master May 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build an AMI #4

Build an AMI #4

jcf commented Apr 11, 2018 •

edited

Loading

jcf commented Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018 •

edited

Loading

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 11, 2018

jcf Apr 12, 2018

jcf commented Apr 12, 2018 •

edited

Loading

jcf Apr 12, 2018

jcf Apr 12, 2018

jcf Apr 12, 2018

jcf Apr 12, 2018

Build an AMI #4

Build an AMI #4

Conversation

jcf commented Apr 11, 2018 • edited Loading

Tasks

jcf commented Apr 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcf Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcf commented Apr 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcf commented Apr 11, 2018 •

edited

Loading

jcf Apr 11, 2018 •

edited

Loading

jcf commented Apr 12, 2018 •

edited

Loading