タケユー・ウェブ日報

Ruby on Rails や Flutter といったWeb・モバイルアプリ技術を武器にお客様のビジネス立ち上げを支援する、タケユー・ウェブ株式会社の技術ブログです。

CloudFormation で CloudWatch Events + Lambda による定期実行タスクを作成する

AWSで定期的な処理を行いたいときは、CloudWatch Events を使い、 Lambda ファンクションの実行をスケジューリングすることで行います。

docs.aws.amazon.com

設定画面

f:id:uzuki05:20200501172633p:plain
CloudWatch Events のルール

f:id:uzuki05:20200501172808p:plain
Lambdaファンクションの実行権限設定

CloudFormation テンプレート

f:id:uzuki05:20200501163233p:plain

Parameters:
  ScheduleExpression:
    Type: String
  Enabled:
    Type: String
    Default: "false"
    AllowedValues:
      - "true"
      - "false"
Conditions:
  isEnabled: !Equals [!Ref Enabled, "true"]
Resources:
  # 実行したい Lambda Function
  Function:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |+
          exports.handler = (event, context, callback) => {
            console.log('LogScheduledEvent');
            console.log('Received event:', JSON.stringify(event, null, 2));
            callback(null, 'Finished');
          };
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Runtime: "nodejs10.x"
      MemorySize: 128
      Timeout: 60

  # ScheduleExpression で指定したスケジュールで、 Function を実行する
  Rule:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: !Ref ScheduleExpression
      State: !If [isEnabled, "ENABLED", "DISABLED"]
      Targets:
        - Id: Lambda
          Arn: !GetAtt Function.Arn

  # スケジュールイベントからのLambda実行を許可
  LambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref Function
      Principal: events.amazonaws.com
      SourceArn: !GetAtt Rule.Arn
  
  # Lambda の実行ロール
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - "sts:AssumeRole"
      Path: /
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

DockerCompose + PostgreSQL Replication

  • Railsの複数データベース機能を開発環境でも有効にしたかった
  • docker-compose up でセットアップから起動まで一発で動くようにしたい

Docker Compose

docker-compose.yml

version: "3"
volumes:
  pg_primary_data:
    driver: local
  pg_readonly_data:
    driver: local
services:
  pg_primary:
    build:
      context: ./docker/pg
      dockerfile: Dockerfile.primary
    command: postgres -c log_destination=stderr -c log_statement=all
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - pg_primary_data:/var/lib/postgresql/data
  pg_readonly:
    build:
      context: ./docker/pg
      dockerfile: Dockerfile.readonly
    command: postgres -c log_destination=stderr -c log_statement=all
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - pg_readonly_data:/var/lib/postgresql/data
    depends_on:
      - pg_primary

docker/pg/Dockerfile.primary

FROM postgres:11-alpine

ENV LANG C.UTF-8

COPY ./setup-primary.sh /docker-entrypoint-initdb.d/setup-primary.sh
RUN chmod 0666 /docker-entrypoint-initdb.d/setup-primary.sh

docker/pg/setup-primary.sh

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE ROLE replication_user LOGIN REPLICATION PASSWORD 'replicationpassword';
EOSQL

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
SELECT * FROM pg_create_physical_replication_slot('node_a_slot');
EOSQL

mkdir $PGDATA/archive

cat >> "$PGDATA/postgresql.conf" <<EOF
wal_level = hot_standby
max_wal_senders = 10
max_replication_slots = 10
synchronous_commit = off
EOF

echo "host replication replication_user 0.0.0.0/0 md5" >> "$PGDATA/pg_hba.conf"

docker/pg/Dockerfile.readonly

FROM postgres:11-alpine

ENV LANG C.UTF-8
ENV ENTRYKIT_VERSION 0.4.0

RUN wget https://github.com/progrium/entrykit/releases/download/v${ENTRYKIT_VERSION}/entrykit_${ENTRYKIT_VERSION}_Linux_x86_64.tgz \
  && tar -xvzf entrykit_${ENTRYKIT_VERSION}_Linux_x86_64.tgz \
  && rm entrykit_${ENTRYKIT_VERSION}_Linux_x86_64.tgz \
  && mv entrykit /bin/entrykit \
  && chmod +x /bin/entrykit \
  && entrykit --symlink

# ログ転送にscpを使用する
RUN apk --update add openssh-client && rm -rf /var/cache/apk/*

COPY ./setup-readonly.sh /setup-readonly.sh
RUN chmod +x /setup-readonly.sh

# https://github.com/docker-library/postgres/blob/primary/11/Dockerfile
ENTRYPOINT [ \
    "prehook", \
        "/setup-readonly.sh", \
        "--", \
    "docker-entrypoint.sh" \
]
CMD ["postgres"]

docker/pg/setup-primary.sh

#!/bin/bash
set -e

if [ ! -s "$PGDATA/PG_VERSION" ]; then
    echo "*:*:*:replication_user:replicationpassword" > ~/.pgpass
    chmod 0600 ~/.pgpass
    until ping -c 1 -W 1 pg_primary
    do
        echo "Waiting for primary to ping..."
        sleep 1s
    done

    until pg_basebackup -h pg_primary -D ${PGDATA} -U replication_user -vP -W
    do
        echo "Waiting for primary to connect..."
        sleep 1s
    done

    sed -i 's/wal_level = hot_standby/wal_level = replica/g' ${PGDATA}/postgresql.conf

    cat > ${PGDATA}/recovery.conf <<EOF
standby_mode = on
primary_conninfo = 'host=pg_primary port=5432 user=replication_user password=replicationpassword application_name=pg_readonly'
primary_slot_name = 'node_a_slot'
EOF

    chown postgres:postgres ${PGDATA} -R
    chmod 700 ${PGDATA} -R
fi

Rails で使う

https://github.com/takeyuweb/rails6-multidb-sample

docker-compose.yml

  app:
    build:
      context: ./docker/app
      dockerfile: Dockerfile.dev
    environment:
      DATABASE_HOST_PRIMARY: pg_primary
      DATABASE_HOST_READONLY: pg_readonly
      DATABASE_USER: postgres
      DATABASE_PASSWORD: password
    user: ruby
    command: ["bundle", "exec", "rails", "s", "-p", "3000", "-b", "0.0.0.0"]
    volumes:
      - .:/src
    ports:
      - 3000:3000
    tty: true
    stdin_open: true
    depends_on:
      - pg_primary
      - pg_readonly

config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode
  # For details on connection pooling, see Rails configuration guide
  # https://guides.rubyonrails.org/configuring.html#database-pooling
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

development:
  # 仮にこれを `hoge:` にすると `schema.rb` は `hoge_schema.rb` になる。
  primary:
    <<: *default
    database: MyApp_development
    host: <%= ENV.fetch('DATABASE_HOST_PRIMARY') { 'db' } %>
    username: <%= ENV.fetch('DATABASE_USER') { 'postgres' } %>
    password: <%= ENV.fetch('DATABASE_PASSWORD') { 'password' } %>
  primary_readonly:
    <<: *default
    database: MyApp_development
    host: <%= ENV.fetch('DATABASE_HOST_READONLY') { 'db' } %>
    username: <%= ENV.fetch('DATABASE_USER') { 'postgres' } %>
    password: <%= ENV.fetch('DATABASE_PASSWORD') { 'password' } %>
    replica: true

test:
  primary:
    <<: *default
    database: MyApp_test
    host: <%= ENV.fetch('DATABASE_HOST_PRIMARY') { 'db' } %>
    username: <%= ENV.fetch('DATABASE_USER') { 'postgres' } %>
    password: <%= ENV.fetch('DATABASE_PASSWORD') { 'password' } %>
  primary_readonly:
    <<: *default
    database: MyApp_test
    host: <%= ENV.fetch('DATABASE_HOST_READONLY') { 'db' } %>
    username: <%= ENV.fetch('DATABASE_USER') { 'postgres' } %>
    password: <%= ENV.fetch('DATABASE_PASSWORD') { 'password' } %>
    replica: true

app/models/application_record.rb

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true
  connects_to database: { writing: :primary, reading: :primary_readonly }
end

Ruby で Firebase User を作成

動機

  • Firebase Auth を使ったWebアプリの Rails System Spec を書きたい
  • フロントエンドで Firebase JavaScript SDK を使っているため、バックエンドでのstubでは書けない

サンプルコード(一部抜粋)

  • Google::Apis::IdentitytoolkitV3::IdentityToolkitService Google::Apis::IdentitytoolkitV3::SignupNewUserRequest で作成できる
  • メールアドレスは実際に受信するなら MailSlurp あたりを使って用意する
require 'rails_helper'
require 'google/apis/identitytoolkit_v3'

RSpec.feature 'login', type: :system, js: true do
  def sign_in(email, password)
    visit root_path
    within(:css, '#login_modal') do
      fill_in('login_email', with: email)
      fill_in('login_password', with: password)
      find_button('Login', wait: 10).click  # Firebase JavaScript SDK によるメールアドレス認証
    end
    expect(page).to have_button('Logout', wait: 10)
  end

  def sign_out
    find_button('Logout', wait: 10).click
  end

  before do
    # Create Firebase User
    # https://qiita.com/asflash8/items/17775895e35272ae7ec8
    # https://firebase.google.com/docs/auth/admin/manage-users?hl=ja
    firebase_service_account_json_key = Rails.application.credentials.dig(:raw, :firebase_service_account_json_key)
    @identity_toolkit_service = Google::Apis::IdentitytoolkitV3::IdentityToolkitService.new
    @identity_toolkit_service.authorization = Google::Auth::ServiceAccountCredentials.make_creds(
      json_key_io: StringIO.new(firebase_service_account_json_key),
      scope: 'https://www.googleapis.com/auth/identitytoolkit'
    )
    user_data = {
      email: 'test@takeyuweb.co.jp',
      email_verified: true,
      password: 'password1234',
      disabled: false
    }
    request = Google::Apis::IdentitytoolkitV3::SignupNewUserRequest.new(user_data)
    @account = @identity_toolkit_service.signup_new_user(request)

    # Signin
    sign_in(user_data.fetch(:email), user_data.fetch(:password))

    @current_user = User.find_by(email: user_data.fetch(:email))
    @password = user_data.fetch(:password)
  end

  after do
    if @account
      request = Google::Apis::IdentitytoolkitV3::DeleteAccountRequest.new(local_id: @account.local_id)
      @identity_toolkit_service.delete_account(request)
    end
  rescue Google::Apis::ClientError => e
    Rails.logger.error(e.message)
  end

  scenario "edit profile" do
    visit root_path
    find_link('Profile').click

    expect(page).to have_field('Name')
    expect(page).to have_field('Telephone')

    fill_in 'Name', with: 'Yuichi Takeuchi'
    fill_in 'Telephone', with: '81487003094'

    # (snip)
  end
end

参考

qiita.com

firebase.google.com

rspec で Firebase ID Token を stub

class SessionsController < ApplicationController
  def create
    id_token = params.required(:id_token)
    user = User.from_firebase(id_token)
    # (snip)
  end
end

class User < ApplicationRecord

  CIRTIFICATE_URL = 'https://www.googleapis.com/robot/v1/metadata/x509/securetoken@system.gserviceaccount.com'
  EXP_LEEWAY = 30.seconds

  def self.from_firebase(id_token)
    firebase_project_id = Rails.application.credentials.dig(:firebase, :project_id)
    valid_iss = "https://securetoken.google.com/#{firebase_project_id}"

    # https://firebase.google.com/docs/auth/admin/verify-id-tokens?hl=ja
    a, decoded_token_header = JWT.decode(id_token, nil, false)

    certificate = Rails.cache.fetch('firebase_securetoken_cirtificate', expires: 1.hour) do
      uri = URI.parse(CIRTIFICATE_URL)
      JSON.parse(Net::HTTP.get_response(uri).body).fetch(decoded_token_header["kid"])
    end

    public_key = OpenSSL::X509::Certificate.new(certificate).public_key
    decoded_token_payload, _ = JWT.decode(
      id_token,
      public_key,
      true,
      exp_leeway: EXP_LEEWAY,   # 有効期限の検証をするが、ゆるめに。 EXP_LEEWAY 秒は大目に見る。
      verify_iat: true,         # 発行時の検証をする
      aud: firebase_project_id,
      verify_aud: true,         # 対象の検証をする
      iss: valid_iss,
      verify_iss: true,         # 発行元の検証をする
      verify_sub: true,         # 件名の存在を検証する
      algorithm: decoded_token_header["alg"]
    )

    Rails.logger.debug("decoded_token_payload: #{decoded_token_payload.inspect}")

    # decoded_token_payload = {
    #   "name"=>"Takeuchi Yuichi",
    #   "picture"=>"https://lh3.googleusercontent.com/a-/AAuE7mAZU7Rh7lIFStzfWGe3tC24qDIX4UIoEWR8426flA",
    #   "iss"=>"https://securetoken.google.com/rails-firebase-sample",
    #   "aud"=>"rails-firebase-sample",
    #   "auth_time"=>1580712233,
    #   "user_id"=>"Qgk3sd1HgoPLVbSy8uXAWnRmWmx1",
    #   "sub"=>"Qgk3sd1HgoPLVbSy8uXAWnRmWmx1",
    #   "iat"=>1580712233,
    #   "exp"=>1580715833,
    #   "email"=>"takeyuweb@gmail.com",
    #   "email_verified"=>true,
    #   "firebase"=>{
    #     "identities"=>{
    #       "google.com"=>["100008179958237311525"],
    #       "email"=>["takeyuweb@gmail.com"]
    #     },
    #     "sign_in_provider"=>"google.com"
    #   }
    # }

    where(uid: decoded_token_payload.fetch("sub")).first_or_create
  end
end
require 'rails_helper'

RSpec.describe "/session", type: :request do
  describe 'create session' do
    context "with valid id_token" do
      it "returns Created" do
        stub_id_token do |id_token|
          post "/session", params: { id_token: id_token }
          expect(response).to have_http_status(:created)
        end
      end
    end

    context "with invalid id_token" do
      it "returns Unauthorized " do
        stub_id_token("exp" => 0) do |id_token|
          post "/session", params: { id_token: id_token }
          expect(response).to have_http_status(:unauthorized)
        end
      end
  end
end
# spec/support/firebase.rb

module FirebaseIdTokenGenerator
  def stub_id_token(override = {}, &block)
    unless defined?(@pkey)
      @pkey, @cert = generate_key_pair
      @kid = 'thekeyid'

      certificates = {
        'dummy' => generate_key_pair[1].to_pem,
        @kid => @cert.to_pem
      }
      WebMock.stub_request(:get, User::CIRTIFICATE_URL).to_return(status: 200, body: certificates.to_json)
    end

    block.call(get_id_token(generate_payload(override)))
  end

  def get_id_token(payload)
    JWT.encode(payload, @pkey, 'RS256', { kid: @kid, typ: 'JWT' })
  end

  def generate_payload(override = {})
    # https://firebase.google.com/docs/auth/admin/verify-id-tokens?hl=ja
    firebase_project_id = Rails.application.credentials.dig(:firebase, :project_id)
    {
      "name"=>"Takeuchi Yuichi",
      "picture"=>"https://test.host/picture.jpeg",
      "iss" => "https://securetoken.google.com/#{firebase_project_id}",
      "aud" => firebase_project_id,
      "auth_time"=> Time.current.to_i,
      "user_id"=>"theuserid",
      "sub"=>"theuserid",
      "iat"=> 1.hour.ago.to_i,
      "exp"=> 1.hour.from_now.to_i,
      "email"=>"yuichi.takeuchi@takeyuweb.co.jp",
      "email_verified"=>true,
      "firebase"=>{
        "identities"=>{
          "google.com"=>["000000000000000000000"],
          "email"=>["yuichi.takeuchi@takeyuweb.co.jp"]
        },
        "sign_in_provider"=>"google.com"
      }
    }.deep_merge(override)
  end

  def generate_key_pair
    ca_passphrase = SecureRandom.alphanumeric
    digest = OpenSSL::Digest::SHA1.new

    issu = OpenSSL::X509::Name.new
    issu.add_entry('C' , 'JP')
    issu.add_entry('ST', 'Saitama')
    issu.add_entry('DC', 'Omiya-ku')
    issu.add_entry('O' , 'TakeyuWeb, Inc.')
    issu.add_entry('CN', 'MexiCasita Test CA')

    issu_rsa = OpenSSL::PKey::RSA.generate(2048)

    issu_cer = OpenSSL::X509::Certificate.new
    issu_cer.not_before = Time.current
    issu_cer.not_after  = 10.years.from_now
    issu_cer.public_key = issu_rsa.public_key
    issu_cer.serial  = 1
    issu_cer.issuer  = issu
    issu_cer.subject = issu
    ex = OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(true)]))
    issu_cer.add_extension(ex)
    issu_cer.sign(issu_rsa, digest)

    return [issu_rsa, issu_cer]
  end
end

RSpec.configure do |config|
  config.include FirebaseIdTokenGenerator

  config.after(:all, type: :request) do
    remove_instance_variable(:@pkey) if defined?(@pkey)
  end
end

rspec で ActionMailer の deliver_later のジョブはすぐに処理する

ActiveJob へのキューインの spec で示したように ActiveJob::Base.queue_adapter = :test を使うと、 perform_later のジョブはキューに溜まるようになる。

しかし、 ActionMailer の deliver_later で非同期送信を使っている場合にまでキューインされてしまうと、 「メールを送信する」ことのテストを書きづらいことがある。

まとめ

  • ActiveJob::TestHelper に含まれる perform_enqueued_jobs を使うと、ブロック内ではエンキューされたジョブはすぐに実行される
  • only: ActionMailer::MailDeliveryJob と指定すれば、メール送信のみすぐに実行できる

サンプルコード

include ActiveJob::TestHelper

it "It sends the server key" do
  expect {
    perform_enqueued_jobs(only: ActionMailer::MailDeliveryJob) { NotifyJob.perform_now(user) }
  }.to change(ActionMailer::Base.deliveries, :count).by(1)
  
  mail = ActionMailer::Base.deliveries.last
  expect(mail.attachments.map(&:filename)).to match_array(["key.pem"])
end

ActiveJob::TestHelper

api.rubyonrails.org

ActiveJob へのキューインの spec

まとめ

  • ActiveJob::Base.queue_adapter = :test を使うと、 ActiveJob::Base.queue_adapter.enqueued_jobs にジョブが入る
  • have_enqueued_job マッチャーでジョブが入っていることを expect する

サンプルコード

it 'enqueues PaymentIntentSuccessedJob' do
  ActiveJob::Base.queue_adapter = :test

  event = StripeMock.mock_webhook_event('payment_intent.succeeded', customer: customer.id)
  expect {
    post "/webhook", params: event.to_h, as: :json
  }.to have_enqueued_job(PaymentIntentSuccessedJob).with(event.id)
end

aws-sdk-ruby で AWS CodeCommit の SSH キーを登録する

まとめ

  • SSHキーペア生成は OpenSSL::PKey::RSA#generate を使う
  • アップロードは Aws::IAM::Client#upload_ssh_public_key を使う

事の起こり

開発中のWebシステムでは、必要に応じてAWS上のリソースを生成しています。 CodeCommitへのコミットに使用するIAMユーザを作成、Git ssh接続用のキーペアを生成し、秘密鍵を使って接続できるようにする一連の作業を自動化することにしました。

実施内容

SSHキーペア生成

必要な鍵の仕様について、ドキュメントによれば

The public key must be encoded in ssh-rsa format or PEM format. The minimum bit-length of the public key is 2048 bits, and the maximum length is 16384 bits.

ということで、

  • 最低2048ビット
  • OpenSSH形式かPEM形式の鍵

を作ればよいようです。

docs.aws.amazon.com

PEM形式であればRuby標準のライブラリだけで生成できます。

docs.ruby-lang.org

rsa = OpenSSL::PKey::RSA.generate(2048)
public_key_body = rsa.public_key.to_pem
private_key_body = rsa.to_pem

なお、OpenSSH形式の場合は net-ssh gem が必要です。

stackoverflow.com

公開鍵アップロード

CodeCommitで使うものですが、IAMのSDKを使います。

iam = Aws::IAM::Client.new(region: aws_region, credentials: credentials)
ssh_public_key = iam.upload_ssh_public_key(
  user_name: user_name,  # IAM ユーザー名
  ssh_public_key_body: public_key_body,
).ssh_public_key

ssh_public_key_id = ssh_public_key.ssh_public_key_id # => "APKXXXXXXXXXXXXXXXXX"
File.open("#{ssh_public_key_id}.pem", "wb") do |file|
  file.write private_key_body
end
鍵を使う

sshユーザー名は Aws::IAM::Client#upload_ssh_public_key の応答に含まれる ssh_public_key_id を使います。

こんな .ssh/config を使うとよいでしょう。

Host user_name.codecommit
      HostName git-codecommit.ap-northeast-1.amazonaws.com
      User APKXXXXXXXXXXXXXXXXX
      Port 22
      IdentityFile ~/.ssh/APKXXXXXXXXXXXXXXXXX.pem
      IdentitiesOnly yes
$ git clone ssh://user_name.codecommit/v1/repos/repository-name