AWS Developer – RDS

  • Provisioned IOPS and DB instance
  • Multi DB engines

RDS vs EC2 hosted

RDS

  • database as a service
  • w/o installing software
  • Amazon takes care of infrastructure, backups, and updates on the DB instance
  • the shell access to the underlying operating system is disabled.
  • log in as privilege user
  • cut off some features
  • Multi-AZ deployments

EC2 hosted

  • full control of you DB
  • various EC2 type to fit your requirement
  • manage multi-AZ DB Cluster yourself

Continue reading AWS Developer – RDS

AWS Developer – Route 53

R53 is global like IAM

Hosted Zones

Simple Policies

  • link the domain to the web server
  • example.com / the DNS name of the ELB (alias)

Weighted

  • 50 50 switch between EC2 instances
  • split traffic to multiple resources, 50% to vCurrent, 50% to beta

Latency

  • route based on latency (cross region)
  • route traffic to the resource that provides the best latency.

Failover

  • active-passive failover
  • health check the London ELB endpoint
  • 2 records: primary and secondary
  • e.g. set your primary in your region, and a DR (disaster recovery) site in another region

Geolocation

  • by continent or country (e.g. all Europe traffic go to EU London)
  • route based on geo location (cross region)
  • shift traffic from resources in one location to resources in another.

Continue reading AWS Developer – Route 53

AWS Developer – Shared Responsibility

Infrastructure Services

aws

    • foundation: compute, storage, db, networking
    • infrastructure: region, availability zone, edge location
    • e.g. hyper-v: aws patch and reboot

customers

    • IAM
    • customer data, platform, app management, os, network, firewall config
    • encryption: client side / server-side, network traffic protection: http/https

Continue reading AWS Developer – Shared Responsibility

AWS Developer – CloudFormation

turn infrastructure into configuration
set up the resources consistently and repeatedly over and over across multiple regions

  • provides a set of application bootstrapping scripts that enable you to install packages, files, and services on the EC2 instances by simply describing them in the CloudFormation template
  • automatic rollback on error is enabled by default, charge for errors. Rollback all the changes and terminate all the created services
  • syntax error failed at template validation. Since the stack will never start creating, there is nothing to roll back.
  • WaitCondition wait for resources to be provisioned
  • use Fn::GetAtt to output data, called Intrinsic Function
  • R53 supported, hosted zones / records creation
  • IAM role creation
  • default template format is json
  • free but pay for the resources it provisions, full root access
  • error occurs: rollback all resources created on failure
  • can’t nested templates

Template

  • architecture infrastructure diagram
  • json or yaml

Recourses List of recourses and associated configuration values, Mandatory

Resources:  # A list of AWS resources and their configuration values
            # CreationPolicy
            # DeletionPolicy
            # DependsOn
            # Metadata
            # UpdatePolicy

AWSTemplateFormatVersion: # An optional template file format version number

Parameters: # An optional list of template parameters (input values supplied at stack creation time)
            # AWS::AccountId
            # AWS::NotificationARNs
            # AWS::NoValue
            # AWS::Partition
            # AWS::Region
            # AWS::StackId
            # AWS::StackName
            # AWS::URLSuffix
Mappings:   # An optional list of data tables used to lookup static configuration values for e.g., AMI names per AZ

Conditions: # a production or test environment

Outputs:    # An optional list of output values like public IP address using the `Fn::GetAtt` function
            # Fn::Base64
            # Fn::Cidr
            # Condition Functions
            # Fn::FindInMap
            # Fn::GetAtt
            # Fn::GetAZs
            # Fn::ImportValue
            # Fn::Join
            # Fn::Select
            # Fn::Split
            # Fn::Sub

            # Python Helper Script
            # cfn-init
            # cfn-signal
            # cfn-get-metadata
            # cfn-hup

Stack

  • the resources that created
  • the end result of an architectural diagram

Status

  • CREATE_COMPLETE
  • UPDATE_IN_PROGRESS
  • UPDATE_COMPLETE
  • UPDATE_FAILED
  • UPDATE_IN_PROGRESS
  • UPDATE_ROLLBACK_COMPLETE
  • UPDATE_ROLLBACK_IN_PROGRESS
  • UPDATE_ROLLBACK_COMPLETE
  • DELETE_FAILED

Change Sets

  • Change Sets presents a summary of the proposed changes CloudFormation will make when a stack is updated
  • Change sets help check how the changes might impact running resources, especially critical resources, before implementing them
  • A stack goes into the UPDATE_ROLLBACK_FAILED state when AWS CloudFormation cannot roll back all changes during an update. Action Point: Continue Update Rollback

Access Control

IAM

  • IAM can be applied with CloudFormation to access control for users whether they can view stack templates, create stacks, or delete stacks
  • IAM permissions need to be provided to the user to the AWS services and resources provisioned, when the stack is created
  • Before a stack is created, AWS CloudFormation validates the template to check for IAM resources that it might create

Service Role

  • A service role is an AWS IAM role that allows AWS CloudFormation to make calls to resources in a stack on the user’s behalf
  • By default, AWS CloudFormation uses a temporary session that it generates from the user credentials for stack operations.
  • For a service role, AWS CloudFormation uses the role’s credentials.
  • When a service role is specified, AWS CloudFormation always uses that role for all operations that are performed on that stack.

Limits

  • 200 stacks per account
  • template description fields are limited to 4096 characters
  • up to 60 parameters and 60 outputs in a template.

References

AWS Developer – SWF

!! coordinate !! work across distributed components

Task = Activity Task and Worker = Activity Worker

Workflow

SWF uses deciders and workers to complete tasks

  • Domain as container, you register an activity in Amazon SWF, you provide a domain
  • Actors can be workflow starters, deciders, or activity workers
    • Starter user submit an order in the website
    • Workers to get task, process and return result
    • Decider to control the coordination of tasks
  • Tasks SWF interacts with activity workers and deciders by providing them with work assignments known as tasks. Data Exchange Between Actors
  • SWF stores tasks and assigns them to workers when they are ready, tracks their progress, and maintains their state, including details on their completion

Workflow

    • the automation of a business process
    • a set of activities that carry out some objective, together with logic that coordinates the activities.

Workflow Execution

  • a running instance of a workflow

Workflow History

  • the state and progress of each workflow execution in its Workflow History
  • your app is stateless as all state is stored in workflow history
  • Markers can be used to record information in the workflow history of a workflow execution

Workers and Deciders are both stateless
long polling: requests will be held open for up to 60 seconds if necessary, to reduce network traffic and unnecessary processing

Workflow Implementation & Execution

  1. Implement Activity workers with the processing steps in the Workflow.
  2. Implement Decider with the coordination logic of the Workflow.
  3. Register the Activities and workflow with SWF.
  4. Start the Activity workers and Decider. Once started, the decider and activity workers should start polling Amazon SWF for tasks.
  5. Start one or more executions of the Workflow. Each execution runs independently and can be provided with its own set of input data.
  6. When an execution is started, SWF schedules the initial decision task. In response, the decider begins generating decisions which initiate activity tasks. Execution continues until your decider makes a decision to close the execution.
  7. View and Track workflow executions

Limits

  • Max workflow can be 1yr

SWF maintains the execution state. It ensures a task is assigned only once and is never duplicated

SWF vs SQS

  • SWF: task-oriented vs SQS: message-oriented
  • SWF: task assigned only once and never duplicated vs message delivered in multiple times and in any order
  • SWF: track execution state vs SQS doesn’t

References

AWS Developer – SNS

Goals

  • Publishers communicate asynchronously with subscribers
  • decouple message publishers from subscribers
  • fan-out messages to multiple recipients at once
  • eliminate polling in your applications

Features

  • publish/subscribe, PUSH notification to a Topic
  • publish to HTTP, HTTPS, Email, Email-JSON, Amazon SQS, Application, AWS Lambda, SMS
  • publish in order, could result in out of order in subscriber side
  • subscriber: needs confirm the subscription or unsubscribe from a topic
  • pricing: pay as you go
  • TTL (undelivered messages will expire)
  • retry: attempts a retry after a failed delivery attempt, total retries / intervals
  • default retry: 3 times in the backoff phase, w/ 20 seconds delay between each retry
  • limits: up to 100 retries and max lifetime is 1hr

Fanout

Publisher -> SNS Topic -> SQS Queue -> EC2 Instances
                    |  -> SQS Queue -> EC2 Instances
                    |  -> SQS Queue -> EC2 Instances

For parallel asynchronous processing

  • e.g. new order, one queue for processing while the other sending to data warehouse
  • e.g. feed production data back to development env for testing on real data

Publish from VPC

SQS vs SNS

  • both messages services
  • SNS is PUSH
  • SQS is PULL

message can be customized for each protocol

References

AWS Developer – SQS

SQS

  • the 1st service in AWS
  • highly available distributed queue system
  • helps build distributed application with decoupled components
  • supports HTTP over SSL (HTTPS) and Transport Layer Security (TLS)
  • SQS – Fanning Out: Create an SNS topic. Then create and subscribe multiple SQS queues to the SNS topic

FIFO Queue

  • provide exactly-once processing
  • have a limited number of transactions per second (TPS)
  • must end with the .fifo suffix
  • support up to 300 messages per second

Standard Queue

  • Message Ordering, can be delivered in multiple times and in any order
  • At-Least-Once Delivery

Polling

  • WaitTimeSeconds parameter of a ReceiveMessage, between 1 and 20
  • Short Polling by default (set WaitTimeSeconds to 0)
  • Long Polling helps reduce the cost (add a message interval), set WaitTimeSeconds from 0 ~ 20

Limits

  • Message attributes: A message can contain up t 10 metadata attributes.
  • Message batch: A single message batch request can include a maximum of 10 messages
  • Message retention: By default, a message is retained for 4 days1 min~14 day, MessageRetentionPeriod
  • Message size: 256 KB, contains a reference to a message payload in Amazon S3
  • Message visibility timeout: default: 30 seconds. The maximum is 12 hours, ChangeMessageVisibility. The visibility timeout is the time during which the message is invisible to workers. If this interval is set to “0”, the message will be immediately available for processing.

References