I remember when DevOps was a new idea and a colleague said, "DevOps separates out the real developers." I agreed with him at the time that most developers do not have the skills to practice DevOps. This was over ten years ago, and development teams have gotten more sophisticated, but I also think that teams need more education on how to practice DevOps and SecDevOps.
I'm starting an article series on SecDevOps (which is like DevOps, but with security). I'll be touching on AWS CodeBuild, AWS CodePipeline, and finally integrating into applications like GitHub and OWASP.
What is Cloud Formation?
CloudFormation is a declarative language for specifying AWS services and you can configure any AWS application or resource using CloudFormation. From the point of Operations as code, CloudFormation is the gospel.
Getting Started with AWS CodeBuild for Serverless
AWS CodeBuild is a fully managed continuous integration service for compiling, deploying, and testing code as part of a development pipeline. You can use CloudFormation to create the build pipeline using declarative YAML code.
AWS documents the format and I provide the example to jump-start the impatient. The service defined creates a build environment that will compile and deploy the source code you have provided in the file s3://my-code-build/MyCode.zip.
In my next installment, I will describe how to replace the MyCode.zip bundle with a hook to GitHub, but I want to start simply because the technology is complex.
The MyCode.zip contains an API Gateway/Lambda serverless stack. It takes three files: buildspec.yml, src/lambda/mycode.py, and src/lambda/template.yml and zips them up. This is a toy and would otherwise be the contents of a pull request, but again: simplicity.
rm -f MyCode.zip
zip MyCode.zip buildspec.yml src/lambda/mycode.py src/lambda/template.yml
aws s3 sync . s3://my-code-build --exclude="*" --include "MyCode.zip"
So for cut and paste's sake, here are the files.
The buildspec.yml contains an analogous command sequence to "make" or "npm" and CodeBuild will execute the commands in this file to build the serverless stack.
# Buildspec Reference Doc: https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html
version: 0.2
phases:
install:
commands:
- echo "[+] Scanning for security vulnerability within dependencies"
# Setup dependency checking
# See: https://aws.amazon.com/blogs/apn/how-to-bake-open-source-security-into-your-aws-codebuild-pipeline/
pre_build:
commands:
- echo "[+] Setting up any variables or dependencies..."
build:
commands:
- echo "[+] Installing dependencies...."
- echo "Running Cloud Formation scripts on `date` in `pwd`"
- aws cloudformation package --template-file src/lambda/template.yml --s3-bucket my-code-deploy --output-template-file packaged-template.yaml
- aws cloudformation deploy --template-file packaged-template.yaml --stack-name MyCode --capabilities CAPABILITY_NAMED_IAM
post_build:
commands:
- echo "Build completed on `date`"
artifacts:
files:
- packaged-template.yaml
discard-paths: yes
The src/lambda/mycode.py is a serverless Lambda function that executes whenever the API event triggers it.
import json
def get_status(event, context):
result = {"hello": "Mr Tim"}
return {
"statusCode": 200,
"body": json.dumps(result)
}
The src/lambda/template.yml is a cloud formation template that builds a service API Gateway stack so that we can expose hello world as a web service https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/status/.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
Monitoring function for the Owlmtn and Azrius Pipeline
Globals:
Function:
Timeout: 5
Resources:
MyCodeFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: .
Handler: MyCode.get_status
Runtime: python3.7
Events:
Status:
Type: Api
Properties:
Path: /status
Method: get
Outputs:
MyCodeApi:
Description: "API Gateway endpoint URL for monitoring function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/status/"
MyCodeFunction:
Description: "MyCode Lambda Function ARN"
Value: !GetAtt MyCodeFunction.Arn
MyCodeFunctionIamRole:
Description: "Implicit IAM Role created for MyCode"
Value: !GetAtt MyCodeFunctionRole.Arn
After building the template, examine the Outputs section in CloudFormation to understand what is happening.
The Buckets used in the CodeBuild Pipeline
The CodeBuild Template will take two S3 buckets. One the build bucket of s3://my-code-build and the deploy bucket s3://my-code-deploy. In a separate Cloud Formation template, I create these buckets as private. Here you can specify the buckets as parameters and reuse this CloudFormation template "as-is."
AWSTemplateFormatVersion: "2010-09-09"
Description: >
Code Pipeline YAML to build S3 Buckets.
From https://s3.amazonaws.com/cloudformation-examples/user-guide/continuous-deployment/basic-pipeline.yml
Parameters:
S3DeployBucket:
Description: The name of the S3 bucket that contains the source artifact, which must be in the same region as this stack
Type: String
S3BuildBucket:
Description: The name of the S3 bucket that contains the source to checkout for build
Type: String
Resources:
ArtifactStoreBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref S3DeployBucket
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
IgnorePublicAcls: true
BlockPublicPolicy: true
RestrictPublicBuckets: true
ArtifactBuildBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref S3BuildBucket
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
IgnorePublicAcls: true
BlockPublicPolicy: true
RestrictPublicBuckets: true
Outputs:
ArtifactStoreBucket:
Description: The bucket containing the deployment content
Value: !Ref ArtifactStoreBucket
ArtifactBuildBucket:
Description: The bucket containing the build content
Value: !Ref ArtifactStoreBucket
The CodeBuild Cloud Formation Template
Putting it all together, finally, we have the cloud formation template and I will highlight the import sections.
The definition of the CodeBuild project.
MyCodeBuildProject:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Location: "my-code-deploy"
Name: MyCode
NamespaceType: NONE
Path: /hello_world
Type: S3
Description: Simple Hello World Lambda API
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
Type: LINUX_CONTAINER
Name: my-code-build-project
ServiceRole: !Ref 'CodeBuildRole'
Source:
BuildSpec: "buildspec.yml"
Location: "my-code-build/MyCode.zip"
Type: S3
Tags:
- Key: name
Value: "my-code"
The role needed of the CodeBuild to build with CloudFormation.
CodeBuildRole:
Type: AWS::IAM::Role
Description: Role for Standard Code Build
Properties:
RoleName: !Join
- '-'
- - !Ref 'AWS::StackName'
- CodeBuild
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Policies:
- PolicyName: CodeBuildReadOnlyPolicy-Monitoring
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- 's3:ListBucket'
- 's3:GetBucketAcl'
- 's3:GetBucketLocation'
Effect: Allow
Resource:
- 'arn:aws:s3:::my-code-build'
- Action:
- 's3:GetObject'
- 's3:GetObjectVersion'
Effect: Allow
Resource:
- 'arn:aws:s3:::my-code-build/*'
- PolicyName: CodeBuildBasePolicy-Monitoring
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Effect: Allow
Resource: '*'
- Action:
- "s3:PutObject"
- "s3:GetObject"
- "s3:GetObjectVersion"
- "s3:GetBucketAcl"
- "s3:GetBucketLocation"
Effect: Allow
Resource:
- 'arn:aws:s3:::my-code-deploy'
- 'arn:aws:s3:::my-code-deploy/*'
- Action:
- "cloudformation:*"
Effect: Allow
Resource: '*'
- Action:
- 'iam:GetRole'
- 'lambda:*'
Effect: Allow
Resource: '*'
Putting all of these snippets together in Cloud Formation you will run the build and it will create a project that can be used to automate your code build process.
aws cloudformation deploy \
--template-file src/delivery/packaged-codePipeline.yaml \
--stack-name MyCodePipeline \
--parameter-overrides PipelineName="my-code-pipeline" \
S3BuildBucket="my-code-build" S3DeployBucket="my-code-deploy" \
--capabilities CAPABILITY_NAMED_IAM
Conclusion
Integrate automation into everything you do - whether it is building code deployment pipelines, building web stacks, or testing. It won't be easy at first and I recommend you use the console to get a feel for what is created; however, don't rely on the console and automate everything. I provide these snippets not for you to copy blindly, but rather to understand and build your own. I hope they help.
Comments