Skip to content
Snippets Groups Projects
Unverified Commit 9fcd0628 authored by v-yussupov's avatar v-yussupov Committed by GitHub
Browse files

Merge pull request #107 from sydkbc/master

Scaling Feature Updates: New AWS Scalable Nodetype defined. Interfaces and monitoring  relationships updated.
parents 4e85c258 ac633a07
No related branches found
No related tags found
No related merge requests found
Showing
with 487 additions and 47 deletions
tosca_definitions_version: tosca_simple_yaml_1_3
interface_types:
radon.interfaces.scaling.ScaleUp:
derived_from: tosca.interfaces.Root
operations:
scale_up:
description: Interface to trigger the scale up action
\ No newline at end of file
- name: create
hosts: localhost
gather_facts: true
hosts: all
gather_facts: yes
remote_user: "{{ ssh_user }}"
vars:
ansible_ssh_private_key_file: "{{ ssh_key_file }}"
......@@ -15,11 +15,6 @@
vpc_subnet_id: "{{ vpc_subnet_id }}"
assign_public_ip: true
register: ec2
- name: wait for instance to come up
delegate_to: "{{ ec2.instances[0].public_dns_name }}"
wait_for_connection:
delay: 60
timeout: 320
- name: set outputs
set_stats:
data:
......@@ -27,3 +22,23 @@
public_ip: "{{ ec2.instances[0].public_ip }}"
private_address: "{{ ec2.instances[0].private_dns_name }}"
id: "{{ ec2.instance_ids[0] }}"
- name: Create temporary inventory for ssh wait
add_host:
name: server
groups: vms
ansible_host: "{{ ec2.instances[0].public_dns_name }}"
ansible_user: "{{ ssh_user }}"
ansible_ssh_private_key_file: "{{ ssh_key_file }}"
ansible_ssh_common_args: >
-o IdentitiesOnly=yes
-o BatchMode=yes
-o UserKnownHostsFile=/dev/null
-o StrictHostKeyChecking=no
- hosts: vms
gather_facts: no
tasks:
- name: Wait for ssh on EC2 instance to come up
wait_for_connection:
delay: 60
sleep: 30
timeout: 320
......@@ -5,55 +5,59 @@
A node type that represents an AWS Lambda function which is instrumented based on a function deployment package from an AWS S3 bucket.
| Name | URI | Version | Derived From |
|:---- |:--- |:------- |:------------ |
| `AwsLambdaFunctionFromS3` | `radon.nodes.aws.AwsLambdaFunctionFromS3` | 1.0.0 | `radon.nodes.abstract.Function` |
| Name | URI | Version | Derived From |
| :------------------------ | :---------------------------------------- | :------ | :------------------------------ |
| `AwsLambdaFunctionFromS3` | `radon.nodes.aws.AwsLambdaFunctionFromS3` | 1.0.0 | `radon.nodes.abstract.Function` |
### Attributes
| Name | Type | Default Value | Description |
|:---- |:---- |:------------- |:----------- |
| `arn` | `string` | | Amazon Resource Name (ARN) for this function |
| `role_arn` | `string` | | ARN for the role of the function |
| `region` | `string` | | AWS region in which the function is deployed |
| Name | Type | Default Value | Description |
| :---------------- | :-------- | :------------ | :----------------------------------------------------- |
| `arn` | `string` | | Amazon Resource Name (ARN) for this function |
| `role_arn` | `string` | | ARN for the role of the function |
| `region` | `string` | | AWS region in which the function is deployed |
| `cpu_upper_bound` | `float` | | CPU upper alerting threshold |
| `ram_upper_bound` | `float` | | RAM upper alerting threshold |
| `adjustment` | `integer` | | Scaling adjustment |
| `callbackUrlCPU` | `string` | | Callback URL that handles the scaling triggering event |
### Properties
| Name | Required | Type | Constraint | Default Value | Description |
|:---- |:-------- |:---- |:---------- |:------------- |:----------- |
| `alias` | `true` | `string` | | "dev" | Lambda function's alias |
| `runtime` | `true` | `string` | `valid_values: [nodejs, nodejs4.3, nodejs6.10, nodejs8.10, nodejs10.x, java8, python2.7, python3.6, python3.7, dotnetcore1.0, dotnetcore2.0, dotnetcore2.1, nodejs4.3-edge, go1.x, ruby2.5]` | "nodejs" | The identifier of the function's runtime |
| `handler` | `true` | `string` | | "index.handler" | The name of the method within your code that Lambda calls to execute your function |
| `memory` | `true` | `integer` | `in_range: [128, 3008]` | 128 | The amount of memory in megabytes that your function has access to |
| `memory_range` | `true` | `range` | `in_range: [128, 3008]` | | Range of function memory in MB to search |
| `concurrency` | `false` | `integer` | `in_range: [0, UNBOUNDED]` | 0 | The amount of concurrency that your function has access to |
| `concurrency_range` | `false` | `range` | `in_range: [1, UNBOUNDED]` | | Range of function concurrency to search |
| `timeout` | `true` | `integer` | `in_range: [1, 900]` | 3 | The amount of time that Lambda allows a function to run before stopping it |
| `statement_id` | `true` | `string` | | | Lambda policy statement identifier |
| `env_vars` | `false` | `map` of `string` | | | The environment variables of the function |
| `s3_bucket_name` | `true` | `string` | | | The S3 bucket name containing the function package to be deployed |
| `s3_bucket_key` | `true` | `string` | | | The S3 key aka. filename referencing the file to be deployed |
| Name | Required | Type | Constraint | Default Value | Description |
| :------------------ | :------- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------- | :--------------------------------------------------------------------------------- |
| `alias` | `true` | `string` | | "dev" | Lambda function's alias |
| `runtime` | `true` | `string` | `valid_values: [nodejs, nodejs4.3, nodejs6.10, nodejs8.10, nodejs10.x, java8, python2.7, python3.6, python3.7, dotnetcore1.0, dotnetcore2.0, dotnetcore2.1, nodejs4.3-edge, go1.x, ruby2.5]` | "nodejs" | The identifier of the function's runtime |
| `handler` | `true` | `string` | | "index.handler" | The name of the method within your code that Lambda calls to execute your function |
| `memory` | `true` | `integer` | `in_range: [128, 3008]` | 128 | The amount of memory in megabytes that your function has access to |
| `memory_range` | `true` | `range` | `in_range: [128, 3008]` | | Range of function memory in MB to search |
| `concurrency` | `false` | `integer` | `in_range: [0, UNBOUNDED]` | 0 | The amount of concurrency that your function has access to |
| `concurrency_range` | `false` | `range` | `in_range: [1, UNBOUNDED]` | | Range of function concurrency to search |
| `timeout` | `true` | `integer` | `in_range: [1, 900]` | 3 | The amount of time that Lambda allows a function to run before stopping it |
| `statement_id` | `true` | `string` | | | Lambda policy statement identifier |
| `env_vars` | `false` | `map` of `string` | | | The environment variables of the function |
| `s3_bucket_name` | `true` | `string` | | | The S3 bucket name containing the function package to be deployed |
| `s3_bucket_key` | `true` | `string` | | | The S3 key aka. filename referencing the file to be deployed |
### Requirements
| Name | Capability Type | Node Type Constraint | Relationship Type | Occurrences |
|:---- |:--------------- |:-------------------- |:----------------- |:------------|
| `host` | `tosca.capabilities.Container` | `radon.nodes.aws.AwsPlatform` | `tosca.relationships.HostedOn` | [1, 1] |
| Name | Capability Type | Node Type Constraint | Relationship Type | Occurrences |
| :-------- | :-------------------------------------- | :----------------------------------- | :------------------------------------------------ | :--------------- |
| `host` | `tosca.capabilities.Container` | `radon.nodes.aws.AwsPlatform` | `tosca.relationships.HostedOn` | [1, 1] |
| `monitor` | `radon.capabilities.monitoring.Monitor` | `radon.nodes.monitoring.PushGateway` | `radon.relationships.monitoring.AWSIsMonitoredBy` | [ 0, UNBOUNDED ] |
### Notes
* Parameters added to the `Standard` interface inputs:
* `aws_region`
* `role_arn`
* `lambda_runtime`
* `function_name`
* `lambda_timeout`
* `lambda_memory`
* `func_alias`
* `permission_id`
* `lambda_handler`
* `lambda_concurrency`
* `env_vars`
* `s3_bucket_name`
* `s3_bucket_key`
- Parameters added to the `Standard` interface inputs:
- `aws_region`
- `role_arn`
- `lambda_runtime`
- `function_name`
- `lambda_timeout`
- `lambda_memory`
- `func_alias`
- `permission_id`
- `lambda_handler`
- `lambda_concurrency`
- `env_vars`
- `s3_bucket_name`
- `s3_bucket_key`
Copyright (c) 2019 Contributors to the RADON project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
\ No newline at end of file
text/plain
\ No newline at end of file
tosca_definitions_version: tosca_simple_yaml_1_3
node_types:
radon.nodes.aws.AwsLambdaFunctionFromS3Scalable:
derived_from: radon.nodes.abstract.Function
metadata:
targetNamespace: "radon.nodes.aws"
abstract: "false"
final: "false"
attributes:
cpu_upper_bound:
type: float
role_arn:
description: ARN for the role of the function
type: string
callbackUrlCPU:
type: string
adjustment:
type: float
arn:
description: Amazon Resource Name (ARN) for this function
type: string
region:
type: string
ram_upper_bound:
type: float
properties:
handler:
type: string
description: Handler for lambda function
default: "index.handler"
s3_bucket_key:
type: string
description: The S3 key aka. filename referencing the file to be deployed
memory:
type: integer
description: Function memory in MB
default: 128
memory_range:
type: range
description: Range of function memory in MB to search
required: false
alias:
type: string
description: Lambda function's alias
required: false
default: "dev"
runtime:
type: string
description: Function runtime (List of possible values - https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html)
default: "nodejs"
concurrency_range:
type: range
description: Range of function concurrency to search
required: false
statement_id:
type: string
description: Lambda policy statement identifier
s3_bucket_name:
type: string
description: The S3 bucket name containing the function package to be deployed
timeout:
type: integer
description: Function timeout in seconds
default: 300
concurrency:
type: integer
description: Function concurrency
required: false
default: 0
env_vars:
type: map
description: Function's environment variables
required: false
entry_schema:
type: string
requirements:
- host:
capability: tosca.capabilities.Container
node: radon.nodes.aws.AwsPlatform
relationship: tosca.relationships.HostedOn
occurrences: [ 1, 1 ]
- monitor:
capability: radon.capabilities.monitoring.Monitor
node: radon.nodes.monitoring.PushGateway
relationship: radon.relationships.monitoring.AWSIsMonitoredBy
occurrences: [ 0, UNBOUNDED ]
interfaces:
scale_up:
type: radon.interfaces.scaling.ScaleUp
operations:
scale_up:
inputs:
adjustment: { default: 1, type: integer }
cpu_upper_bound: { default: 80.0, type: float }
ram_upper_bound: { default: 80.0, type: float }
callbackUrlCPU: { default: "null", type: string }
description: Operation for scaling up.
implementation:
primary: scaleUp
timeout: 0
Standard:
type: tosca.interfaces.node.lifecycle.Standard
inputs:
s3_bucket_key:
type: string
required: true
default: { get_property: [ SELF, s3_bucket_key ] }
aws_region:
type: string
required: true
default: { get_property: [ SELF, host, region ] }
role_arn:
type: string
required: true
default: { get_attribute: [ SELF, host, role_arn ] }
lambda_runtime:
type: string
required: true
default: { get_property: [ SELF, runtime ] }
function_name:
type: string
required: true
default: { get_property: [ SELF, name ] }
callbackUrlCPU:
type: string
required: false
default: { get_property: [ radon.policies.scaling.ScaleUp, callbackUrlCPU ] }
lambda_timeout:
type: integer
required: true
default: { get_property: [ SELF, timeout ] }
lambda_memory:
type: integer
required: true
default: { get_property: [ SELF, memory ] }
func_alias:
type: string
required: true
default: { get_property: [ SELF, alias ] }
permission_id:
type: string
required: true
default: { get_property: [ SELF, statement_id ] }
lambda_handler:
type: string
required: true
default: { get_property: [ SELF, handler ] }
s3_bucket_name:
type: string
required: true
default: { get_property: [ SELF, s3_bucket_name ] }
lambda_concurrency:
type: integer
required: true
default: { get_property: [ SELF, concurrency ] }
env_vars:
type: string
required: false
default: { get_property: [ SELF, env_vars ] }
cpu_upper_bound:
type: float
required: false
default: { get_property: [ radon.policies.scaling.ScaleUp, cpu_upper_bound ] }
s3_bucket_key:
type: string
required: true
default: { get_property: [ SELF, s3_bucket_key ] }
aws_region:
type: string
required: true
default: { get_property: [ SELF, host, region ] }
role_arn:
type: string
required: true
default: { get_attribute: [ SELF, host, role_arn ] }
function_name:
type: string
required: true
default: { get_property: [ SELF, name ] }
adjustment:
type: float
required: false
default: { get_property: [ radon.policies.scaling.ScaleUp, adjustment ] }
ram_upper_bound:
type: float
required: false
default: { get_property: [ radon.policies.scaling.ScaleUp, ram_upper_bound ] }
operations:
create:
implementation:
primary: create
timeout: 0
delete:
implementation:
primary: delete
timeout: 0
artifacts:
create:
type: radon.artifacts.Ansible
file: create.yml
delete:
type: radon.artifacts.Ansible
file: delete.yml
scaleUp:
type: radon.artifacts.Ansible
file: scaleUp.yml
![](https://img.shields.io/badge/Status:-TESTING-yellow)
![](https://img.shields.io/badge/%20-DEPLOYABLE-blueviolet)
## AWS Lambda Function From S3 Node Type
A node type that represents an AWS Lambda function which is instrumented based on a function deployment package from an AWS S3 bucket.
| Name | URI | Version | Derived From |
| :-------------------------------- | :------------------------------------------------ | :------ | :------------------------------ |
| `AwsLambdaFunctionFromS3Scalable` | `radon.nodes.aws.AwsLambdaFunctionFromS3Scalable` | 1.0.0 | `radon.nodes.abstract.Function` |
### Attributes
| Name | Type | Default Value | Description |
| :---------------- | :-------- | :------------ | :----------------------------------------------------- |
| `arn` | `string` | | Amazon Resource Name (ARN) for this function |
| `role_arn` | `string` | | ARN for the role of the function |
| `region` | `string` | | AWS region in which the function is deployed |
| `cpu_upper_bound` | `float` | | CPU upper alerting threshold |
| `ram_upper_bound` | `float` | | RAM upper alerting threshold |
| `adjustment` | `integer` | | Scaling adjustment |
| `callbackUrlCPU` | `string` | | Callback URL that handles the scaling triggering event |
### Properties
| Name | Required | Type | Constraint | Default Value | Description |
| :------------------ | :------- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------- | :--------------------------------------------------------------------------------- |
| `alias` | `true` | `string` | | "dev" | Lambda function's alias |
| `runtime` | `true` | `string` | `valid_values: [nodejs, nodejs4.3, nodejs6.10, nodejs8.10, nodejs10.x, java8, python2.7, python3.6, python3.7, dotnetcore1.0, dotnetcore2.0, dotnetcore2.1, nodejs4.3-edge, go1.x, ruby2.5]` | "nodejs" | The identifier of the function's runtime |
| `handler` | `true` | `string` | | "index.handler" | The name of the method within your code that Lambda calls to execute your function |
| `memory` | `true` | `integer` | `in_range: [128, 3008]` | 128 | The amount of memory in megabytes that your function has access to |
| `memory_range` | `true` | `range` | `in_range: [128, 3008]` | | Range of function memory in MB to search |
| `concurrency` | `false` | `integer` | `in_range: [0, UNBOUNDED]` | 0 | The amount of concurrency that your function has access to |
| `concurrency_range` | `false` | `range` | `in_range: [1, UNBOUNDED]` | | Range of function concurrency to search |
| `timeout` | `true` | `integer` | `in_range: [1, 900]` | 3 | The amount of time that Lambda allows a function to run before stopping it |
| `statement_id` | `true` | `string` | | | Lambda policy statement identifier |
| `env_vars` | `false` | `map` of `string` | | | The environment variables of the function |
| `s3_bucket_name` | `true` | `string` | | | The S3 bucket name containing the function package to be deployed |
| `s3_bucket_key` | `true` | `string` | | | The S3 key aka. filename referencing the file to be deployed |
### Requirements
| Name | Capability Type | Node Type Constraint | Relationship Type | Occurrences |
| :-------- | :-------------------------------------- | :----------------------------------- | :------------------------------------------------ | :--------------- |
| `host` | `tosca.capabilities.Container` | `radon.nodes.aws.AwsPlatform` | `tosca.relationships.HostedOn` | [1, 1] |
| `monitor` | `radon.capabilities.monitoring.Monitor` | `radon.nodes.monitoring.PushGateway` | `radon.relationships.monitoring.AWSIsMonitoredBy` | [ 0, UNBOUNDED ] |
### Notes
- Parameters added to the `Standard` interface inputs:
- `aws_region`
- `role_arn`
- `lambda_runtime`
- `function_name`
- `lambda_timeout`
- `lambda_memory`
- `func_alias`
- `permission_id`
- `lambda_handler`
- `lambda_concurrency`
- `env_vars`
- `s3_bucket_name`
- `s3_bucket_key`
text/x-web-markdown
\ No newline at end of file
File suppressed by a .gitattributes entry or the file's encoding is unsupported.
image/jpeg
\ No newline at end of file
File suppressed by a .gitattributes entry or the file's encoding is unsupported.
image/jpeg
\ No newline at end of file
---
- hosts: localhost
tasks:
- name: Lambda "{{ function_name }}" deploy
lambda:
name: "{{ function_name }}"
s3_bucket: "{{ s3_bucket_name }}"
s3_key: "{{ s3_bucket_key }}"
runtime: "{{ lambda_runtime }}"
region: "{{ aws_region }}"
role: "{{ role_arn }}"
handler: "{{ lambda_handler }}"
timeout: "{{ lambda_timeout }}"
memory_size: "{{ lambda_memory }}"
environment_variables: "{{ env_vars }}"
register: lambda_info
- name: "Put concurrency of Lambda function on AWS : {{ function_name }}"
command: >-
aws lambda put-function-concurrency
--function-name {{ function_name }}
--reserved-concurrent-executions {{ lambda_concurrency }}
--region {{ aws_region }}
when: (lambda_concurrency is defined) and (lambda_concurrency != 0)
- name: Create function alias name
lambda_alias:
function_name: "{{ function_name }}"
name: "{{ func_alias }}"
region: "{{ aws_region }}"
state: present
register: alias_info
- name: "Add invocation permission of Lambda function on AWS : {{ permission_id }}"
lambda_policy:
function_name: "{{ function_name }}"
statement_id: "{{ permission_id }}"
action: lambda:InvokeFunction
region: "{{ aws_region }}"
alias: "{{ func_alias }}"
principal: "s3.amazonaws.com"
- name: Set attributes
set_stats:
data:
arn: "{{ alias_info.alias_arn }}"
role_arn: "{{ role_arn }}"
region: "{{ aws_region }}"
- name: set outputs regarding to scaling policies
set_stats:
data:
cpu_upper_bound: "{{ cpu_upper_bound }}"
ram_upper_bound: "{{ ram_upper_bound }}"
adjustment: "{{ adjustment }}"
callbackUrlCPU: "{{ callbackUrlCPU }}"
text/plain
\ No newline at end of file
---
- hosts: localhost
tasks:
- name: "Remove function alias name"
lambda_alias:
function_name: "{{ function_name }}"
name: "{{ func_alias }}"
region: "{{ aws_region }}"
state: absent
ignore_errors: true
- name: "Lambda {{ function_name }} undeploy"
lambda:
name: "{{ function_name }}"
region: "{{ aws_region }}"
role: "{{ role_arn }}"
handler: "{{ lambda_handler }}"
state: absent
text/plain
\ No newline at end of file
---
- name: scaleUp
hosts: localhost
gather_facts: no
become: true
become_method: sudo
vars:
notification_obj: "{{ notification | regex_replace('[^a-zA-Z0-9:,-]') }}"
notification_obj1: "{{ notification_obj.split(':') }}"
notification_obj2: "{{ notification_obj1[2].split(',') }}"
monitored_function_name: "{{ notification_obj2[0] }}"
tasks:
- name: Start scaling up of Lambda function {{ monitored_function_name }}
debug:
msg: "Scaling up initiated."
- name: See what's in the notification file from monitoring that was supplied to opera notify "{{ notification }}"
debug:
msg: "{{ notification }}"
- name: Fecth the Labda function current memory size in order to increase it.
shell: aws lambda get-function-configuration --region eu-central-1 --function-name {{ monitored_function_name }}
register: current_function
- name: Get Current AWS Memory Size
set_fact:
memory: "{{ (current_function.stdout_lines | join('\n') | from_json).MemorySize }}"
- name: Adjust AWS Memory size
set_fact:
memory: "{{ 2*memory|int }}"
- name: Scale Up Lambda function by incesing the Memory Size.
shell: aws lambda update-function-configuration --function-name {{ monitored_function_name }} --region eu-central-1 --memory-size {{ memory }}
- name: Finish scaling up of {{ monitored_function_name }}
debug:
msg: "Scaling up concluded."
......@@ -22,7 +22,7 @@ node_types:
capabilities:
host:
occurrences: [ 1, UNBOUNDED ]
valid_source_types: [ radon.nodes.aws.AwsLambdaFunction, radon.nodes.aws.AwsS3Bucket, radon.nodes.VM.EC2, radon.nodes.aws.AwsApiGateway, radon.nodes.aws.AwsDynamoDBTable, radon.nodes.aws.AwsLambdaFunctionFromS3, radon.nodes.datapipeline.PipelineBlock ]
valid_source_types: [ radon.nodes.aws.AwsLambdaFunction, radon.nodes.aws.AwsS3Bucket, radon.nodes.VM.EC2, radon.nodes.aws.AwsApiGateway, radon.nodes.aws.AwsDynamoDBTable, radon.nodes.aws.AwsLambdaFunctionFromS3, radon.nodes.datapipeline.PipelineBlock, radon.nodes.aws.AwsLambdaFunctionFromS3Scalable ]
type: tosca.capabilities.Container
interfaces:
Standard:
......
......@@ -11,6 +11,11 @@ node_types:
port:
type: integer
default: { get_attribute: [ SELF, docker, port ] }
requirements:
- host:
capability: tosca.capabilities.Node
node: radon.nodes.VM.EC2
relationship: tosca.relationships.HostedOn
capabilities:
host:
occurrences: [ 1, UNBOUNDED ]
......@@ -21,6 +26,13 @@ node_types:
type: tosca.interfaces.node.lifecycle.Standard
operations:
configure:
inputs:
ssh_user:
type: string
default: { get_attribute: [ SELF, host, ssh_user ] }
ssh_key_file:
type: string
default: { get_attribute: [ SELF, host, ssh_key_file ] }
implementation:
primary: create
timeout: 0
......
......@@ -3,6 +3,10 @@
hosts: all
gather_facts: true
become: true
become_method: sudo
vars:
ansible_ssh_private_key_file: "{{ ssh_key_file }}"
ansible_ssh_user: "{{ ssh_user }}"
tasks:
- name: Prerequirements for apt module
shell:
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment