/ AUTOMATION

Extending Cloud Assembly with Action Based eXtensibility for ServiceNow Integration

Over the last week, we’ve both been diving deep into extensibility with VMware’s new Cloud Automation Services, specifically Cloud Assembly. This was particularly interesting to us as previously we spent the majority of our time living and breathing vRealize Orchestrator in our consulting days.

There are two methods for extensibility:

  1. Action Based eXtensibility (ABX) which utilises serverless functions to execute code
  2. vRealize Orchestrator (vRO) which utilises a traditional approach of workflow based execution often used by other enterprise orchestration platforms

There is a fair amount to cover so we decide to split the blog post into two parts. This blog post will focus on ABX extensibility and not long after we’ll release vRO extensibility.

For more information on the Cloud Automation Services, here are the official blog posts from VMware:

Subscriptions

For those familiar with vRealize Automation’s Event Broker Service (EBS), Cloud Assembly has the same concept where you can configure subscriptions to trigger actions based on certain lifecycle events and filters.

Event Topics and Execution Flow for Provisioning

Here are the Event Topics available today, we suspect as the product evolves more will be made available.

Below you’ll see us demonstrate and use Compute provision post and Compute post removal event topics as part of our Enterprise ITSM use case.

If you’re new to the concept of subscription and events it’s probably good to understand the execution flow or order when an instance is provisioned. Here is a diagram to help visualise this:

Note: Compute provision pre is before creation of an instance, and Compute provision post is after.

Event Schema

In term of understanding what data is available at each event, the events schema should be referenced. The schema describes the structure of the events payload or inputProperties.

The event schema can be viewed by clicking on the event topic or in the subscription. The below example is when clicking into the event topic Compute provision post

Event Filtering

Filters can be applied on subscriptions to set boolean based conditions for the event, I.e. the event and action would only trigger if the boolean expression is ‘true’.

This is particularly useful for controlling when events and actions are triggered, E.g. Say, you only wanted to execute an action based on a specific project.

Other common scenarios are for blueprints, components, tags or even custom properties. Hint, hit “⌥ + Space” on Mac or “Alt + Space” on Windows to show the options available in the filter input.

Runnable Items

Once you have configured the subscription filters, you then specify the item to execute for this new subscription. The items appear in the drop down list, its worth mentioning that you should test them independently first before integrating into your subscription.

Action Based eXtensibility (ABX)

Action Based eXtensibility allows for the execution of serverless functions to extend the product capability based on your use case. ABX today, supports two run-time environments, Node.js or Python. Our view is this can be thought of as a lightweight orchestration engine, providing a quick and efficient programmable method for integrating with other Cloud services.

At the moment, ABX relies on Amazon Web Services (AWS) Lambda which means you must have an active subscription and complete some prerequisites such as IAM configuration and adding the AWS endpoint into Cloud Assembly. For details on how to get started visit the official page

Payload

The event payload data contains all the relevant properties that are passed through during that lifecycle state (Event Topic), it’s important to understand this as certain states may contain different data.

Once you know what is available in the payload data you can use the inputProperties easily in your action.

A useful way to examine the payload is to create a separate action and a subscription with the relevant state.

Log Payload Example

import json

def handler(context, inputs):
    print(json.dumps(inputs, indent=2))

Payload Output Example

{
  "componentId": "Cloud_Machine_1",
  "customProperties": {
    "image": "ubuntu"
  },
  "componentTypeId": "Cloud.Machine",
  "requestId": "2ac72cdfda856c7557b347d5c582a",
  "deploymentId": "2ac72cdfda856c7557b347d5c582a",
  "placementIds": [
    "b24bfa74-2dfd-4ac8-b091-b37610246228-1922e32131ad147557af6ff7ef880",
    "b24bfa74-2dfd-4ac8-b091-b37610246228-8c9d1f2cf124ba7557af6fea0b610"
  ],
  "__metadata": {
    "orgId": "a70acdfd-6050-4f2a-abaa-8d9a567ca491",
    "headers": {
      "blocking": "false",
      "runnableId": "8a76ae98672362180167347411c6007f",
      "runnableType": "extensibility.abx",
      "uber-trace-id": "ed9222423cadc56d:ed9222423cadc56d:0:1",
      "eventTraceEntryId": "3a515c29-f91c-40cb-a7ec-d591c80c7149"
    },
    "targetId": "http://10.246.3.66:8282/provisioning/config/extensibility-callbacks/56f0d9c62982ee7557b347d7b1f11",
    "userName": "tphan@vmware.com",
    "eventType": "ANY",
    "timeStamp": 1542840257306,
    "sourceType": "provisioning",
    "targetType": "ExtensibilitySubscriptionCallback",
    "description": "Compute reservation",
    "eventTopicId": "compute.reservation.pre",
    "correlationId": "2ac72cdfda856c7557b347d5c582a",
    "sourceIdentity": "http://10.246.3.66:8282/provisioning/config/extensibility-callbacks/56f0d9c62982ee7557b347d7b1f11",
    "correlationType": "contextId"
  },
  "projectId": "b24bfa74-2dfd-4ac8-b091-b37610246228",
  "tags": {},
  "resourceIds": [
    "3ddad1277ff4727557b347d6ad330"
  ]
}

Inputs

Like vRealize Orchestrator you can pass inputs into the script, see example below:

It’s also possible to directly access any item within the schema even if not specified under inputs in the ui. See the below example to access a property value within customProperties in the event schema.

 inputs['customProperties']['someProperty']

Outputs

Outputs are used for returning values or updating properties withín the payload for use in subsequent actions. You can update properties within the same event and the subsequent action event will receive the updated payload.

Below is a code example for update the properties within an ABX action.

    outputs = {}
    outputs['customProperties'] = inputs['customProperties']
    outputs['customProperties']['SomePropetyToUpdate'] = someValue;
    return outputs

Enterprise ITSM Use Case (ServiceNow)

A common enterprise use case for any Cloud Management Platform or Automation is integrating with an IT Service Management (ITSM) and Configuration Management Database (CMDB) platform such as ServiceNow for compliance. Here we’ll demonstrate integrating with ServiceNow for CMDB and ITSM using ABX. Because our ServiceNow instance is a Cloud service, this is a perfect way to leverage ABX.

For this example we have broken the ServiceNow integration down into four serverless actions. The first three actions are executed at provisioning time, at the Compute provision post event, each at a different priority in this case for sequential execution. The events used and priority order may vary depending on the customer requirements. The image outlines the actions and order of execution based on subscription priority.

The last action is executed at the Compute removal post event.

For this use case we filtered all event subscription based on a conditional blueprint property which exists on our blueprints requiring SNOW integration.

event.data[“customProperties”][“enable_servicenow”] === “true”

CMDB Integration

Get Additional VM Details (Python)

Elements considered as part of the action:

  1. Additional details are required for the CI creation as sufficient data is not available in the payload
  2. A request is made to the CAS IaaS API to get the required details, authenticating with a stored identity token. This value is stored securely in AWS Systems Manager Parameter Store (SSM) and referenced in code.
  3. customProperties are updated with additional properties to be used in subsequent actions
from botocore.vendored import requests
import json
import boto3
client = boto3.client('ssm','ap-southeast-2')

def handler(context, inputs):
    baseUri = inputs['url']
    casToken = client.get_parameter(Name="casToken",WithDecryption=True)
    
    url = baseUri + "/iaas/login"
    headers = {"Accept":"application/json","Content-Type":"application/json"}
    payload = {"refreshToken":casToken['Parameter']['Value']}
    
    results = requests.post(url,json=payload,headers=headers)
    
    bearer = "Bearer "
    bearer = bearer + results.json()["token"]
    
    deploymentId = inputs['deploymentId']
    resourceId = inputs['resourceIds'][0]
    
    print("deploymentId: "+ deploymentId)
    print("resourceId:" + resourceId)
    
    machineUri = baseUri + "/iaas/machines/" + resourceId
    headers = {"Accept":"application/json","Content-Type":"application/json", "Authorization":bearer }
    resultMachine = requests.get(machineUri,headers=headers)
    print("machine: " + resultMachine.text)
    
    print( "serviceNowCPUCount: "+ json.loads(resultMachine.text)["customProperties"]["cpuCount"] )
    print( "serviceNowMemoryInMB: "+ json.loads(resultMachine.text)["customProperties"]["memoryInMB"] )
    
    #update customProperties
    outputs = {}
    outputs['customProperties'] = inputs['customProperties']
    outputs['customProperties']['serviceNowCPUCount'] = int(json.loads(resultMachine.text)["customProperties"]["cpuCount"])
    outputs['customProperties']['serviceNowMemoryInMB'] = json.loads(resultMachine.text)["customProperties"]["memoryInMB"]
    return outputs

ServiceNow CMDB Configuration Item Creation Action (Python)

Elements considered as part of the action:

  1. Passing the ServiceNow instance URL as input

  2. Storing the ServiceNow instance credentials in AWS Systems Manager Parameter Store (SSM) to meet security requirements for storing secrets

    Note: Appropriate additional permissions need to be allocated to your CAS AWS role to allow Lambda to access SSM Parameter Store and decrypt secrets.

  3. Parsing various properties from the event payload, such as name, cpu, memory, etc.
  4. Reading the ServiceNow CMDB unique record identifier (sys_id) response, passing as an output and writing to the custom property serviceNowSysId during the creation. This value is subsequently used to mark the CI as retired on the destroy of the instance
from botocore.vendored import requests
import json
import boto3
client = boto3.client('ssm','ap-southeast-2')

def handler(context, inputs):

    snowUser = client.get_parameter(Name="serviceNowUserName",WithDecryption=False)
    snowPass = client.get_parameter(Name="serviceNowPassword",WithDecryption=True)
    table_name = "cmdb_ci_vmware_instance"
    url = "https://" + inputs['instanceUrl'] + "/api/now/table/{0}".format(table_name)
    headers = {'Content-type': 'application/json', 'Accept': 'application/json'}
    payload = {
        'name': inputs['customProperties']['serviceNowHostname'],
        'cpus': int(inputs['customProperties']['serviceNowCPUCount']),
        'memory': inputs['customProperties']['serviceNowMemoryInMB'],
        'correlation_id': inputs['deploymentId'],
        'disks_size': int(inputs['customProperties']['provisionGB']),
        'location': "Sydney",
        'vcenter_uuid': inputs['customProperties']['vcUuid'],
        'state': 'On',
        'sys_created_by': inputs['__metadata']['userName'],
        'owned_by': inputs['__metadata']['userName']
        }
    results = requests.post(
        url,
        json=payload,
        headers=headers,
        auth=(snowUser['Parameter']['Value'], snowPass['Parameter']['Value'])
    )
    print(results.text)

    #parse response for the sys_id of CMDB CI reference
    if json.loads(results.text)['result']:
        serviceNowResponse = json.loads(results.text)['result']
        serviceNowSysId = serviceNowResponse['sys_id']
        print(serviceNowSysId)

        #update the serviceNowSysId customProperty
        outputs = {}
        outputs['customProperties'] = inputs['customProperties']
        outputs['customProperties']['serviceNowSysId'] = serviceNowSysId;
        return outputs

CMDB Configuration Item Retire Action (Python)

This actions makes a REST call to ServiceNow to mark the CI as retired based on the custom property “serviceNowSysId” which was previously written to the instance on CI creation.

from botocore.vendored import requests
import json
import boto3
client = boto3.client('ssm','ap-southeast-2')

def handler(context, inputs):
    snowUser = client.get_parameter(Name="serviceNowUserName",WithDecryption=False)
    snowPass = client.get_parameter(Name="serviceNowPassword",WithDecryption=True)
    tableName = "cmdb_ci_vmware_instance"
    sys_id =inputs['customProperties']['serviceNowSysId']
    url = "https://" + inputs['instanceUrl'] + "/api/now/"+tableName+"/{0}".format(sys_id)
    headers = {'Content-type': 'application/json', 'Accept': 'application/json'}
    payload = {
        'state': 'Retired'
        }

    results = requests.put(
        url,
        json=payload,
        headers=headers,
        auth=(inputs['username'], inputs['password'])
    )
    print(results.text)

Example data as a result of this ABX integration can be seen below in ServiceNow CMDB.

ITSM Integration

Elements considered as part of the action include:

  1. Passing the ServiceNow instance URL as input
  2. Storing the ServiceNow instance credentials in AWS’s Systems Manager Parameter Store to meet security requirements of storing secrets

Change Creation Action (Python)

from botocore.vendored import requests
import json
import boto3
client = boto3.client('ssm','ap-southeast-2')

def handler(context, inputs):
    snowUser = client.get_parameter(Name="serviceNowUserName",WithDecryption=False)
    snowPass = client.get_parameter(Name="serviceNowPassword",WithDecryption=True)
    table_name = "change_request"
    url = "https://" + inputs['instanceUrl'] + "/api/now/table/{0}".format(table_name)
    headers = {'Content-type': 'application/json', 'Accept': 'application/json'}
    payload = {
        'short_description': 'Provision CAS VM Instance'      
              }
    results = requests.post(
        url,
        json=payload,
        headers=headers,
        auth=(snowUser['Parameter']['Value'], snowPass['Parameter']['Value'])
    )
    print(results.text)

Note: These examples are provided as a means to demonstrate the use case, but whilst functional should not be considered production ready, additional considerations should be taken for production including error handling.

You can import a copy of the ABX actions mentioned above using the bundle below.

abx-snow-cloudbrokersblog-bundle.zip

Next up we’ll look at the same ServiceNow integration use case but instead use vRO to extend Cloud Assembly.