22 October 2018 / AUTOMATION, OPERATIONS

Operationalizing vRealize Automation - Logging

A common scenario I see is effort is often expended on integrating or automating as much as possible, whilst equivalent effort isn’t put into automating or scaling operations, this often leads to platform perception issues around service availability when there is failure as a result of a downstream system. Operations teams can end up in a cycle of chasing failed requests rather than proactively preventing failure. The first step to scaling vRealize automation operationally is having a solid logging framework in place to quickly identify automation failure and attribute downstream failures to root cause. The purpose of this blog post is to highlight a simple and consistent approach to logging as part of an overall development framework, when working with vRealize orchestrator, which is the integration layer between vRA and downstream systems.

vRealize Log Insight Queries for vRealize Orchestrator

Log insight via the vRealize Orchestrator content pack has native support for parsing vRealize Orchestrator (vRO) scripting logs, the same logs that are otherwise visible in the vRO client. Beyond development, taking a top down approach and looking at individual issues via the vRO client doesn’t operationally scale, with the potential for logs to be located on on of many vRA/vRO nodes, and logs are only visible on the locally connected node when using the vRO java client. vRealize Log Insight provides an out of the box integration to query many vRO nodes and/or many environments simply.

Errors within vRO can be identified in vRealize Log Insight using the error priority of info, warn, error. This correlates with the use of System.log, System.warn and System.error.

vRO Errors using vRealize Log Insight query

product contains vro
filepath contains /var/log/vco/app-server/integration-scripting.log
priority contains error

Errors can also be grouped by context or traceid for association or grouping with a vRA request, the context is a unique identifier assigned by vRA and passed into vRA and vRO logs for end to end request trace ability.

Catching Errors in vRealize Orchestrator

The default error handler within vRealize orchestrator allows for any exception to be caught and redirected. This method is generally the standard approach for catching errors within a workflow.

Logging Standardisation

Standardisation is a key principle that should always be forefront when working on automation. The standardisation of logging using a consistent approach allows for simple query analysis using log analytic tools such as vRealize Log Insights. A simple approach thats allows for identifying an unknown error is to append or concatenate error tags or error codes in a standard format, this allows for a regex query to be performed to identify the associate downstream system with any potential error. The example below shows appending an integration identifier to an exception using an action in combination with a default error handler, this can be extended to suit the business logic needed, example, errors could be attributed to a particular blueprint or business group depending on requirements.

    return  ("[integration="+'"' + tag +'"'+"] " + errorCode);

Custom Field Extraction

Custom field extraction can be used to identify reusable standardised log patterns via regex queries.

Extracting a custom field involves highlighting the text you want to build a query for.
Once the data in selected, the regex to match the data and field name identifier can then be assigned.

Building Queries from Custom Fields

Query to group errors by integration failure percent

product contains vro
filepath contains /var/log/vco/app-server/integration-scripting.log
priority contains error

grouped by Integration

Non-Time series

In this example we are now able to give a business context to a failure, without having to know the downstream error, this allows for decisions to be made on effort investment and improving reliability of systems or integrations, additionally each failure can be drilled into and troubleshooted down to log level.

Operational Dashboards

Once you have built up your queries to provide visibility, these queries can then be saved to shared dashboards for operational use, or the queries can be used for proactive notification via SMTP or vRealize Operation integration.

Integration Testing Dashboards

Taking the same concept further, the same logging framework can be extended for integration test reporting. Creating an additional field for integration status and executing wrapper integration test workflows either as scheduled workflows, or via the vRO REST API, leveraging test framework tools such as pester. These tests may be extended upon and leveraged for integration testing during the release process.

return ("[integration="+'"' + tag +'"'+"][integationStatus="+'"' + status +'"'+"] " + errorCode);

Once we have an additional custom field extracted for the integration status, we can start logging successful or failed integration tests and tracking the status via queries and rolled up dashboards.

Query to group errors by integration pass/failure rate

product contains vro
filepath contains /var/log/vco/app-server/integration-scripting.log
integration contains integrationIdentifier

grouped by integration, integrationStatus