Version: 12 - TBD

Connectors

About

Connectors are an Engine resources that allow your flow scripts to interact with the outside world - anything and everything outside of the Engine platform. Whenever you want to issue a command to a program, run a script on a remote system, or query a database - all that is performed through connectors.

There are different connector types, each for a particular type of endpoint / protocol.

Connectors are called through the Engine function this.connect(). Each call creates a separate execution of type "connection", with its own inputs and outputs.

note

Connectors enable outbound communication from Engine. If you want to call Engine from a third party system you can do so with a webhook.

video explainer: connections

Can't see the video? Watch it on YouTube

video explainer: connectors

Can't see the video? Watch it on YouTube

Usage

There are two ways you can go about using a connector.

The first is to create a connector resource, configure it, and use it in your flow scripts. This is useful if most parameters of your connector are static (e.g. host and port of a database).

The second is to define the connector in your flow script on the fly. This makes sense if the connector parameters change all the time or you're simply testing a connection and you just want to create a disposable connector that you don't intend to reuse.

Creating and using a connector resource

Click on '+ Create' -> 'Connector' -> 'OPENAI'. Call it "my_openai_connector". You can configure the connector attributes in a form or in YAML format. For configuring using the YAML format switch to 'Code view'.

The form for configuring a connector

The code view for configuring a connector

note

The configuration must be a valid YAML document: yaml.org/spec It is also possible to enter a JSON config, since YAML is a superset of JSON. Invalid lines in the configuration (e.g. lines beginning with #) are discarded.

Here's how you can use this newly created and configured connector in a flow script. Note that the first parameter of this.connect() is the name of the connector.

import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    models = this.connect('my_openai_connector').get('output_value')
    this.log(models)
    return this.success('all done')

Overriding inputs

You can override inputs which are stored in the connector by specifying different values when using the connector. Let's modify the flow script from before to use a different API key.

import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    models = this.connect('my_openai_connector', api_key='anotherapikey').get('output_value')
    this.log(models)
    return this.success('all done')

warning

When you override a key in the input value that is a YAML object (or Python dictionary), the old value will be merged (and not replaced) with the new value. This can result in an invalid configuration.

Overriding a key that is not an object. This works fine:

# connector config of `my_connector`
my_key: 1

# flow script
this.connect('my_connector', my_key=2)

# resulting config of the connection execution
my_key: 2

Overriding a key that is an object. This results in a merged object, that leads to an error, since the connector does not expect the temp_path key with the mode_name run_commands.

# connector config of `my_connector`
mode:
    mode_name: execute_script
    temp_path: /tmp

# flow script
this.connect('my_connector', mode={
    'mode_name': 'run_commands',
    'script': 'uptime',
})

# resulting config of the connection execution
mode:
    mode_name: run_commands
    temp_path: /tmp
    script: uptime

To avoid this, only define keys in a connector that are valid accross all modes that you will use, and define the rest in the flow script. You can also create multiple connectors for different modes, and use the appropriate connector in the flow script.

Defining connectors within flow scripts

If you don't want to use a connector resource, you can define the connector parameters in the flow script. Note, that you need to specify connector_type as well (if an existing connector resource is used, this parameter is already supplied).

import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    inputs = {
        'url': 'https://httpbin.org/post',
        'method': 'post',
        'data': 'payload',
    }
    child_execution = this.connect(connector_type='REST', **inputs, run=False)
    child_execution.run_async()
    # do other stuff
    child_execution.wait()
    outputs = child_execution.get('output_value')
    this.log(outputs)
    return this.success('all done')

Attach Vault Secrets

To be able to attach secrets to a connector, a working vault integration must be configured beforehand.

Please refer to Vault Integration for details.

Order of Input Application

Inputs for a connection execution can be specified in different places. Inputs from all places are merged into one combined input set before being used by the connector. Inputs are applied in the following order:

Value of the connector
Vault secrets associated with the connector
Inputs specified in the flow script

Inputs which are applied later can override keys of inputs which were applied before. If, for example, the connector specifies a key port and the vault secret also contains a key port, the value of the vault secret will be used.

Analysing and testing connections

You can extend the functionality of your connectors by downloading the freely available Connection Analysis & Test bundle. With this bundle you can, for example, get the schema of a database or test the connection to an FTP server by simply clicking on a button in the UI of your connector.