Settings and Locks
With settings it is possible to store structured data in Engine. Locks can be used to synchronize the processing of parallel running executions.
Use Cases
Settings can be used
- to store configuration parameters which are used by multiple flows
- by executions to read input, store outputs/logging/reports
- as lock objects
Concept
Each setting has a name and can contain any JSON-serializable data structure. Comment lines (lines starting with a hash sign) will be discarded and the order of key-value pairs will be sorted alphabetically upon saving. Settings are stored as JSON data structures and therefore behave differently than strings.
Settings can be accessed by ID or name. The value of a setting can only be read or written as a whole.
Settings also double as lock objects.
Using Settings
Settings can be manipulated via the user interface, via the REST API, and via flow scripts. The examples in this document are limited to one method per use case. The method described is interchangeable with any of the other methods.
To manipulate Settings using the command line you need an authorization token. Please see the Authentication documentation on how to obtain an authorization token.
Since settings are stored as JSON data structures, you can't just save any value. For example lines starting with the # symbol get discarded upon saving.
Store Configuration Parameters
You can manually or automatically store configuration parameters in settings which can be read by executions.
Store configuration parameters using the command line:
$ curl -X POST 'https://<my-workspace-name>.cloudomation.com/api/latest/setting' -d '{"name":"notification_emails","value":["toni@example.com","cory@example.com"]}' -H "Authorization: $TOKEN"
Read the configuration parameter in a flow:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# we use the setting named "notification_emails" and access its value
emails = system.setting('notification_emails').get('value')
this.connect(
'my-smtp-server',
name='send notification',
from_='noreply@example.com',
to=emails,
subject='notification from Engine',
text='test email content',
)
return this.success('all done')
If you choose to change the emails which should receive notifications you only need to update it in one place: the setting value:
$ curl -X PATCH 'https://<my-workspace-name>.cloudomation.com/api/latest/setting/notification_emails?by=name' -d '{"value":["toni@example.com","cory@example.com","tracy@example.com"]}' -H "Authorization: $TOKEN"
and with the next execution your flow scripts will read and use the new value.
Store Outputs/Logging/Reports
Your flow scripts can write the value of a setting to store the result of some processing, store logging of some processing, or store a report which was generated:
Storing the result of some processing in a setting:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# do some processing
result = 42
# store the result
system.setting('occurrences_found').save(value=result)
return this.success('all done')
Other flow scripts can read the value and adapt their behaviour accordingly:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
count = system.setting('occurrences_found').get('value')
if count > 32:
this.connect(
connector_type='SMTP',
smtp_host='mail.example.com',
from_='no-reply@example.com',
to='kevin@example.com',
subject='counter alert',
text=f'found {count} occurrences'
)
return this.success('all done')
The value can also be retrieved using the REST API:
$ curl 'https://<my-workspace-name>.cloudomation.com/api/latest/setting/occurrences_found?by=name' -H "Authorization: $TOKEN" | jq .
{
"setting": {
"name": "occurences_found",
"value": 42,
...
}
}
Lock Objects
It is possible for an execution to acquire a lock on a setting.
Each setting can be locked by one execution at a time. Other executions waiting to acquire a lock on the same setting will wait in the status WAITING_LOCK
until it becomes available or a timeout occurs.
Currently only exclusive locks on settings can be acquired. No shared-lock mechanism is implemented.
Make sure only one cloud-vm is provisioned at once.
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# we try to acquire the lock
system.setting('cloud-vm-lock').acquire()
# start the flow which launches the cloud-vm
this.flow('create-cloud-vm')
# use the cloud-vm
this.connect('cloud-vm', script='sleep 30')
# delete the cloud-vm
this.flow('remove-cloud-vm')
# free the lock
system.setting('cloud-vm-lock').release()
return this.success('all done')
If the cloud-vm-lock
is free when an execution of this flow runs, the execution will acquire the lock and continue its processing.
When a second execution of the flow is started during the processing, it cannot acquire the lock and wait in the status WAITING_LOCK
. Per default a lock wait timeout of 60 seconds is used. If the lock cannot be acquired in this time, the second execution will fail with a LockTimeoutError
exception.
If instead the lock becomes free within the timeout, the second execution will immediately acquire it and contine its processing.
If several executions are waiting for the same lock, the order in which they acquire the lock is not defined.