Search for a command to run...
I am a Lead Engineer from India. Love to blog about serverless and help teams design and develop serverless architecture. An AWS cloud practitioner. Blogs about AWS Services utilising AWS CDK, CloudFormation, Terraform.
No comments yet. Be the first to comment.
Transformers, the backbone of many state-of-the-art NLP models such as BERT, GPT has revolutionized the way we approach natural language understanding tasks. One key innovation in transformers is their ability to handle entire sequences of tokens sim...
When an User prompts the questions to the LLM, it produces the response based on the training data. The pre-trained LLM data is its parametric data. It might not have an context specific to our query. LLM still produces an answer confidently. It is t...
In the fast-evolving field of artificial intelligence, Large Language Models (LLMs) have rapidly transformed how we approach language processing. These models, such as ChatGPT, Claude, and Bard, have proven their utility across a range of application...
Amazon S3 is a powerful, scalable storage service, but when multiple clients or processes attempt to write to the same S3 object simultaneously, things can get tricky. Since S3 doesn’t natively handle concurrent writes or provide object locking, it’s...
When working with distributed systems in a microservice architecture, we must always architect our services with failure in mind. When the transaction happens between a microservice, we have to handle the failures. The Saga pattern can be used in a f...
AWS step function is used to orchestrate the workflow by coordinating multiple AWS services. Step function will have multiple tasks which can call other services. Each service expects different input parameters. As the workflow moves from one task to another task, the Step function needs a way to manipulate and filter the input JSON.
In this blog post, we will see various fields which can be used to manipulate and filter the input JSON.
In the Step function state definition, we can use the below fields that filter and control the flow of JSON from state to state.
Step function uses an Amazon States Language to define the definition. In Amazon States Language
For example, if state input data contains the following values:
{
"companyName": "XYZ",
"employeeName": ["John", "Eric", "William"],
"department": {
"software": true
}
}
The following reference paths would return the following.
$.companyName=> "XYZ"
$.employeeName=> ["John", "Eric", "William"]
$.department.software=> true
InputPath is used to select a portion of the state input. For example, suppose the input to your state includes the following.
{
"comment": "InputPath Example",
"employee": {
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32
},
"address": {
"home": "Home address",
"office": "Office address"
},
"country": "India"
}
If we are interested only in address of the employee, we can use InputPath "InputPath": "$.address"
With the previous InputPath, the below JSON is passed as the input to the next state.
{
"home": "Home address",
"office": "Office address"
}
We can use a Data flow simulator to verify on the AWS console.
With the Parameters field, we can create a key-value pair that can be passed as input. The values can either be static values or a part of the input or the context object. For key-value pairs where the value is selected using a path, the key name must end in .$.
For example, suppose you provide the following input.
{
"comment": "Parameters Example",
"employee": {
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32
},
"address": {
"home": "Home address",
"office": "Office address"
},
"country": "India"
}
We can select some of the fields using parameters as below.
"Parameters": {
"comment": "Parameters Example",
"employeeDetails": {
"firstName.$": "$.employee.firstName",
"lastName.$": "$.employee.lastName",
"continent": "Asia"
}
}
Given the previous input and the Parameters field, this is the JSON that is passed.
{
"comment": "Parameters Example",
"employeeDetails": {
"firstName": "Rahul",
"lastName": "Lokurte",
"continent": "Asia"
}
}
From the above output, we see that the static field continent is added and firstName and lastName are derived from the input. Also notice, the key has $ at the end, so that, it knows, it should derive the data from input JSON.
When the task of the state is completed, if we would like to manipulate the result of the task, we use the ResultSelector field. With the ResultSelector field, we can create a key-value pair, where the values are static or selected from the task's result. The output of ResultSelector replaces the task's result and is passed to ResultPath.
The step functions task usually returns the metadata along with the response. If we want only a few of the fields from the metadata, we can use ResultSelector to extract the data.
For example, suppose we get the output as given below.
{
"output": {
"RequestId": "ABCDEFGH",
"ExecutedVersion ": "$LATEST ",
"Payload": {
"statusCode ": 200,
"body": ["Hello World"]
},
"SdkHttpMetadata ": {
"AllHttpHeaders ": {
"X-Amz-Executed-Version": ["$LATEST "],
"x-amzn-Remapped-Content-Length": ["0 "],
"Connection": ["keep - alive "],
"x-amzn-RequestId": ["123456"],
"Content-Length ": ["28"],
"Content-Type ": ["application / json "]
},
"StatusCode ": 200
},
"outputDetails": {
"truncated": false
}
},
"resource": "invoke",
"resourceType": "lambda"
}
If we are interested only in resourceType and RequestId, we can use the ResultSelector as below.
"ResultSelector": {
"RequestId .$": "$.output.RequestId",
"ResourceType.$": "$.resourceType"
},
With the given input, using ResultSelector produces
{
"ResultSelector": {
"RequestId ": "ABCDEFGH",
"ResourceType": "lambda"
}
}
We want to control what JSON data needs to be passed to the task output. We have different combinations of data. We may want to just pass the result to output or we may want to combine the result of the task and the input which was given to the task. We can control all the scenarios using the ResultPath field.
Suppose, we have the input to the task as below.
{
"comment": "ResultPath Example",
"employee": {
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32
},
"address": {
"home": "Home address",
"office": "Office address"
},
"country": "India"
}
Let us consider that, we are passing this input to a lambda function, which generates a unique employee Id as the output.
{
"employeeId": "1234"
}
If we want all the input data which was given to the lambda and the result of lambda, we can use "ResultPath": "$.TaskResult"
The output includes the result of the Lambda function and the original input.
{
"comment": "ResultPath Example",
"employee": {
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32
},
"address": {
"home": "Home address",
"office": "Office address"
},
"country": "India",
"TaskResult": {
"employeeId": "1234"
}
}
We can also add the result of lambda into a child node of the input by using "ResultPath": "$.employee.outputlambda"
The output of the Lambda function is inserted as a child of the employee node in the input.
{
"comment": "ResultPath Example",
"employee": {
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32,
"outputlambda": {
"employeeId": "1234"
}
},
"address": {
"home": "Home address",
"office": "Office address"
},
"country": "India"
}
We can select the portion of the task output and pass it to the next state using OutputPath . This helps us to filter irrelevant information, and pass only the portion of JSON that is relevant.
Suppose, we have the output as below.
{
"comment": "ResultPath Example",
"employee": {
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32
},
"address": {
"home": "Home address",
"office": "Office address"
},
"country": "India",
"TaskResult": {
"employeeId": "1234"
}
}
If we want only information about employees, we can use $.employee and we get the below response.
{
"firstName": "Rahul",
"lastName": "Lokurte",
"age": 32
}
If you don't specify an OutputPath the default value is $. This passes the entire JSON node to the next state.
In this blog post, we saw various fields provided by Amazon state language which can be used to manipulate and filter the JSON contents in the Step function workflow. Depending on the requirement of a specific task, we can pass the portion of the input JSON and we can send a portion of output JSON to the next task.