This blog will take a deep dive into these embeddings, explaining how they are constructed, their mathematical foundations, and how they empower transformer models to achieve incredible performance.

Token Embeddings: Giving Words Meaning

What Are Token Embeddings?

Token embeddings represent the semantic meaning of words in a vector space. In simple terms, they map words (or subwords) into numerical vectors that capture their meanings. Words with similar meanings are mapped to nearby points in this vector space.

For example:

• The word "king" might be represented as [0.5,0.2,0.8,...][0.5, 0.2, 0.8, ...][0.5,0.2,0.8,...].

• The word "queen" might be represented as [0.6,0.3,0.7,...][0.6, 0.3, 0.7, ...][0.6,0.3,0.7,...].

• Their closeness in the vector space reflects their semantic similarity.

Tokenization and Vocabulary

Transformers begin by breaking down text into smaller units called tokens. These tokens can be words, subwords, or characters, depending on the tokenizer used (e.g., WordPiece, BPE, or SentencePiece). The tokenizer then assigns each token a unique token ID based on a pre-defined vocabulary.

Example

Consider the sentence:

• "I love NLP."

The vocabulary might look like this:

• "I" → Token ID 1

• "love" → Token ID 2

• "NLP" → Token ID 3

The sentence is tokenized as: [1, 2, 3]

Each token ID is mapped to an embedding vector using an embedding matrix.

Mathematical Representation of Token Embeddings

The embedding matrix E is a learnable matrix of size [V×d], where:

• V is the size of the vocabulary.

• d is the embedding dimension.

When a token ID is encountered, its embedding is obtained by indexing the corresponding row in E:

Token Embedding=E[Token ID]

For example:

• For Token ID 1 (I): E[1]=[0.1,0.3,0.4]

• For Token ID 2 (love): E[2]=[0.2,0.6,0.5]

• For Token ID 3 (NLP): E[3]=[0.7,0.9,0.8]

Positional Embeddings: Encoding Sequence Order

Why Are Positional Embeddings Needed?

Transformers process tokens in parallel, unlike RNNs, which process them sequentially. This parallelism means transformers lack an inherent understanding of the order of tokens. For example, the sentences:

• "I love NLP" and "NLP love I" would appear identical without positional information.

To address this, positional embeddings encode the position of each token in the sequence, allowing the model to differentiate between tokens based on their order.

Mathematical Construction of Fixed Positional Embeddings

The fixed sinusoidal positional embeddings, introduced in the original transformer paper, provide a deterministic way to represent positions. These embeddings are computed using sine and cosine functions, ensuring smooth variations across dimensions and positions.

The Formula

For a sequence length L and embedding size d, positional embeddings are computed as follows:

• For even dimensions (2i):

PE(pos,2i)=sin⁡(pos/100002id)

• For odd dimensions (2i+1):

PE(pos,2i+1)=cos⁡(pos/100002id)

Where:

• pos is the position of the token in the sequence.

• i is the dimension index.

Example

Consider a sequence of length L=4 and embedding size d=6. For position pos=2:

• Dimension 0 (2i=0):

PE(2,0)=sin⁡(2/10000^0/6)=sin⁡(2)≈0.909

• Dimension 1 (2i+1=12i+1 = 12i+1=1):

PE(2,1)=cos⁡(2/10000^0/6)=cos⁡(2)≈−0.416

• Higher dimensions follow the same principle, with increasing divisors.

Key Properties of Sinusoidal Embeddings

• Periodic Nature: The periodicity of sine and cosine functions allows the model to generalize to sequences longer than those seen during training.

• Smooth Variations: Neighboring positions have embeddings with small, smooth differences.

• Unique Representations: The combination of sine and cosine ensures each position is uniquely encoded.

Combining Token and Positional Embeddings

The final input to a transformer model is obtained by adding the token embeddings and the positional embeddings element-wise. This addition provides the model with both:

1. What: The semantic meaning of tokens (via token embeddings).

2. Where: The position of tokens in the sequence (via positional embeddings).

Example

Consider the sentence "I love NLP":

• Token Embeddings: [[0.1,0.3,0.4],[0.2,0.6,0.5],[0.7,0.9,0.8]]

• Positional Embeddings: [[0.01,0.02,0.03],[0.04,0.05,0.06],[0.07,0.08,0.09]]

Final Input Embeddings:

[[0.11,0.32,0.43],[0.24,0.65,0.56],[0.77,0.98,0.89]]

Trainable vs. Fixed Positional Embeddings

• Fixed Positional Embeddings:

  • Sinusoidal embeddings are fixed and not updated during training.

  • Advantage: Simpler and generalizable.

  • Example: Used in the original transformer model.

• Learnable Positional Embeddings:

  • Each position is assigned a unique embedding, which is learned during training.

  • Advantage: More flexible for specific tasks.

  • Example: Used in models like GPT and BERT.

Libraries and Implementation

• TensorFlow/Keras: You can create sinusoidal positional embeddings using tf.keras.layers.Layer. Alternatively, KerasNLP provides high-level APIs for transformer models.

• PyTorch: PyTorch allows you to manually construct embeddings using torch functions or use pre-built models from torch.nn.

• Hugging Face Transformers: Hugging Face provides ready-to-use transformer models, including pre-trained positional embeddings.

Conclusion

Token embeddings and positional embeddings are the foundation of transformer models. Token embeddings capture meaning, while positional embeddings encode order, together enabling models to process sequences effectively. By understanding their mathematical and practical implementations, we can better appreciate the inner workings of transformers and build more effective NLP solutions. Whether using fixed sinusoidal embeddings or trainable embeddings, these components are indispensable in modern natural language processing.

Reference Research Paper: Attention Is All You Need

1. It lacks the context of the query.

2. It hallucinates and confidently produces a response.

To overcome this challenges, we can provide our own data from different sources within the organisations. The own data that we provide to LLMs are termed as non-parametric data. Now we ask LLM to verify our own data and the capabilities of pretrained data together when it produces an response to a query. This is termed as Retrieval Augmented Generation.

Different Research Paradigms of RAG

The research on RAG is continuously evolving. It can be categrized into 3 stages.

1. Naive RAG

2. Advanced RAG

3. Modular RAG

Naive RAG

Most of the initial research was done in this space.It is a traditional process that includes indexing,retrieval and Generation. It is called as Retrieve-Read framework.

Indexing involves cleaning and extracting the raw data from various sources. Due to LLM models context limitations, we need to break the text to smaller chunks. The chunks are encoded into vector representation using the embedding model and stored in a vector database. This step is very important as in retrieval stage, we use it to do similarity searches.

When we receive an input query, we use the same encoding that was used during Indexing to transform the query to vector representation. Now, we compute the similarity scores with the query and stored chunks vectors. We can extract configurable number of chunks that are matched with the query. This context is passed to LLMs as a context information along with the original input query.

In generation phase, the LLM uses the context information and the original query to formulate the response. We can also pass extra conversational history, so the LLM can use it further to enhance its response. The response will depend on the parametric knowledge that the LLM model has.

Drawbacks

Some of the drawback are

1. Retrieval challenge - If we ask same question multiple times, it may respond with different answers everytime. It struggles with accuracy and also it cannot recall what the answer were given previously. So, we may get different chunks in the context and miss crucial informations.

2. Generation challenges - The model may still hallucinate and it may produce a response that is not provided in the context. It shows bias towards specific outputs. It may not generate quality response and answers are not reliable, if it generates different response every time with the same question.

3. Data challenges - The context that was provided may not be sufficient for the models to provide a response when we do different tasks with the same data. Specifically, if we have data from different sources, it may provide repeated similar answers for different tasks. Sometimes, the models will just respond to the retrieved content without providing additional content from the parametric data it has been pretrained.

Advanced RAG

It is an improvement on the limitations that we have in Naive RAG. For quality of retrieval, it introduces pre-retrieval and post-retrieval steps. To improve the indexing, it uses some of advanced techniques like sliding window, fine-grained segementation, metadata.

During pre-retrieval, its main focus is to optimize the indexing and original input query. The query optimization will help in making the original question more clearer. It helps in retrieval process, if the context is accurate. It involves query rewriting, query-transfomation, query-expansion

During post-retrieval, It mainly focuses on reranking the chunks and compressing of context, so that only relevant information are passed to the models. It ensures that LLM does not get the unrelated context that may overweigh the original intentions.

Modular RAG

As more advanced in RAG space is evolving, we are leaning more towards the Modular approach of tackling RAG systems. The main focus is adaptability. It supports complex and efficient retrival mechanisms. It adds different kinds of the specialized modules to enhance particular components in RAG systems.

• Search Module is added to enhance the similarity searches and also improving the retrieval through fine tuning.The search module helps in efficient searching across different data sources like search engines, databases, knowledge graphs.

• RAGFusion adds a capability for expanding user queries and can do parallel vector searches. Thus it will have more insights in the provided data.

• Memory module adds a capability to the LLM retrival memory. It can retain most of the informations in memory and it will have relavant information more closer to the data. It adds more depth to the knowledge base.

• Routing module helps in getting relavant information by selecting most appropriate data source to a given input query. It can combine from various data sources or access specific database.

• Predict module enhances LLM capability to predict accurately, so the models can filter out unnecessary information to the input query.

By using these modules, it makes significant improvements in retrieval quality, thereby advancing the capabilities of RAG systems.

Reference Source from research paper: https://arxiv.org/abs/2312.10997

Understanding Large Language Models (LLMs)

Large Language Models are advanced machine-learning systems trained on extensive datasets. These models use techniques in deep learning to understand, generate, and analyze human language, making them invaluable for natural language processing (NLP) tasks. LLMs rely on extensive data and millions (or billions) of parameters to generate responses that mimic human language patterns.

While LLMs have remarkable abilities, they’re not perfect. LLMs have significant limitations in retaining context over extended interactions and recalling information precisely, especially as the dialogue grows. To overcome these limitations, Retrieval Augmented Generation (RAG) was developed as a solution to improve the memory, accuracy, and relevance of information provided by LLMs.

What is Retrieval Augmented Generation (RAG)?

Retrieval Augmented Generation combines traditional information retrieval techniques with generative language models. In simpler terms, RAG integrates two key AI processes:

• Retrieval: This component searches external databases or knowledge bases to fetch relevant information based on the prompt or question.

• Generation: This component generates responses based on the retrieved information, enhancing the response's relevance and accuracy.

By merging these two, RAG allows AI models to pull in highly specific information not contained within the original training dataset, allowing them to answer questions with increased precision and reliability.

Why Do We Need Retrieval Augmented Generation?

While LLMs demonstrate immense potential, they face inherent challenges:

Limited Contextual Memory

LLMs have limited capacity for maintaining context, especially in lengthy interactions. They are prone to “forgetting” or distorting details from previous interactions as the conversation grows. RAG addresses this limitation by retrieving contextually relevant information each time a query is made, providing the model with accurate information even in extended interactions.

Outdated or Static Knowledge

LLMs are trained on datasets available up to a specific cutoff date, making them unable to account for more recent developments or niche knowledge areas. With RAG, LLMs can retrieve and integrate up-to-date information from external databases, ensuring that responses remain relevant and accurate.

Lack of Precision in Complex Queries

LLMs may produce general responses to queries that require domain-specific knowledge. RAG improves accuracy in such cases by fetching precise, domain-specific data relevant to the user's question, ensuring that even complex queries are met with appropriate responses.

How Retrieval Augmented Generation Overcomes These Challenges

Incorporating a retrieval mechanism within an LLM system addresses these challenges effectively. RAG enables systems to connect with external resources, making it possible to:

• Recall precise information on demand.

• Dynamically update the response with the most relevant content.

• Maintain the conversational flow by bridging gaps in the model’s inherent knowledge.

With RAG, models can generate responses that are not only coherent and contextually appropriate but also rich in current and relevant data.

Examples of RAG-Enabled Use Cases

The RAG technique has enabled several advanced applications across industries:

• Customer Support Systems: Chatbots equipped with RAG can fetch real-time information, making them more effective in handling user-specific queries.

• Medical Research: RAG-powered models can search through vast repositories of medical journals, providing healthcare professionals with accurate and up-to-date information.

• Legal Research: RAG allows legal professionals to quickly retrieve relevant case laws and precedents, making their work more efficient.

• Finance and Investment Analysis: RAG can enhance investment platforms by providing financial analysts with the latest market insights and trends.

Understanding the Challenge: Concurrent Writes in Amazon S3

When multiple clients attempt to write to the same S3 object (i.e., the same key) at the same time, S3 will accept all requests. However, only the data from the request with the most recent timestamp will be stored. This means that earlier writes can be overwritten by later ones, leading to potential data loss or inconsistency.

Moreover, S3 does not support native object locking. This absence of built-in locking mechanisms means that you need to handle concurrency at the application level. Below, we’ll discuss several strategies you can implement to manage concurrent writes and achieve data consistency.

1. Application-Level Locking

Application-level locking involves creating your own locking mechanism to control access to an S3 object. This ensures that only one process can write to an object at a time, avoiding race conditions.

Implementation Example: Lock Management Using DynamoDB

You can use a DynamoDB table to manage locks. Each S3 object key corresponds to an entry in the table. Before writing to S3, your application checks if the object is locked. If it’s not locked, the application creates a lock entry in DynamoDB and then writes to S3.

import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.resource('dynamodb')
s3 = boto3.client('s3')

lock_table = dynamodb.Table('S3ObjectLocks')

def acquire_lock(key):
    try:
        lock_table.put_item(
            Item={'S3Key': key, 'Lock': True},
            ConditionExpression='attribute_not_exists(S3Key)'
        )
        return True
    except ClientError as e:
        print(f"Lock not acquired: {e.response['Error']['Message']}")
        return False

def release_lock(key):
    lock_table.delete_item(Key={'S3Key': key})

def write_to_s3(bucket, key, data):
    if acquire_lock(key):
        try:
            s3.put_object(Bucket=bucket, Key=key, Body=data)
            print(f"Successfully wrote to {key}")
        finally:
            release_lock(key)
    else:
        print(f"Failed to acquire lock for {key}")

# Example usage
write_to_s3('my-bucket', 'my-object', 'Hello World!')

Pros:

• Provides fine-grained control over access to S3 objects.

• It can be tailored to specific use cases.

Cons:

• Adds complexity to the application.

• Potential for deadlocks or orphaned locks if not carefully managed.

2. Using S3 Versioning

S3 Versioning allows you to keep multiple versions of an object. When versioning is enabled, each time you write to an object, S3 stores it as a new version instead of overwriting the existing one. This ensures that no data is lost if concurrent writes occur.

Implementation Example: Enabling Versioning and Retrieving Versions

import boto3

s3 = boto3.client('s3')

# Enable versioning on a bucket
s3.put_bucket_versioning(
    Bucket='my-bucket',
    VersioningConfiguration={
        'Status': 'Enabled'
    }
)

# Upload multiple versions of an object
s3.put_object(Bucket='my-bucket', Key='my-object', Body='First version')
s3.put_object(Bucket='my-bucket', Key='my-object', Body='Second version')

# Retrieve all versions of the object
versions = s3.list_object_versions(Bucket='my-bucket', Prefix='my-object')
for version in versions.get('Versions', []):
    print(f"Version ID: {version['VersionId']}, Data: {s3.get_object(Bucket='my-bucket', Key='my-object', VersionId=version['VersionId'])['Body'].read().decode()}")

Pros:

• Prevents data loss by storing all versions.

• Allows you to revert to previous versions if necessary.

Cons:

• Storage costs can increase due to multiple versions.

• Additional logic is required to manage and clean up old versions if necessary.

3. Pre-Signed URLs with Conditional Headers

Pre-signed URLs allow you to generate a temporary URL that grants permission to perform specific operations on an S3 object. By using conditional headers, you can ensure that the write operation only proceeds if certain conditions are met, such as the object not having been modified since it was last accessed.

Implementation Example: Conditional PUT Request with Pre-Signed URL

import boto3
from botocore.client import Config

s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

# Generate a pre-signed URL that only allows the write if the object's ETag matches
url = s3.generate_presigned_url(
    ClientMethod='put_object',
    Params={
        'Bucket': 'my-bucket',
        'Key': 'my-object',
        'ContentType': 'text/plain'
    },
    ExpiresIn=3600,
    HttpMethod='PUT',
    Conditions=[
        {"If-Match": "d41d8cd98f00b204e9800998ecf8427e"}  # Replace with actual ETag of the object
    ]
)

# Use the pre-signed URL to upload the object (e.g., with requests or a web client)
print(f"Pre-signed URL: {url}")

Pros:

• Provides a lightweight way to conditionally allow operations.

• Doesn’t require managing locks or versioning.

Cons:

• Requires careful handling of conditions like ETags or timestamps.

• Only suitable for scenarios where condition-based writes are sufficient.

4. Using AWS S3 Object Lock for Compliance/Governance

AWS S3 Object Lock is primarily designed to prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely, which is useful for compliance and data retention. While not a direct solution for concurrency, it can ensure data immutability.

Implementation Example: Enabling Object Lock

import boto3

s3 = boto3.client('s3')

# Enable Object Lock on the bucket
s3.put_object_lock_configuration(
    Bucket='my-bucket',
    ObjectLockConfiguration={
        'ObjectLockEnabled': 'Enabled',
        'Rule': {
            'DefaultRetention': {
                'Mode': 'GOVERNANCE',
                'Days': 30
            }
        }
    }
)

# Upload an object with Object Lock
s3.put_object(
    Bucket='my-bucket',
    Key='my-object',
    Body='Content to protect',
    ObjectLockMode='GOVERNANCE',
    ObjectLockRetainUntilDate='2024-12-31T00:00:00.000Z'
)

Pros:

• Ensures data immutability, which can prevent accidental overwrites.

• Useful for regulatory compliance and data retention.

Cons:

• Not specifically designed for managing concurrent writes.

• Requires careful management to avoid unintended data retention or access issues.

Choosing the Right Strategy

Each strategy has its pros and cons, and the right approach depends on your specific use case:

• Application-Level Locking is ideal when you need fine-grained control over who can write to an object and when.

• S3 Versioning is best when you want to keep all versions of an object and manage them post-facto.

• Pre-Signed URLs with Conditional Headers are effective for lightweight, condition-based write controls.

• S3 Object Lock is more about ensuring data immutability, especially in compliance scenarios.

In some cases, you may need to combine these strategies. For example, you could use S3 Versioning alongside Application-Level Locking to ensure both consistency and auditability of writes to your S3 bucket.

Conclusion

Managing concurrent writes and object locking in Amazon S3 requires thoughtful consideration and careful implementation. While S3 does not provide native support for object locking or handling concurrency, the strategies outlined above can help you achieve the desired consistency and reliability in your application.

By implementing these techniques, you can effectively manage concurrent writes, avoid race conditions, and ensure that your data remains consistent and secure in Amazon S3.

Introduction

The Saga pattern is a design pattern for managing distributed transactions across multiple microservices or components in a distributed system. Traditional two-phase commit protocols can become complex and brittle in distributed environments, where failures are common, and coordinating transactions across multiple services can lead to increased latency and potential for deadlock.

The Saga pattern addresses these challenges by breaking down a long-lived transaction into a series of smaller, independent transactions, known as saga steps. Each saga step represents a unit of work that can be completed or compensated independently of other steps. If a failure occurs during the execution of a saga, compensating transactions are used to undo the work performed by previously completed steps, ensuring eventual consistency.

Implementing the Saga Pattern with AWS Step Functions

In AWS Step Functions, you can model a saga as a state machine where each state represents a saga step. The state machine transitions between states based on the success or failure of each step, executing compensating transactions as needed to maintain consistency.

To implement the Saga pattern with AWS Step Functions:

1. Define Saga Steps: Identify the individual steps of the saga and map them to state machine states.

2. Handle Success and Failure: Define transitions between states based on the success or failure of each step.

3. Implement Compensating Transactions: For each saga step, define compensating transactions to undo the effects of completed steps in case of failure.

4. Manage State and Data: Use input and output data to track the state of the saga and pass information between steps.

Design Considerations

When designing sagas with AWS Step Functions, consider the following best practices:

• Error Handling: Handle errors gracefully at each step of the saga and implement retry logic where appropriate.

• Timeouts: Set timeouts for each step to prevent long-running tasks from blocking the execution of the saga.

• Idempotency: Ensure that each step of the saga is idempotent to handle retries and duplicate requests.

• Data Consistency: Use consistent data models and transactional updates to maintain data consistency across saga steps.

Walkthrough Example: Implementing a Saga Pattern for Order Processing

In this example, we'll implement a saga pattern to manage the order processing workflow in an e-commerce system. The order processing workflow consists of three main steps:

1. Place Order: Initiates the order processing and reserves inventory.

2. Charge Payment: Charges the customer's payment method.

3. Fulfill Order: Ships the ordered items to the customer.

If any step fails, we'll need to compensate by undoing the work of the previously completed steps.

Step 1: Define the State Machine

First, let's define the state machine using AWS Step Functions' JSON-based state language. Each state represents a step in the order processing saga.

{
  "Comment": "Order Processing Saga",
  "StartAt": "PlaceOrder",
  "States": {
    "PlaceOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:PlaceOrderFunction",
      "Next": "ChargePayment",
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "Next": "CompensatePlaceOrder"
      }]
    },
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:ChargePaymentFunction",
      "Next": "FulfillOrder",
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "Next": "CompensateChargePayment"
      }]
    },
    "FulfillOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:FulfillOrderFunction",
      "End": true,
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "Next": "CompensateFulfillOrder"
      }]
    },
    "CompensatePlaceOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:CompensatePlaceOrderFunction",
      "End": true
    },
    "CompensateChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:CompensateChargePaymentFunction",
      "End": true
    },
    "CompensateFulfillOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:CompensateFulfillOrderFunction",
      "End": true
    }
  }
}

Step 2: Implement Lambda Functions

Next, implement the Lambda functions for each step of the saga:

• PlaceOrderFunction: Initiates the order processing and reserves inventory.

import boto3

def place_order(order_details):
    # Dummy logic to place order and reserve inventory
    order_id = '123456'
    return order_id

def lambda_handler(event, context):
    order_details = event['order_details']
    order_id = place_order(order_details)
    return {
        'orderId': order_id
    }

• ChargePaymentFunction: Charges the customer's payment method.

import boto3

def charge_payment(order_id, payment_details):
    # Dummy logic to charge payment
    payment_id = 'PAY-789'
    return payment_id

def lambda_handler(event, context):
    order_id = event['orderId']
    payment_details = event['payment_details']
    payment_id = charge_payment(order_id, payment_details)
    return {
        'paymentId': payment_id
    }

• FulfillOrderFunction: Ships the ordered items to the customer.

import boto3

def fulfill_order(order_id):
    # Dummy logic to fulfill order and ship items
    return True

def lambda_handler(event, context):
    order_id = event['orderId']
    success = fulfill_order(order_id)
    return {
        'success': success
    }

• CompensatePlaceOrderFunction: Compensates for the PlaceOrder step (e.g., releases reserved inventory).

import boto3

def compensate_place_order(order_id):
    # Dummy logic to release reserved inventory
    return True

def lambda_handler(event, context):
    order_id = event['orderId']
    success = compensate_place_order(order_id)
    return {
        'success': success
    }

• CompensateChargePaymentFunction: Compensates for the ChargePayment step (e.g., refunds the charged amount).

import boto3

def compensate_charge_payment(payment_id):
    # Dummy logic to refund charged amount
    return True

def lambda_handler(event, context):
    payment_id = event['paymentId']
    success = compensate_charge_payment(payment_id)
    return {
        'success': success
    }

• CompensateFulfillOrderFunction: Compensates for the FulfillOrder step (e.g., cancels the order shipment).

import boto3

def compensate_fulfill_order(order_id):
    # Dummy logic to cancel order shipment
    return True

def lambda_handler(event, context):
    order_id = event['orderId']
    success = compensate_fulfill_order(order_id)
    return {
        'success': success
    }

Step 3: Test the Saga

Once the state machine and Lambda functions are implemented, test the saga by simulating different scenarios:

• Scenario 1 (Success): All steps are completed successfully.

Input

{
  "order_details": {
    "item_id": "123",
    "quantity": 2,
    "customer_id": "456"
  },
  "payment_details": {
    "amount": 100,
    "payment_method": "credit_card"
  }
}

Output

{
  "orderId": "123456",
  "paymentId": "PAY-789",
  "success": true
}

• Scenario 2 (Partial Failure): PlaceOrder and ChargePayment are completed successfully, but FulfillOrder fails. Ensure that compensating transactions are triggered to undo the completed steps.

Input

{
  "order_details": {
    "item_id": "789",
    "quantity": 1,
    "customer_id": "123"
  },
  "payment_details": {
    "amount": 50,
    "payment_method": "paypal"
  }
}

Output

{
  "success": false,
  "compensatePlaceOrder": true,
  "compensateChargePayment": true
}

• Scenario 3 (Complete Failure): PlaceOrder fails. Ensure that compensating transactions are triggered to undo any partially completed steps.

Input

{
  "order_details": {
    "item_id": "789",
    "quantity": 3,
    "customer_id": "123"
  },
  "payment_details": {
    "amount": 150,
    "payment_method": "credit_card"
  }
}

Output

{
  "success": false,
  "compensateChargePayment": true
}

Conclusion

In conclusion, implementing the Saga pattern using AWS Step Functions provides a powerful mechanism for managing distributed transactions in serverless applications. By breaking down long-lived transactions into smaller, independent steps and orchestrating them with Step Functions, you can achieve eventual consistency and fault tolerance in your distributed systems. Experiment with different saga designs and workflows to find the best approach for your application needs.

AWS Step Functions offers an array of testing and debugging tools. In this blog post, we'll focus on the TestState API.

TestState API helps us to test a single state in the state machine workflow. So, we can test a state without creating a state machine and execute the complete flow. This helps resolve common errors such as path issues, and input/output issues at the development time and at a fast pace.

Supported States

The following states are supported for TestState.

• Pass

• Wait

• Task

• Choice

• Succeed

• Fail

Informative levels

Step Functions offers three distinct inspection levels: INFO, DEBUG, and TRACE. The different inspection levels, provide us the right amount of detail to suit our debugging and validation needs.

• INFO: At this level, we receive essential information about the state execution, including the status and the next state to transition to.

• DEBUG: At this level, it provides additional insights into the state execution. Beyond the status and next state, you gain access to intermediate details, especially useful when using input and output data processing filters like InputPath or ResultPath.

• TRACE: At this level, it not only includes the status and next state but also provides deep into the intermediate and final data processing results.

Using AWS CLI with TestState

We can use AWS CLI with TestState for testing each state of the state machine in silo.

Suppose, we have a passstate in our Stepfunction as shown below.

{
      "Type": "Pass",
      "Next": "Second State",
      "Parameters": {
        "payload.$": "$.customer.firstName"
      }
    }

To test this state, we can use below AWS CLI command

aws stepfunctions test-state \
    --definition '{
      "Type": "Pass",
      "Next": "Second State",
      "Parameters": {
        "payload.$": "$.customer.firstName"
      }
    }' \
    --role-arn arn:aws:iam::1234567843:role/example \
    --input '{"customer": {
        "firstName": "rahul",
        "lastName": "lokurte"
    }
    }'

The output is as shown below.

{
  "nextState": "Second State",
  "output": "{\"payload\":\"rahul\"}",
  "status": "SUCCEEDED"
}

Conclusion

As seen in this short blog, we can use the TestState API to test the individual states in a state machine which opens up a lot of opportunities for testing the step functions thoroughly.

1. AWS CloudFormation
2. AWS Cloud Development Kit

With CloudFormation, we have to write a lot of YAML templates or JSON files. As AWS adds more services, we have to add more files to CloudFormation. It becomes difficult to work with lots of files. YAML/JSON is based on data serialization and not an actual programming language. The AWS CDK will overcome the limitations of cloud formation by enabling the reuse of code and proper testing.

AWS CDK is a framework that allows developers to use familiar programming languages to define AWS cloud infrastructure and provision it. CDK provides the Constructs cloud component that cover many of the AWS services and features. It helps us to define our application infrastructure at high level.

we will create a Lambda Function and the infrastructure around lambda function using AWS CDK.

Create a new directory on your system.

mkdir cdk-greetapp && cd cdk-greetapp

We will use cdk init to create a new Javascript CDK project:

cdk init --language javascript

The cdk init command creates a number of files and folders inside the cdk-greetapp directory to help us organize the source code for your AWS CDK app.

We can list the stacks in our app by running the below command. It will show CdkGreetappStack.

_$ cdk ls
CdkGreetappStack

Let us install AWS lambda construct library.

npm install @aws-cdk/aws-lambda

Edit the file ***lib/cdk-greetapp-stack.js to create an AWS lambda resource as shown below.

const cdk = require("@aws-cdk/core");
const lambda = require("@aws-cdk/aws-lambda");

class CdkGreetappStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);
    // defines an AWS Lambda resource
    const greet = new lambda.Function(this, "GreetHandler", {
      runtime: lambda.Runtime.NODEJS_14_X,
      code: lambda.Code.fromAsset("lambda"),
      handler: "greet.handler",
    });
  }
}

module.exports = { CdkGreetappStack };

• Lambda Function uses NodeJS 14.x runtime
• The handler code is loaded from the directory named lambda where we will add the lambda code.
• The name of the handler function is greet.handler where greet is the name of file and handler is exported function name.

Lets create a directory name lambda in root folder and add a file greet.js.

mkdir lambda
cd lambda
touch greet.js

Add the lambda code to greet.js

exports.handler = async function (event) {
  console.log("request:", JSON.stringify(event, undefined, 2));
  let response = {
    statusCode: 200,
    body: `Hello ${event.path}. Welcome to CDK!`,
  };
  return response;
};

Before deploying the AWS resource, we can take a look on what resources will be getting created by using below command.

cdk diff

NOTE: If we have multiple profiles set in our system, we need to tell cdk to look into particular profile. This can be done, by adding below key-value in cdk.json which was generated when we created a CDK project.

"profile": "<YOUR_PROFILE_NAME>"

Now, once we are ok with the resources which will be created, we can deploy it using below command

cdk deploy

Let us open the AWS Lambda console

Select Amazon API Gateway AWS Proxy from the Event template list.

Click on Test, we can see that, we get the proper response as shown below.

Conclusion

we saw how to create a lambda function and also lambda resource by using AWS Cloud Development Kit. We also saw various commands related to CDK for initiating projects, deploying the resources to AWS. The code repository link is here

If you want to do hands-on, I have created a Youtube Video. You can check out here

In modern cloud development, infrastructure-as-code has become a crucial practice. Tools like Terraform enable developers and operations teams to define and manage their infrastructure in a declarative manner. However, provisioning and testing infrastructure in a live AWS environment can be time-consuming and costly. That's where LocalStack comes in. LocalStack provides a local development environment that emulates various AWS services, allowing developers to create and test AWS resources locally. In this blog post, we'll explore how to leverage Terraform and LocalStack together for local development and testing of AWS infrastructure.

What is LocalStack?

LocalStack is an open-source development tool that provides a local environment for emulating various AWS services. It allows developers to create and test AWS resources locally, without the need for a live AWS account or incurring any costs. With LocalStack, you can emulate services such as AWS Lambda, S3, DynamoDB, SQS, SNS, EventBridge, and many more.

Getting Started with LocalStack

To get started with LocalStack, follow these steps to install and set it up on your local development machine:

1. Prerequisites: Ensure that you have Docker installed on your machine. LocalStack runs inside a Docker container, so having Docker installed is a requirement.

2. Install LocalStack: Open your terminal or command prompt and run the following command. We are installing Localstack CLI, but there are multiple ways to install localstack. Check the official guide https://docs.localstack.cloud/getting-started/installation/#localstack-cli

    python -m pip install localstack
   

3. Start LocalStack: To start LocalStack with the default configuration, run the following command in your terminal:

    localstack start
   

Getting Started with Terraform

Terraform is a powerful infrastructure-as-code tool that allows you to define and manage your infrastructure using declarative configuration files. In this section, we'll guide you through the process of installing Terraform and writing Terraform configuration files to manage your AWS infrastructure.

Installing Terraform

To get started with Terraform, follow these steps to install it on your local development machine:

1. Download and Install Terraform: Visit the official Terraform website at terraform.io and download the appropriate Terraform binary for your operating system.

2. Verify Installation: Open a new terminal or command prompt window and run the following command to verify that Terraform is installed correctly:

    terraform --version
   

   You should see the Terraform version displayed in the output, indicating a successful installation.

tflocal: Terraform with LocalStack

tflocal - a small wrapper script to run Terraform against LocalStack. The tflocal command line interface can be installed using pip as shown below.

pip install terraform-local

Application Development with Terraform and LocalStack

In this section, we'll explore how to develop an application locally using Terraform and LocalStack. We'll deploy a sample AWS Lambda function that performs an addition operation on two numbers. By leveraging Terraform and LocalStack together, we can develop and test our application locally without incurring any AWS costs.

Our sample application consists of the following components:

1. Lambda Function: The core logic of the application resides in an AWS Lambda function. This function takes two input numbers from the Eventbridge event, adds them together, and returns the result.

2. EventBridge Rule: We'll use AWS EventBridge to schedule the execution of our Lambda function. We'll configure a scheduled rule in EventBridge to trigger the Lambda function every minute.

Terraform Configuration

1. Create a new directory: Create a new directory on your local machine to store the Terraform configuration files for the sample application. Navigate to this directory in your terminal.

mkdir scheduler-localstack && cd scheduler-localstack

1. Create the Terraform configuration files: Create a new file named main.tf and open it in a text editor. Add the following Terraform configuration to the file:

provider "aws" {
  region = "us-east-1" # Replace with your desired region
}

resource "aws_cloudwatch_event_rule" "scheduled_event" {
  name                = "my_scheduled_event"
  description         = "My scheduled event rule"
  schedule_expression = "rate(1 minutes)"
  is_enabled = true
}

resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.scheduled_event.name
  target_id = "my_lambda_function"
  arn       = aws_lambda_function.my_lambda_function.arn
  input     = <<JSON
            {
  "number1": 4,
  "number2": 5
}
JSON
}

data "archive_file" "my_lambda_function_archive" {
  type        = "zip"
  source_file = "${path.module}/code/lambda_function.py"
  output_path = "my_lambda_function.zip"
}


resource "aws_lambda_function" "my_lambda_function" {
  function_name    = "my_lambda_function"
  role             = aws_iam_role.lambda_role.arn
  handler          = "lambda_function.lambda_handler"
  runtime          = "python3.8"
  filename         = "my_lambda_function.zip"
  source_code_hash = data.archive_file.my_lambda_function_archive.output_base64sha256
}

resource "aws_lambda_permission" "allow_cloudwatch" {
  statement_id  = "AllowCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.my_lambda_function.arn
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.scheduled_event.arn
}

resource "aws_iam_role" "lambda_role" {
  name = "my_lambda_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

  inline_policy {
    name = "cloudwatch_logs_policy"

    policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
EOF
  }
}

resource "aws_cloudwatch_log_group" "lambda_log_group" {
  name = "/aws/lambda/${aws_lambda_function.my_lambda_function.function_name}"
}

output "log_group_name" {
  value = aws_cloudwatch_log_group.lambda_log_group.name
}

Below is a brief overview of the configuration:

1. Provider Configuration: Defines the provider configuration for AWS. The aws the provider is responsible for authenticating and interacting with AWS services. In this case, we specify the desired region as us-east-1.

2. CloudWatch Event Rule: Defines an AWS CloudWatch Event Rule. The rule is used to schedule the execution of an action, in this case, our Lambda function. We provide a name and description for the rule and set the schedule_expression to "rate(1 minutes)", which triggers the rule every minute. The is_enabled line will enable the rule immediately upon deployment.

3. CloudWatch Event Target: Defines the target for the CloudWatch Event Rule. The rule attribute references the name of the CloudWatch Event Rule defined earlier. The target_id is set to "my_lambda_function" that identify the Lambda function as the target. The arn attribute points to the ARN (Amazon Resource Name) of the Lambda function. The input field specifies the input data to be passed to the Lambda function. In this case, it's a JSON object with number1 and number2 as inputs.

4. Lambda Function Code Archive: The archive_file data source creates a zip archive of the Lambda function code. The source_file attribute specifies the path to the lambda_function.py file, which contains the code for the Lambda function. The resulting zip archive is named my_lambda_function.zip and will be used in the subsequent resource block.

5. Lambda Function: Defines the AWS Lambda function. The function_name attribute specifies the name of the Lambda function. The role attribute references the ARN of the IAM role associated with the Lambda function. The handler attribute specifies the entry point for the Lambda function code. The runtime attribute specifies the runtime environment for the Lambda function, in this case, Python 3.8. The filename attribute specifies the name of the zip file containing the Lambda function code. The source_code_hash attribute references the output of the archive_file data source, ensuring that the Lambda function is updated when the code changes.

6. Lambda Function Permission: Grants permission to CloudWatch Events to invoke the Lambda function. The statement_id attribute is a unique identifier for the permission statement. The action attribute specifies the action that is allowed, which is lambda:InvokeFunction in this case. The function_name attribute references the ARN of the Lambda function. The principal attribute specifies the entity that is granted permission, which is events.amazonaws.com for CloudWatch Events. The source_arn attribute references the ARN of the CloudWatch Event Rule.

7. IAM Role for Lambda: Defines an IAM role for the Lambda function. The name attribute specifies the name of the role. The assume_role_policy attribute defines the trust relationship policy document, allowing the Lambda service to assume this role. The inline_policy block defines an inline IAM policy attached to the role. In this case, it grants permissions for CloudWatch Logs to create log groups, create log streams, and put log events.

8. AWS CloudWatch Log Group: Creates an AWS CloudWatch Log Group for the Lambda function. The name attribute is set using an interpolated string, "/aws/lambda/${aws_lambda_function.my_lambda_function.function_name}" which represents the desired naming convention for the log group. It includes the function name of the AWS Lambda function.

9. Terraform Output: output block defines an output variable named log_group_name that displays the name of the AWS CloudWatch Log Group created for the Lambda function. The value is set as aws_cloudwatch_log_group.lambda_log_group.name, which retrieves the name attribute of the aws_cloudwatch_log_group resource.

Lambda Function:

The core functionality of our application is implemented using an AWS Lambda function. The lambda function is responsible for processing the input event, performing the desired computation, and returning the result. The lambda function, located at the root of our project under the code folder, contains the following code:

import boto3
import json

def lambda_handler(event, context):
    print("Lambda Function Name: ", context.function_name)
    print("Event is: ", event)
    number1 = event["number1"]
    number2 = event["number2"]

    result = number1 + number2

    print("Sum:", result)

    return {"statusCode": 200, "body": json.dumps({"result": result})}

The lambda function is written in Python and consists of a single handler function named lambda_handler. This function is automatically invoked by AWS Lambda when an event triggers its execution.

Upon invocation, the function receives two parameters: event and context. The event parameter contains the input data passed to the lambda function, while the context parameter provides information about the execution context and runtime environment.

In our lambda function, we first print the name of the lambda function using context.function_name and the input event using event. This allows us to verify the function execution logs and ensure the correct inputs are received.

Next, we extract the values of number1 and number2 from the event payload. These values represent the numbers to be added together.

The lambda function performs the addition operation and stores the result in the result variable.

We then print the calculated sum using print("Sum:", result). This line helps us verify that the computation was successful and the expected result is obtained.

Finally, we construct a JSON response with a status code of 200 and a body containing the calculated result.

The lambda function encapsulates the business logic of our application and is executed by LocalStack in response to the scheduled event defined earlier in the Terraform configuration.

Deploying the Infrastructure with Terraform and LocalStack

Now that we have our Terraform configuration file ready, let's deploy the infrastructure using Terraform and LocalStack. Follow the steps below:

1. Open your command line interface (CLI) or terminal.

2. Navigate to the directory where you have saved the Terraform configuration file.

3. Initialize the Terraform working directory by running the following command:

    tflocal init
   

   This command initializes the working directory and downloads the necessary provider plugins defined in your Terraform configuration.

4. Once the initialization is complete, you can run the following command to deploy the infrastructure:

    tflocal apply
   

   Terraform will read the configuration file, create the necessary resources, and prompt you to confirm the execution. Type yes and press Enter to proceed.

5. Terraform will start provisioning the infrastructure components specified in the configuration file. It will create the CloudWatch Event Rule, Lambda function, IAM role, and associated permissions.

6. Once the command is executed, the log_group_name output variable will be displayed, which contains the name of the AWS CloudWatch Log Group associated with the Lambda function. This output value can be noted down and used to verify the application's behavior and inspect the logs generated by the Lambda function in the next section.

Verifying the Application

To ensure a seamless verification process of the application, let's begin by installing the awslocal command-line tool. awslocal is a utility that is a thin wrapper around the aws command line interface for use with LocalStack.

To install awslocal, follow the steps below:

Open your command-line interface and execute the following command to install awslocal using pip:

pip install awscli-local

This command will download and install the awslocal package. With awslocal installed, we are now ready to proceed with the verification of our application.

Now that we have deployed our infrastructure using Terraform and LocalStack, let's verify the application by checking the result printed by the Lambda function.

• Open your command line interface (CLI) or terminal.

• Make sure that LocalStack is still running and that the necessary AWS services are emulated.

• Use the AWS CLI to check the logs generated by the Lambda function. Run the following command:

    awslocal --endpoint-url=http://localhost:4566 logs describe-log-groups
  

  We get the output as shown below

    {
        "logGroups": [
            {
                "logGroupName": "/aws/lambda/my_lambda_function",
                "creationTime": 1685365365216,
                "metricFilterCount": 0,
                "arn": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/my_lambda_function:*",
                "storedBytes": 7639
            }
        ]
    }
  

  This command will list all the log groups available in LocalStack.

• Identify the log group associated with the Lambda function. It should have a name similar to my_lambda_function. Note the name of the log group for the next step.

• Fetch the logs for the Lambda function by running the following command:

    awslocal --endpoint-url=http://localhost:4566 logs filter-log-events --log-group-name /aws/lambda/my_lambda_function
  

The command will display the log events generated by the Lambda function. Look for the log entry that includes the result of the addition.

 {
            "logStreamName": "2023/05/29/[$LATEST]0a013874ae1b00b65f24f5930a8da077",
            "timestamp": 1685366855633,
            "message": "START RequestId: 8acdc34d-f5cb-4f30-97d2-97522f0d22dc Version: $LATEST",
            "ingestionTime": 1685366855727,
            "eventId": "144"
        },
        {
            "logStreamName": "2023/05/29/[$LATEST]0a013874ae1b00b65f24f5930a8da077",
            "timestamp": 1685366855644,
            "message": "Lambda Function Name:  my_lambda_function",
            "ingestionTime": 1685366855727,
            "eventId": "145"
        },
        {
            "logStreamName": "2023/05/29/[$LATEST]0a013874ae1b00b65f24f5930a8da077",
            "timestamp": 1685366855656,
            "message": "Event is:  {'number1': 4, 'number2': 5}",
            "ingestionTime": 1685366855727,
            "eventId": "146"
        },
        {
            "logStreamName": "2023/05/29/[$LATEST]0a013874ae1b00b65f24f5930a8da077",
            "timestamp": 1685366855668,
            "message": "Sum: 9",
            "ingestionTime": 1685366855727,
            "eventId": "147"
        }

The log events show the following details:

• The start of the Lambda function execution with a unique RequestId and the corresponding version.

• Confirmation of the Lambda function name, which in this case is my_lambda_function.

• The event data is passed to the Lambda function, which is a dictionary containing the numbers 4 and 5.

• The result of the computation, which is the sum of the numbers 4 and 5, equaling 9.

These log events demonstrate that the Lambda function was executed successfully and performed the expected computation. By inspecting the log events, we can confirm that the application is functioning correctly and producing the desired outcome.

Conclusion

In conclusion, LocalStack combined with Terraform provides a powerful solution for developing and testing AWS-based applications locally. With LocalStack, developers can emulate various AWS services, allowing them to build and test their infrastructure and applications without incurring any costs or dependencies on the actual AWS cloud environment. Terraform complements LocalStack by enabling infrastructure-as-code, making it easy to manage and provision AWS resources.

Throughout this blog, we explored the installation and setup of LocalStack, starting and configuring it to emulate AWS services. We also delved into developing infrastructure using Terraform, guiding readers through the process of installation, writing Terraform configuration files, and deploying resources. By combining Terraform with LocalStack, developers can create, manage, and provision AWS resources in a consistent and repeatable manner.

Furthermore, we examined a sample application deployment using Terraform and LocalStack, showcasing how to deploy a Lambda function triggered by a scheduled event. This practical example highlighted the seamless integration between LocalStack, Terraform, and the AWS services they emulate, providing a comprehensive local development and testing environment.

Finally, we verified the application's functionality by inspecting the log events generated by the deployed Lambda function. By filtering the log events, we observed the successful execution of the Lambda function, along with the input event data and the expected result.

In summary, LocalStack and Terraform offer a compelling combination for local AWS development and testing, enabling developers to build, deploy, and validate AWS infrastructure and applications efficiently. Incorporating these tools into your development workflow can significantly enhance productivity and accelerate the delivery of reliable and robust AWS solutions.
At the end of this blog, if you're interested in exploring a practical implementation of the concepts discussed, you can refer to the scheduler-localstack GitHub project. This project provides a code reference and example infrastructure setup using Terraform and LocalStack for developing and testing AWS Lambda functions locally.

AWS intrinsic functions in AWS SAM templates are a powerful feature that allows you to dynamically generate values at runtime based on the context of your application. These functions are provided by AWS CloudFormation and can be used in your AWS SAM templates as well.

In this blog post, we will explore the basics of AWS intrinsic functions in AWS SAM templates and how you can use them to build dynamic and flexible serverless applications on AWS.

What are AWS Intrinsic Functions in AWS SAM templates?

AWS Intrinsic Functions in AWS SAM templates are a set of special functions that allow you to generate values dynamically at runtime based on the context of your serverless application. These functions are provided by AWS CloudFormation, which is the underlying service on which AWS SAM templates are built. It can be used to simplify the creation and management of serverless resources in your AWS SAM templates.

Why use AWS Intrinsic Functions in AWS SAM templates?

AWS Intrinsic Functions in AWS SAM templates allow you to create dynamic and flexible serverless applications on AWS. They allow you to reference other resources, perform string manipulation, and even generate random numbers, among other things. By leveraging these functions, you can reduce duplication, increase reusability, and simplify the management of your serverless resources.

Let's discuss some of the commonly used AWS intrinsic functions in AWS SAM templates.

Ref

Ref is an intrinsic function in AWS SAM templates that are used to reference other resources in your template. It returns the value of the named resource or parameter.

When you use Ref in your AWS SAM template, AWS CloudFormation automatically resolves the reference to the value of the named resource or parameter at deployment time. This means that you can use Ref to get the ARN, ID, or other attributes of a resource in your template.

Here's an example of how you can use Ref in your AWS SAM template:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-bucket
  MyLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./my-lambda-function
      Handler: index.handler
      Runtime: nodejs14.x
      Environment:
        Variables:
          BucketName: !Ref MyBucket

In this example, we have defined an S3 bucket named "MyBucket" and a Lambda function named "MyLambdaFunction". The Lambda function has an environment variable named "BucketName" that references the S3 bucket using Ref. This allows us to dynamically set the bucket name in our Lambda function based on the name of the S3 bucket resource defined in our AWS SAM template.

Fn::Sub

Fn::Sub is an intrinsic function in AWS SAM templates that are used to substitute variables in a string with their corresponding values at deployment time. It allows you to create more flexible and dynamic templates that can adapt to different environments and configurations. The syntax for Fn::Sub is as follows:

!Sub string [ variablesMap ]

The string parameter is the string that you want to substitute variables in. The variablesMap parameter is an optional map that defines the variables and their values that you want to substitute. Here's an example of how you can use Fn::Sub in your AWS SAM template:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${EnvironmentName}-my-bucket-${AWS::Region}

In this example, we have defined an S3 bucket named "MyBucket" with a dynamic bucket name that includes the environment name and AWS region. The environment name is a variable defined in the variables map using the !Sub intrinsic function. The AWS::Region variable is automatically provided by AWS CloudFormation.

Condition functions

In some cases, you may want to conditionally create resources based on certain criteria. AWS Intrinsic Functions such as Fn::If and Fn::Equals allow you to define conditions and control the creation of resources accordingly. For example, you can conditionally create an S3 bucket only if a specific parameter is set to true, or you can enable different features based on the environment or deployment stage.

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Condition: CreateBucketCondition
    Properties:
      ...
Conditions:
  CreateBucketCondition: !Equals [!Ref CreateBucket, "true"]

In this example, the Fn::Equals function is used to conditionally create an S3 bucket based on the value of the CreateBucket parameter.

Input and output data manipulation:

SAM templates often require exchanging data between resources or accessing outputs from other AWS CloudFormation stacks. AWS Intrinsic Functions provide powerful capabilities for manipulating input and output data within your templates.

Cross-stack references:

When you have multiple AWS CloudFormation stacks that need to communicate with each other, you can use the Fn::ImportValue function to reference outputs from other stacks. This function allows you to retrieve values exported from another stack and use them as input for your resources. For example:

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          API_URL: !ImportValue MyApiStack-ApiUrl-${Stage}
          ...

In this example, the Fn::ImportValue function is used to retrieve the API URL from the MyApiStack stack's output and assign it to the API_URL environment variable. This enables your serverless function to interact with the API exposed by the MyApiStack.

Selecting specific elements from arrays:

In some cases, you may have an array of values and only need to extract specific elements. The Fn::Select function allows you to retrieve a single element from a list based on its index. For example:

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          Region: !Select [0, !Split ["-", !Ref AWS::Region]]
          ...

In this example, the Fn::Select function is used to split the AWS::Region value using the dash ("-") as the separator and then select the first element from the resulting array. This can be useful when you need to extract a specific portion of a value for configuration purposes.

Getting attribute values from resources:

SAM templates often require accessing specific attribute values from resources. The Fn::GetAtt function allows you to retrieve specific attributes of a resource. For example:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      ...
  MyBucketName:
    Type: AWS::SSM::Parameter
    Properties:
      Type: String
      Value: !GetAtt MyBucket.Name
      ...

n this example, the Fn::GetAtt function is used to retrieve the Name attribute of the MyBucket resource and assign it as the value for an AWS Systems Manager Parameter (AWS::SSM::Parameter). This enables you to store and reference the bucket name in a parameter that can be used by other resources.

By leveraging these AWS Intrinsic Functions for input and output data manipulation, you can create more dynamic and interconnected serverless applications within your SAM templates. These functions enable you to exchange data between resources, retrieve specific attribute values, and perform transformations to adapt the data for your application's needs.

Conclusion

AWS Intrinsic Functions provide powerful capabilities for manipulating input and output data within SAM templates. Whether you need to reference outputs from other stacks, select specific elements from arrays, or retrieve attribute values from resources, these functions allow you to effectively exchange and transform data within your serverless applications. By incorporating input and output data manipulation using AWS Intrinsic Functions, you can enhance the flexibility and interoperability of your SAM templates, leading to more robust and scalable serverless applications on AWS.

AWS Lambda is a powerful serverless computing service that lets you run code without provisioning or managing servers. In this blog post, we will explore some of the most useful AWS CLI commands for the AWS Lambda function.

Before running the AWS CLI commands, we need to Install the AWS CLI on your computer. You can download the CLI from AWS CLI. Open a command prompt or terminal window and run the following command.

aws configure

When prompted, enter the access key ID and secret access key. Enter the default region you want to use for the CLI. Optionally, you can also set a default output format for the CLI. Once you have entered all the required information, the AWS CLI will be configured and ready to use.

With the AWS CLI setup, you can now start using it to manage your AWS resources from the command line.

Create a Lambda function

You'll need to provide the function name, runtime, handler function name, IAM role ARN, and the path to your function package:

aws lambda create-function --function-name my-lambda-function --runtime nodejs16.x --role arn:aws:iam::123456789012:role/my-lambda-role --handler index.handler --zip-file fileb://function-package.zip

To allow your Lambda function to access other AWS resources, you need to create an IAM role that grants the necessary permissions. You can use the AWS CLI to create an IAM role with the required permissions.

aws iam create-role --role-name my-lambda-role --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name my-lambda-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Invoke a Lambda Function

Once your function is created, you can test it using the AWS CLI. You can use the following command to invoke your function: For example, you can create a role with the "AWSLambdaBasicExecutionRole" policy attached to it:

aws lambda invoke --function-name my-lambda-function --payload '{ "name": "Rahul" }' response.json

Update the Lambda function

If you need to make changes to your Lambda function, you can use the AWS CLI to update it. You'll need to provide the function name, the path to the updated function package, and any other parameters that need to be updated:

aws lambda update-function-code --function-name my-lambda-function --zip-file fileb://function-package.zip

Delete the Lambda function

When you're done with your Lambda function, you can use the AWS CLI to delete it:

aws lambda delete-function --function-name my-lambda-function

List all Lambda functions

This command lists all the Lambda functions in your AWS account region ap-south-1.

aws lambda list-functions --region ap-south-1

Single Lambda Function

This command retrieves information about the specified Lambda function, including its configuration and code.

aws lambda get-function --function-name my-lambda-function --region ap-south-1

Publish a new version of Lambda

This command publishes a new version of the specified Lambda function with the specified description.

aws lambda publish-version --function-name my-lambda-function --description "New version"

List all versions of a Lambda function

This command lists all the versions of the specified Lambda function

aws lambda list-versions-by-function --function-name my-lambda-function

Create an alias for a Lambda function

This command creates a new alias for the specified Lambda function with the specified version number.

aws lambda create-alias --function-name my-lambda-function --name my-alias --function-version 1

List all Lambda function names

This command lists the names of all Lambda functions in your account.

aws lambda list-functions --query 'Functions[*].FunctionName'

List all versions of a Lambda function

This command lists all the versions of the specified Lambda function.

aws lambda list-versions-by-function --function-name my-lambda-function --query 'Versions[*].Version'

Get the ARN of a Lambda function

This command retrieves the ARN of the specified Lambda function.

aws lambda get-function --function-name my-lambda-function --query 'Configuration.FunctionArn'

Lambda functions with a specific prefix

This command lists all Lambda functions in your account that start with the prefix "my-lambda-function"

aws lambda list-functions --query "Functions[?starts_with(FunctionName, 'my-lambda-function')].FunctionName"

Lambda functions with a specific prefix but not containing another string

This command lists all Lambda functions in your account that start with the prefix "my-lambda-function" but does not contain the string "test"

aws lambda list-functions --query "Functions[?starts_with(FunctionName, 'my-lambda-function') && !contains(FunctionName, 'test')].FunctionName"

Get the environment variables of a specific Lambda function

This command retrieves the environment variables of the specified Lambda function.

aws lambda get-function-configuration --function-name my-lambda-function --query 'Environment.Variables'

List all the layers of a specific Lambda function

This command lists the names of all the layers of the specified Lambda function.

aws lambda list-layers --query 'Layers[].LayerName'

Event source mappings of a specific Lambda function

This command lists the ARNs of all the event source mappings of the specified Lambda function.

aws lambda list-event-source-mappings --function-name my-lambda-function --query 'EventSourceMappings[].EventSourceArn'

Conclusion

In conclusion, AWS CLI is a powerful tool that allows you to manage your Lambda functions and their configurations from the command line. With AWS CLI, you can easily create, update, delete, and test Lambda functions, as well as retrieve information about their configurations, versions, aliases, and event source mappings.

In this blog, we discussed several AWS CLI commands with examples to help you manage your Lambda functions. We covered how to create a new Lambda function, update an existing function, invoke a function, and test a function locally using the aws lambda command-line tool. Additionally, we explored how to use JMESPath queries to extract specific data from Lambda functions, such as their configuration, environment variables, aliases, and versions.

By leveraging the power of AWS CLI and its rich feature set, you can automate the deployment and management of your Lambda functions and streamline your serverless workflows. Whether you are a developer, DevOps engineer, or system administrator, AWS CLI is a must-have tool in your arsenal for managing AWS Lambda functions efficiently and effectively.

The applications running in EKS pods can use AWS SDK/AWS CLI to make a call to the S3 bucket. In order to connect successfully, we need to distribute the AWS credentials to the containers. If we have multiple applications running, we have to distribute the credentials to all the containers. In order to have control of the distribution, AWS recommends creating the IAM role for the service account, which will help in providing the least privileged access to a particular container. With IRSA, the pod which is associated with the service account will get permission to talk to the other AWS services.

EKS pods are the Kubernetes resources, whereas S3 is an AWS resource. So, to have a good handshake between the two, we need an IRSA to be created.

Prerequisites

• Running Elastic Kubernetes cluster (EKS)

• kubectl

• AWS account

• Permission to create the IAM role and Policies

• AWS CLI

• eksctl

Create an OIDC provider

For the EKS cluster created, by default there will be an OIDC issuer URL associated with it. To use the IAM role for the service account, the OIDC provider needs to exist with OIDC issuer URL. To create the OIDC provider, we will use eksctl command line utility.

>> eksctl utils associate-iam-oidc-provider --cluster demo-cluster --approve

The demo-cluster is the name of the EKS cluster.

To verify if an OIDC provider is created, we need to run the following command.

>> aws iam list-open-id-connect-providers

We will get the below details if the OIDC provider was successfully created.

{
    "OpenIDConnectProviderList": [
        {
            "Arn": "arn:aws:iam::17483678901:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0"
        }
    ]
}

Create an IAM policy

Create the file s3-policy.json with the below contents.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::demo-bucket"
        }
    ]
}

We will create an IAM policy to allow access to get the object from the S3 bucket named demo-bucket using the below command.

aws iam create-policy --policy-name s3-policy --policy-document file://s3-policy.json

Create a service account

Create a file demo-service-account.yaml. We have to create a Kubernetes service account with the name demo-sa in the namespace demo-s3 using the below yaml file.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-sa
  namespace: demo-s3

Now apply the below command to create a service account.

kubectl apply -f demo-service-account.yaml

Create an IAM Role and attach the policy

Create a file demo-role-relationship.yaml for the trust policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::17483678901:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0:aud": "sts.amazonaws.com",
          "oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0:sub": "system:serviceaccount:demo-s3:demo-sa"
        }
      }
    }
  ]
}

Once it is done, we need to create the IAM role and attach the trust policy.

aws iam create-role --role-name s3-role --assume-role-policy-document file://demo-role-relationship.json"

Now, we will attach the IAM policy to the role created above.

aws iam attach-role-policy --role-name s3-role --policy-arn=arn:aws:iam::17483678901:policy/s3-policy

Annotate the service account with the IAM role

The service account needs to be annotated to the IAM role using the below command.

kubectl annotate serviceaccount -n demo-s3 demo-sa eks.amazonaws.com/role-arn=arn:aws:iam::17483678901:role/s3-role

Testing the connectivity between Pod and S3 Bucket

Create a file s3-demo-deployment.yaml . Notice that, we are using the 'default' service account that does not have access to the S3 bucket as we have only configured access to the 'demo-sa' service account.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: demo-s3
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: default
      initContainers:
      - name: demo-aws-cli
        image: amazon/aws-cli
        command: ['aws', 's3', 'cp', 's3://demo-bucket/test.txt, '-'
      containers:
      - name: my-app
        image: nginx

Now, let us run the deployment file using kubectl apply -f s3-demo-deployment.yaml.

kubectl get pods -n demo-s3
NAME                                                READY   STATUS    
nginx-8665cf9655-7h5fs                 0/1         Init:CrashLoopBackoff

Observe above that, the pod did not come up successfully as the Init container was not able to access the s3 bucker demo-bucket. We can also check this using the below command.

kubectl logs -l app=nginx -c demo-aws-cli -n demo-s3

download failed: s3://demo-bucket/test.txt to - An error occurred (403) when calling the HeadObject operation: Forbidden

Now, let us change the service account to demo-sa in the init container as shown below.

 spec:
      serviceAccountName: demo-sa
      initContainers:
      - name: demo-aws-cli
        image: amazon/aws-cli
        command: ['aws', 's3', 'cp', 's3://demo-bucket/test.txt, '-'

Now reapply using the command kubectl apply -f s3-demo-deployment.yaml.

NAME                                                READY   STATUS    RESTARTS   AGE
nginx-12455f5f955-9g8gs                    1/1     Running       0               1d

Let us verify if it was able to connect to S3 and print the contents in the console.

kubectl logs -f nginx-12455f5f955-9g8gs -n demo-s3

Hi!!!! Testing IAM role for a service account.

As we have used the demo-sa service account, it was able to successfully connect to S3 bucket.

Conclusion

In this blog, we have seen what is IRSA and how to use it to access the S3 bucket from the EKS pods. We have also seen, how the pod is connected to a particular service account and how it uses IRSA to connect to the S3 bucket.

• What is GitOps?

• ArgoCD tool

• Installing ArgoCD locally

• Setting up the app for Argocd

• Syncing out-of-sync app

What is GitOps?

GitOps is a framework used to automate the process of provisioning the infrastructure. The Git repository is the center of the GitOps framework. The concept has evolved from the DevOps culture. DevOps is a mature framework that focuses more on the communication and collaboration of different teams in an organization. GitOps is the kind of branch of DevOps that focuses on using Git repositories for infrastructure and application deployments.

GitOps uses configuration files stored as code(Infrastructure as Code). The good practice of GitOps requires us to keep the infrastructure-related code repository separate from the application-related code repository. The Git repository is a source of truth for all the infrastructure definitions. Any configuration change of manual updates to the infrastructure would be overwritten to the configuration defined in the Git repository.

ArgoCD tool

ArgoCD is a declarative, GitOps tool used to do continuous delivery for Kubernetes. ArgoCD uses the GitOps principles of using the Git repository as a source of truth. The application definitions, configurations, and environments will be stored in a separate Git repository from the application code repository and also declarative and version controlled.

ArgoCD will be part of the k8s cluster. ArgoCD agent pulls the k8s manifest changes and applies them to the k8s cluster. ArgoCD is configured to track the Git repository that contains manifest files. ArgoCD monitors for any changes to manifest files and tries to be in sync with the k8s cluster.

ArgoCD workflow

• The developer commits the code and creates a tag that will trigger the CI pipeline.

• The CI pipeline builds the artifact and then creates a docker image which then will be stored in the docker hub.

• Once the docker image is created, the CI server can update the manifest files with a new/updated docker image.

• Once manifest files are updated, the argoCD will be out of sync.

• ArgoCD will update the cluster to be in sync with the manifest files as Git is the source of truth.

Installing ArgoCD locally

Prerequisite:

The tools which are required for ArgoCD to be installed locally are:

• Docker

• Kubernetes cluster ( minikube, kind, docker-desktop)

• Kubectl tool

ArgoCD can be installed within the Kubernetes cluster or external to the cluster. We will be installing ArgoCD inside the docker-desktop Kubernetes cluster.

First, we need to create a namespace to install ArgoCD by using the below command.

kubectl create namespace argocd

ArgoCD provides the manifest yaml file that can be applied to the locally running Kubernetes cluster.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

verify the installation

kubectl get pods -n argocd
NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   1          2d10h
argocd-applicationset-controller-695df5687d-lcghx   1/1     Running   1          2d10h
argocd-dex-server-54c99dd844-vt69d                  1/1     Running   1          2d10h
argocd-notifications-controller-7d7568845c-m5txb    1/1     Running   4          2d10h
argocd-redis-6d8cffcc47-jbwpd                       1/1     Running   1          2d10h
argocd-repo-server-647c6568c9-m6gt6                 1/1     Running   2          2d10h
argocd-server-8bdf66d5-jfxjh                        1/1     Running   2          2d10h

Once it is installed, we can open the argoCD instance locally by port-forwarding it to a host port.

kubectl port-forward svc/argocd-server -n argocd 8080:443

We will get the login screen as shown below.

The username is admin and password we need to get using the below command.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Once logged in, we should see the homepage as shown below. The application tiles will be empty.

Setting up the app for Argocd

To set up the application, we need a Git repository that would contain the manifest yaml files to install into the Kubernetes cluster. We will be using the git repository. https://github.com/rahulmlokurte/argocd-demo . The repository contains the manifest file for creating a namespace, deployment that uses Nginx image, and the service yaml.

test-ns.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test
spec: {}

test-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-demo
  name: nginx-demo
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-demo
  strategy: {}
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
      - image: nginx
        name: nginx-demo
        resources: {}

test-svc.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: nginx-demo
  name: nginx-demo
  namespace: test
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: nginx-demo

To create an application on ArgoCD, click on the New APP and add the following details.

• Application Name: argocd-demo

• Project Name: default

• Sync Policy: manual

• Source Repository URL: https://github.com/rahulmlokurte/argocd-demo

• revision: main

• path: deployments

• Cluster URL: https://kubernetes.default.svc

• namespace: test

Once the above details are entered, click on Create App, then we can see the app listed in the application tiles.

We can see the status as Missing and out of sync as there is a difference in the manifest files defined in Git and the actual Kubernetes cluster.

As we have selected the sync policy as manual, we need to manually click on the nginx-demo button. Once we click on sync, it will try to sync the Kubernetes cluster with the configuration stored in Git.

Desktop kubectl get pods -n test
NAME                          READY   STATUS    RESTARTS   AGE
nginx-demo-5d7c8d874d-c5l7w   1/1     Running   0          11s

Once it is running, we can see the ArgoCD application is in-sync as shown below.

Syncing out-of-sync app

Let us now change the replicas in the deployment file from '1' to '3'.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-demo
  name: nginx-demo
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-demo
  strategy: {}
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
      - image: nginx
        name: nginx-demo
        resources: {}

Now, we see the argoCD app shows out of sync because, in the Kubernetes cluster, we only have 1 pod, but in Git, it is 3 replicas.

kubectl get pods -n test
NAME                          READY   STATUS    RESTARTS   AGE
nginx-demo-5d7c8d874d-c5l7w   1/1     Running   0          11s

Now, Once we click on the sync button, we see the Kubernetes cluster with 3 replicas of pods and the argoCD application tile shows synced.

Desktop kubectl get pods -n test
NAME                          READY   STATUS    RESTARTS   AGE
nginx-demo-5d7c8d874d-c5l7w   1/1     Running   0          175m
nginx-demo-5d7c8d874d-n8895   1/1     Running   0          15s
nginx-demo-5d7c8d874d-qm4zt   1/1     Running   0          15s

Conclusion

In this blog post, we have seen the GitOps process and how the ArgoCD tool can be used that implements GitOps. We also see, how argoCD keeps track of changes in Git and tries to be in sync between the Kubernetes cluster and Git repository.

The GitHub repository for this blog post can be seen at https://github.com/rahulmlokurte/argocd-demo

Install Checkov

We can install checkov from pip using the following command:

pip install checkov

Once installed, we can verify that it is working by running the following command:

checkov -v

Configure Checkov

There are many ways to configure Checkov. We can configure it to run against a specific file or directory using the options --file and --directory respectively. We can also specify multiple files using the -f option.

We will see how to use Checkov to scan Terraform plan file in the following section.

When we write a Terraform code, we first need to initialize the Terraform configuration. This is done by running the following command:

terraform init

Next, we need to generate the Terraform plan. This is done by running the following command:

terraform plan -out my-tf.plan

So, terraform plan will be stored in the file my-tf.plan.

For checkov to scan, we can convert the Terraform plan file to JSON using the following command:

terraform show -json my-tf.plan | jq '.' > my-tf.json

We used jq to convert the JSON to a multi-line format to make it easier to read and scan the result.

Once, we have a JSON file, we can run the following command to scan the JSON file to verify our Terraform configuration:

checkov -f my-tf.json

Checkov Policy

Checkov has a set of predefined policies which are used to scan our Terraform configuration. The policies are provided for various providers like AWS, GCP, Azure, etc. The policies which it checks depend on which provider we are using and which resources are being used. For example, if we are using AWS, there is a policy CKV_AWS_41 which ensures that no hardcoded credentials such as AWS access key ID and secret access key are used. If we use checkov in the continuous integration pipeline, we can scan the Terraform plan file for such configurations, and revert the changes made to terraform configuration if any of the policies are violated.

The checkov has many policies and if we do not want checkov to scan for all the policies, we can specify the policies to be skipped using the option --skip-check. For example, if we do not want checkov to scan for the policy CKV_AWS_41, we can use the following command:

checkov -f my-tf.json --skip-check CKV_AWS_41

If we want to skip a certain part of Terraform definition block, we can add the comment inside our terraform file, so checkov can ignore the block. For example, if we want to skip the block resource "aws_s3_bucket" "my-bucket", we can add the following comment in the terraform file:

resource "aws_s3_bucket" "my-bucket" {
  region        = var.region
    #checkov:skip=CKV_AWS_20:The bucket is a public static content host
  bucket        = local.bucket_name
}

Conclusion

As we saw in this article, we can use checkov to scan various configuration files and policies. We have also seen how to use checkov to scan Terraform plan files. We can skip some policies, and we can skip certain parts of terraform configuration. If you are interested to learn more about checkov policies, do check out the Checkov Policies and do check the checkov website for more information.

pre-commit hooks automatically scan the codebase and point out the issues with a code such as linting errors, style violations, missing semicolons, etc. This helps you to ensure that the reviewers can focus on the business logic and not on the trivial details. The pre-commit framework supports multiple programming languages and can be used to run a set of checks for multiple programming languages.

Install pre-commit

The pre-commit package manager can be installed using the pip command:

pip install pre-commit

Once done, verify that the pre-commit is installed by running the following command:

pre-commit --version

How to use pre-commit hooks

The pre-commit hooks can be used in any programming language. The only requirement is to have a .pre-commit-config.yaml file in the root of the repository. Inside this file, you can specify the hooks that you want to run.

• The hooks are defined as a list of dictionaries. Each dictionary contains the following keys: repos is the root key that contains the repository of the hook. rev is the revision of the hook. hooks is the list of hooks that you want to run.

• Each hook is defined as a dictionary. The hook contains the following keys: id is the id of the hook. name is the name of the hook. entry is the entry point of the hook. language is the programming language of the hook.

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v3.2.0
  hooks:
    - id: check-yaml
    - id: end-of-file-fixer
    - id: trailing-whitespace
- repo: https://github.com/psf/black
  rev: 19.3b0
  hooks:
    - id: black

If we want to run pre-commit hooks on every commit, we need to install the pre-commit hook using the following command:

pre-commit install

We can also run the pre-commit against all the files in the repository using the following command:

pre-commit run --all-files

How to use pre-commit hooks in Terraform

Let us now see how to use pre-commit hooks in Terraform. The project pre-commit-terraform already has the pre-commit hooks to take care of Terraform files.

Please Check out the code from the GitHub repository TerraformCode and then add the file .pre-commit-config.yaml to the root of the repository with the following content:

repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.62.3
    hooks:
      - id: terraform_fmt
      - id: terraform_validate

We are using the pre-commit-terraform repository to run the hooks such as terraform fmt and terraform validate. terraform fmt is a tool that formats the Terraform files. terraform validate is a tool that validates the Terraform files. These two hooks are run on every commit.

We can install the pre-commit hooks using the following command:

pre-commit install

Now, for every commit you make locally, it will check the above two hooks for formatting the terraform files and validating the terraform files. In this way, we can ensure that the terraform files are formatted and validated before we commit them. There are various hooks supported by pre-commit-terraform. Please refer to the pre-commit-terraform repository for more details.

Github Link: https://github.com/rahulmlokurte/aws-usage/tree/main/terraform/terraform-modules

In simpler terms, when you code the application, you try to make it modular. In the same way, we can create our terraform files in a modular way. The main advantage of doing that is, the modules become reusable. I can then call the modules from the root module or call the module inside the module. We can also publish the modules to the terraform registry or your private registries in your organization. Then any team can pull the modules and terraform automatically downloads the modules using the source and version information.

In this blog, we will see how to create modules and what module_scope is, using terraform.

Module Blocks

Terraform HCL language provides us the Module Block resource, which we can use to define the multiple resources together to form one module. In each repository, we will have a minimum of one root module which is the current working directory where you initialize the terraform using terraform init. We can call the child module using the below syntax.

module "module_name" {
    source = ""
    version = ""
}

• module_name is the local name, which can be used for referring to the instance.
• source is the mandatory argument where we can point to a local directory containing terraform configuration files or a remote registry in which case, you specify the URL where terraform configuration files are defined.
• version is mandatory only when you use the modules from the registry.

The module also supports input variables and meta-arguments. We can create multiple instances using one module block definition using the meta-argument count and we can iterate over the multiple instances using meta-argument for_each. We can also explicitly mention the dependencies, if there are any dependent modules that must be completed, before the current module. It is done through meta-argument depends_on

module scopes

The resources defined in one module is not visible to another module. This helps in making each resource unique in a particular module namespace. If you want to use the resources between the modules, we need to explicitly output the resource.

To understand terraform module, we need to understand how module scope works. So let us understand the module scope with an example.

Suppose we are provisioning an AWS lambda function resource and we need to associate the role to the lambda function, we can use the module concept here.

Project Structure

• Create a new directory and move to the directory.

mkdir terraform-modules && cd terraform-modules

• Create a main.tf and variables.tf file in the root directory.

main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.50.0"
    }
  }
}

provider "aws" {
  # Configuration options
  region                  = var.region
  profile                 = var.aws_profile
  shared_credentials_file = var.shared_credentials_file
  default_tags {
    tags = var.tags
  }
}

variables.tf

variable "region" {
  description = "Deployment Region"
  default     = "ap-south-1"
}

variable "aws_profile" {
  description = "Given name in the credential file"
  type        = string
  default     = "rahul-admin"
}

variable "shared_credentials_file" {
  description = "Profile file with credentials to the AWS account"
  type        = string
  default     = "~/.aws/credentials"
}

variable "tags" {
  description = "A map of tags to add to all resources."
  type        = map(string)
  default = {
    application = "Learning-Tutor"
    env         = "Test"
  }
}

• Create a folder modules/lambda for lambda configurations and modules/roles for role configuration.

• Create a hello-lambda.tf file and add the resources as shown below.

resource "aws_lambda_function" "hello_lambda" {
  function_name = "hello-lambda"
  role = var.aws_lambda_function_role_arn
}

• Now create a variables.tf in modules/lambda and add the below variables.

variable "aws_lambda_function_role_arn" {
  type = string
}

Now, we can call the lambda module from our root module i.e main.tf as shown below.

module "lambdas" {
  source = "./modules/lambda"
  aws_lambda_function_role_arn = module.roles.lambda_role_arn
}

As we see, for the role, we need to call another module i.e roles module, and access the resource lambda_role_arn which is defined in the roles module. As we cannot directly access the resources between each module, we have to explicitly expose it as an output as will be done next.

• Create an iam.tf file in module/role folder and add the below content.

resource "aws_iam_role" "lambda_role" {
  name = "Hello-lambda-role"
  assume_role_policy = data.template_file.lambda_assume_role_policy.rendered
}

• Create a source.tf file in the module/role folder and add the below content.

data "template_file" "lambda_assume_role_policy" {
  template = file("${path.module}/templates/lambda_assume_role_policy.json")
}

• Create a folder module/role/templates and add the below policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Lambda",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Now, as we cannot access the resources from one module into another module, we need to explicitly create an output and expose the resource for another module.

• Create a outputs.tf file in the module/role folder and add the below output.

output "lambda_role_arn" {
  value = aws_iam_role.lambda_role.arn
}

Now, we can access lambda_role_arn in another module, so we can use aws_lambda_function_role_arn = module.roles.lambda_role_arn in our lambda module as shown below.

module "lambdas" {
  source = "./modules/lambda"
  aws_lambda_function_role_arn = module.roles.lambda_role_arn
}

Conclusion

In this blog post, we saw what are Terraform modules and what a Terraform scope is. We have learned how to use the resources from one module in another module.

GitHub Repo: https://github.com/rahulmlokurte/aws-usage/tree/main/terraform/terraform-modules

In this blog post, we will create an AWS EC2 instance using Terraform.

Create an AWS EC2 instance

• Create a file called main.tf and put the following code in it:

provider "aws" {
region = "ap-south-1b"
}

This tells Terraform that you are going to be using the AWS provider and that you wish to deploy your infrastructure in the ap-south-1b region.

• Add the following code to main.tf, which uses the aws_instance resource to deploy an EC2 Instance

resource "aws_instance" "hello" {
ami = "ami-0d51ghh9cbfagedf68"
instance_type = "t2.micro"
}

The general syntax for a Terraform resource is:

resource "<PROVIDER>_<TYPE>" "<NAME>" {
[CONFIG …]
}

Where PROVIDER is the name of a provider (e.g., aws), TYPE is the type of resources to create in that provider (e.g., instance), NAME is an identifier you can use throughout the Terraform code to refer to this resource (e.g., hello), and CONFIG consists of one or more arguments that are specific to that resource (e.g., ami = "aim 0d51ghh9cbfagedf68").

• In a terminal, go into the folder where you created main.tf , and run the terraform init

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws"
....
Terraform has been successfully initialized!

terraform init will tell Terraform to scan the code, figure out what providers you’re using, and download the code for them. By default, the provider code will be downloaded into a .terraform folder.

• Now that you have the provider code downloaded, run the terraform plan command:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
(...)
+ aws_instance.example
ami: "ami-4567803n"
availability_zone: "<computed>"
ebs_block_device.#: "<computed>"
ephemeral_block_device.#: "<computed>"
instance_state: "<computed>"
instance_type: "t2.micro"
key_name: "<computed>"
network_interface_id: "<computed>"
placement_group: "<computed>"
private_dns: "<computed>"
private_ip: "<computed>"
public_dns: "<computed>"
public_ip: "<computed>"
root_block_device.#: "<computed>"
security_groups.#: "<computed>"
source_dest_check: "true"
subnet_id: "<computed>"
tenancy: "<computed>"
vpc_security_group_ids.#: "<computed>"
Plan: 1 to add, 0 to change, 0 to destroy.

The plan command lets you see what Terraform will do before actually doing it. This is a great way to sanity-check your changes before unleashing them onto the world. The output of the plan command is a little like the output of the diff command: resources with a plus sign (+) are going to be created, resources with a minus sign (-) are going to be deleted, and resources with a tilde sign (~) are going to be modified in-place. In the output above, you can see that Terraform is planning on creating a single EC2 Instance and nothing else, which is exactly what we want.

• To actually create the instance, run the terraform apply command:

$ terraform apply
(...)
Terraform will perform the following actions:
# aws_instance.example will be created
+ resource "aws_instance" "hello" {
+ ami = "ami-0c55b159cbfafe1f0"
+ arn = (known after apply)
+ associate_public_ip_address = (known after apply)
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ id = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t2.micro"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = (known after apply)
(...)
}
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:

You’ll notice that the terraform apply command shows you the same plan output and asks you to confirm if you actually want to proceed with this plan. So while the plan is available as a separate command, it’s mainly useful for quick sanity checks and during code reviews, and most of the time you’ll run apply directly and review the planned output it shows you. Type in “yes” and hit enter to deploy the EC2 Instance.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_instance.example: Creating…
aws_instance.example: Still creating… [10s elapsed]
aws_instance.example: Still creating… [20s elapsed]
aws_instance.example: Creation complete after 26s
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Congrats, you’ve just deployed a server with Terraform! To verify this, you can log in to the EC2 console, and you’ll see something like this:

Conclusion

In this blog post, we saw how to provision the EC2 instance using Terraform. We also learned the terraform commands.

Github Link: https://github.com/rahulmlokurte

In this blog post, we will do the following below items:

• Create a VPC
• Create Subnets
• Create an Internet Gateway
• Create Route Tables
• Create Routes in RouteTable
• Connect Route Table to Subnets
• Create a Security Group

Create a VPC

When you create an AWS account, you will get the default VPC in your account. We can create a custom VPC by using Terraform resource aws_vpc. The required attribute is cidr_block. It is the short form of Classless Inter-Domain Routing block. It provides the required number of IP addresses to the users. It also has a number after '/' which indicates, the number of bits for the network ID. In cidr_block, each number is of 8bits. so, in the example 10.0.0.0/16, there are a total of 8 * 4 = 32 bits and 16 is the number of bits that represent network ID. So, first 16 bits that is, 10.0 is fixed and from the remaining 16 bits, we will be able to create 2 ^ 16 unique IP addresses.

resource "aws_vpc" "vpc-learning" {
  cidr_block = "10.0.0.0/16"
  tags = {
    "Name" = "vpc-learning"
  }
}

Create Subnets

If we have a large network, then it becomes complex to manage the network. So, it is logical divided into Subnets. There are two types of Subnets. One is a private subnet that is not exposed to the outside world and another is a public subnet that is exposed to the internet.

We can create a Subnet by using Terraform resource aws_subnet. The required attribute is cidr_block and VPC, in which the subnet needs to be created. Here the cidr_block will have fixed the first 16 bits as defined by the VPC which we created earlier. Here 24 bits represent network ID. So, we have only 8 bits, which can be changed. So, we will be able to create 2 ^ 8 unique IP addresses. The map_public_ip_on_launch indicates that the public IP address will be assigned to the instances within the subnet.

resource "aws_subnet" "vpc-learning-public-subnet-1" {
  cidr_block = "10.0.1.0/24"
  vpc_id = aws_vpc.vpc-learning.id
  map_public_ip_on_launch = true
  availability_zone = "ap-south-1a"
}

Create an Internet Gateway

As the public IP address are assigned earlier, we can allow the instances to be accessible through the internet. we can use the aws_internet_gateway terraform resource and we need to point to the vpc as shown below.

resource "aws_internet_gateway" "vpc-learning-internet-gateway" {
  vpc_id = aws_vpc.vpc-learning.id
}

Create Route Tables

We can create Route Tables using the aws_route_table resource with terraform.

resource "aws_route_table" "vpc-learning-public-route-table" {
  vpc_id = aws_vpc.vpc-learning.id
}

Create Routes in RouteTable

We need to create a Routes in the RouteTable which was created earlier. We can use aws_route resource using terraform. Also, we make sure that these routes will be access through the internet. So, we need to provide the internet gateway using the attribute gateway_id as shown below.

resource "aws_route" "vpc-learning-public-route" {
  route_table_id = aws_route_table.vpc-learning-public-route-table.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = aws_internet_gateway.vpc-learning-internet-gateway.id
}

Connect Route Table to Subnets

Once the route tables are created, we need to associate the route table to the Subnets which were created earlier. We can use aws_route_table_association resource of terraform.

resource "aws_route_table_association" "vpc-learning-public-route-table-association" {
  route_table_id = aws_route_table.vpc-learning-public-route-table.id
  subnet_id = aws_subnet.vpc-learning-public-subnet-1.id
}

Create a Security Group

We need to define the security group to control the incoming and outgoing network calls. we can use the aws_security_group resource in terraform. We need to give the incoming and outgoing network call by using terraform attributes ingress and outgress respectively. Below, we are only allowing incoming from SSH i.e from 22 port and from HTTP i.e. 80 port, and the outgoing to unrestricted.

resource "aws_security_group" "vpc-learning-security-group" {
  name = "vpc-learning-default-security-group"
  vpc_id = aws_vpc.vpc-learning.id
  ingress {
    from_port = 22
    protocol = "tcp"
    to_port = 22
    cidr_blocks = [var.ingress_egress_cidr_blocks]
  }
  ingress {
    from_port = 80
    protocol = "tcp"
    to_port = 80
    cidr_blocks = [var.ingress_egress_cidr_blocks]
  }

  egress {
    from_port = 0
    protocol = "-1"
    to_port = 0
    cidr_blocks = [var.ingress_egress_cidr_blocks]
  }
}

Deploy using Terraform

Let us initiate the directory with terraform by issuing a command terraform init.

Once done, issue a command terraform plan -out plan-out where it will display what all resources will be provisioned and put the plan in the file named plan-out.

Now, we can apply the actual resource by using the command terraform apply "plan-out"

Verify the resources in AWS Console.

• VPC

• Subnets

• Internet Gateway

• Route Table showing association with subnets.

• Security Groups showing Inbound and outbound

Conclusion

In this blog post, we have looked into the working of virtual private cloud in your AWS. We have looked into creating VPC and Subnets. Also, we have exposed the endpoints using the internet gateway and created a route table with subnet association. We also looked into how the security group helps us with the traffic to the network.

GitHub Repo: https://github.com/rahulmlokurte/aws-usage/tree/main/terraform/vpc-learning

In this blog, we will set up a rule to run the lambda function on schedule every 2 minutes using Terraform.

Project Structure

• Create a new directory and move to the directory

  mkdir lambda-schedule-event-bridge && cd lambda-schedule-event-bridge
  

• Create a resource folder to store the code for lambda

  mkdir resources
  

Create a Lambda

Create a folder profile-generator-lambda in the resources folder and add the index.js file with the below content. Also, initialize the node project and install the dependencies.

mkdir profile-generator-lambda && cd profile-generator-lambda
touch index.js
npm init -y
npm install faker

const faker = require("faker/locale/en_IND");

exports.handler = async (event, context) => {
  let firstName = faker.name.firstName();
  let lastName = faker.name.lastName();
  let phoneNumber = faker.phone.phoneNumber();
  let vehicleType = faker.vehicle.vehicle();

  let response = {
    firstName: firstName,
    lastName: lastName,
    phoneNumber: phoneNumber,
    vehicleType: vehicleType,
  };

  return {
    statusCode: 200,
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      profile: response,
    }),
  };
};

Terraform Providers

Now, we will write the terraform scripts in HCL language where we will utilize the AWS provider. Create a main.tf file in the root directory of the project and add the below content.

touch main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.50.0"
    }
  }
}

provider "aws" {
  # Configuration options
  region                  = var.region
  profile                 = var.aws_profile
  shared_credentials_file = var.shared_credentials_file
  default_tags {
    tags = var.tags
  }
}

• Each Terraform module must declare which providers it requires so that Terraform can install and use them. Provider requirements are declared in a _requiredproviders block.

• AWS providers require configurations like cloud regions, profiles, credential_files to be available before they can be used.

• All the values will be provided from the variables parameters.

Let's create variables.tf in the root directory of the project and add the below content.

touch variables.tf

variable "region" {
  description = "Deployment Region"
  default     = "ap-south-1"
}

variable "aws_profile" {
  description = "Given name in the credential file"
  type        = string
  default     = "rahul-admin"
}

variable "shared_credentials_file" {
  description = "Profile file with credentials to the AWS account"
  type        = string
  default     = "~/.aws/credentials"
}

variable "tags" {
  description = "A map of tags to add to all resources."
  type        = map(string)
  default = {
    application = "Learning-Tutor"
    env         = "Test"
  }
}

Each input variable accepted by a module must be declared using a variable block. The label after the variable keyword is a name for the variable, which must be unique among all variables in the same module.

• A default value makes the variable optional.
• A Type indicates what value types are accepted for the variable.
• A description specifies the input variable's documentation.

Terraform Modules

Now, we will use the module terraform-aws-modules/lambda/aws to create lambda and lambda layer infrastructure.

Create the files lambda.tf in the root directory of the project and add the below content.

touch lambda.tf

module "profile_generator_lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.7.0"
  # insert the 28 required variables here
  function_name = "profile-generator-lambda"
  description   = "Generates a new profiles"
  handler       = "index.handler"
  runtime       = "nodejs14.x"
  source_path   = "${path.module}/resources/profile-generator-lambda"

  tags = {
    Name = "profile-generator-lambda"
  }
}

EventBridge Rule

Create the files event_bridge.tf in the root directory of the project and add the below content.

touch event_bridge.tf

resource "aws_cloudwatch_event_rule" "profile_generator_lambda_event_rule" {
  name = "profile-generator-lambda-event-rule"
  description = "retry scheduled every 2 min"
  schedule_expression = "rate(2 minutes)"
}

resource "aws_cloudwatch_event_target" "profile_generator_lambda_target" {
  arn = module.profile_generator_lambda.lambda_function_arn
  rule = aws_cloudwatch_event_rule.profile_generator_lambda_event_rule.name
}

resource "aws_lambda_permission" "allow_cloudwatch_to_call_rw_fallout_retry_step_deletion_lambda" {
  statement_id = "AllowExecutionFromCloudWatch"
  action = "lambda:InvokeFunction"
  function_name = module.profile_generator_lambda.lambda_function_name
  principal = "events.amazonaws.com"
  source_arn = aws_cloudwatch_event_rule.profile_generator_lambda_event_rule.arn
}

As can be seen above, the schedule_expression attribute has a rate of 2 minutes. It means, it triggers the lambda function which is in the target using the resource aws_cloudwatch_event_target.profile_generator_lambda_target. We also give lambda permission, so that, the event will be able to invoke the function.

Run Terraform scripts

Let us run the 3 basic commands of terraform to create the resources in AWS.

• Initialize the Terraform which will download all the providers and modules used in the configuration.

terraform init

• Check if a plan matches the expectation and also store the plan in output file plan-out

terraform plan -out  "plan-out"

• Once the plan is verified, apply the changes to get the desired infrastructure components.

terraform apply "plan-out"

Verify the infrastructure on AWS

• Lambda with Event Bridge

If you check the cloud watch logs, you can see that the lambda function is getting invoked every 2 minutes.

Delete the resources

We can delete the resources created by terraform by issuing the below command so that we do not get billed for the resources.

terraform destroy

Conclusion

In this blog post, we saw how to create a lambda function and event bridge rule using terraform. We also saw, how to schedule the lambda function every 2 minutes by creating an event bus rule.

Github Repository : https://github.com/rahulmlokurte/aws-usage/tree/main/terraform/lambda-schedule-event-bridge

Developers push the code multiple times a day. Packaging our applications as containers, enables our application to be updated multiple times without downtime. Kubernetes helps us to run the containerized applications.

We can create a Kubernetes cluster in our local computer by using tools such as minikube and kind. In this blog, we will look into the Kind.

Prerequisite:

• Docker

Tools :

• kubectl - A Kubernetes command-line tool that allows us to run commands against Kubernetes clusters. We can deploy applications, inspect and monitor the cluster resources, and view logs using kubectl.

• kind - It is a tool that lets us run the Kubernetes cluster on our local computer.

Install Kubectl

1. macOS

2. Windows

Install Kind

macOS

$brew install kind

Windows

$curl.exe -Lo kind-windows-amd64.exe https://kind.sigs.k8s.io/dl/v0.11.1/kind-windows-amd64
$Move-Item .\kind-windows-amd64.exe c:\some-dir-in-your-PATH\kind.exe

Create a Kubernetes cluster

Once kind is installed, create a kubernetes cluster using the command

$kind create cluster --name learning-kubernetes-dev

When the cluster is created, the cluster configuration file is stored at the location ${home}/.kube/config

We can create multiple kubernetes clusters. Let us define one more cluster for production

$kind create cluster --name learning-kubernetes-prod

To verify, if the cluster has been created, issue the below command.

$kind get clusters

learning-kubernetes-dev
learning-kubernetes-prod

To interact with a particular cluster, we need to specify the kind-<cluster-name> as a context in kubectl as shown below.

➜  $kubectl cluster-info --context kind-learning-kubernetes-dev
Kubernetes control plane is running at https://127.0.0.1:52083
CoreDNS is running at https://127.0.0.1:52083/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

➜  $kubectl cluster-info --context kind-learning-kubernetes-prod
Kubernetes control plane is running at https://127.0.0.1:52449
CoreDNS is running at https://127.0.0.1:52449/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To verify which context is currently used, we can issue the below command.

$kubectl config get-contexts

CURRENT   NAME                            CLUSTER                         AUTHINFO                        NAMESPACE
          kind-learning-kubernetes-dev    kind-learning-kubernetes-dev    kind-learning-kubernetes-dev
*         kind-learning-kubernetes-prod   kind-learning-kubernetes-prod   kind-learning-kubernetes-prod

The indicates the current cluster. So the current cluster context is *kind-learning-kubernetes-prod

To switch between the clusters, we can issue the below command.

$kubectl config use-context kind-learning-kubernetes-dev

Switched to context "kind-learning-kubernetes-dev"

Let us verify again, this time, the current context is kind-learning-kubernetes-dev

$kubectl config get-contexts
CURRENT   NAME                            CLUSTER                         AUTHINFO                        NAMESPACE
*         kind-learning-kubernetes-dev    kind-learning-kubernetes-dev    kind-learning-kubernetes-dev
          kind-learning-kubernetes-prod   kind-learning-kubernetes-prod   kind-learning-kubernetes-prod

Namespace

The virtual clusters which are created within the physical cluster are called namespaces. If we have multiple teams/projects, we can use the same cluster, but for each team/project, we can define the namespaces.

If we do not define any namespaces, then we get a default namespace. We can use the below command to see the namespaces.

kubectl get namespaces

NAME                 STATUS   AGE
default              Active   23m

Create a namespace

We can define multiple namespaces in kubernetes clusters. In our learning-kubernetes-dev, we can have two namespaces. One is used by team-1 and the other is used by team-2.

Let's first switch the context to create a namespace in the learning-kubernetes-dev cluster

$kubectl config use-context kind-learning-kubernetes-dev

create a file called team1-namespace.json with the below content.

{
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "name": "team-1",
    "labels": {
      "name": "team-1"
    }
  }
}

Use the below command to create a namespace.

$kubectl create -f team1-namespace.json

namespace/team-1 created

create a file called team2-namespace.json with the below content.

{
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "name": "team-2",
    "labels": {
      "name": "team-2"
    }
  }
}

Use the below command to create a namespace.

$kubectl create -f team2-namespace.json

namespace/team-2 created

Use the below command to verify the namespaces.

$kubectl get namespace

NAME                 STATUS   AGE
default              Active   44m
team-1               Active   2m13s
team-2               Active   18s

Create a Pod in a Namespace

A Kubernetes namespace provides the scope for Pods, Services, and Deployments in the cluster.

let's create a simple Deployment and Pods for both Team-1 and Team-2 namespace. Team-1 needs an NGINX server of 1 instance and Team-2 needs 2 instances.

$kubectl create deployment nginx-team-1 --image nginx -n team-1 --replicas=1

deployment.apps/nginx-team-1 created

$kubectl create deployment nginx-team-2 --image nginx -n team-2 --replicas=2
deployment.apps/nginx-team-2 created

Let's verify the number of pods created for team-1 and team-2

kubectl get pods -n team-1

NAME                            READY   STATUS    RESTARTS   AGE
nginx-team-1-544dd7f5bd-qk5g6   1/1     Running   0          113s

kubectl get pods -n team-2

NAME                            READY   STATUS    RESTARTS   AGE
nginx-team-2-55f89d7667-2wd9n   1/1     Running   0          87s
nginx-team-2-55f89d7667-8xd95   1/1     Running   0          87s

Conclusion

In this blog post, we saw how easy it is to create a Kubernetes cluster in your local computer. Once installed, we can play with Kubernetes and it helps us learn about pods, deployments. We saw how to create a namespace so that we can use one cluster with multiple teams deploying applications on the same cluster.

GitHub Repository: https://github.com/rahulmlokurte/devops-stuff/tree/main/kubernetes/kubernetes-kind

This is just the starting point. Check out the below documents to get into the depth of Kubernetes.

• https://kubernetes.io/
• https://kind.sigs.k8s.io/docs/user/quick-start/
• https://www.katacoda.com/courses/kubernetes/playground

There are three main commands that we run when we want to provision an Infrastructure.

• terraform init: This command is used to initialize the working directory containing terraform configuration files. It is safe to run multiple times. The command will never delete your existing infrastructure or a state. During initialization, it searches the terraform configuration files for the providers defined. These providers are published as a plugin. Once it finds the providers, it tries to download and install the providers in the terraform directory.
• terraform plan: With this command, terraform figures out what it needs to do. It sees the real provisioned resources and compares it to the desired configuration. If Terraform detects that no changes are needed to resource instances or to root module output values, terraform plan will report that no actions need to be taken. You can use the optional -out=FILE option to save the generated plan to a file on disk.
• terraform apply: This command provisions the resources proposed in the terraform plan. Before actually applying the plan, it prompts us to confirm the changes. We can also pass the filename of a saved plan file created earlier with terraform plan -out=.... In this case, Terraform will apply the changes in the plan without any confirmation prompt.

When we provide the terraform configurations, the terraform builds a dependency graph. By looking into the dependency graph, it can make sure that the resources are provisioned accordingly. Whenever we run the terraform planand terraform apply, the terraform looks into a dependency graph to generate the plans and refresh the states.

Workflow in Action:

1. Create a new directory and write the configuration Files.

$mkdir terraform-in-action && cd terraform-in-action

$touch main.tf

Add the below code in main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.50.0"
    }
  }
}

provider "aws" {
  # Configuration options
  region                  = var.region
  profile                 = var.aws_profile
  shared_credentials_file = var.shared_credentials_file
  default_tags {
    tags = var.tags
  }
}

You might see an error on the line 12,13,14. It is because, we are referencing the variables, which are not present. So let us create a new file variable.tf

$touch variables.tf

Add the below code in variables.tf

variable "region" {
  description = "Deployment Region"
  default     = "ap-south-1"
}

variable "aws_profile" {
  description = "Given name in the credential file"
  type        = string
  default     = "rahul-admin"
}

variable "shared_credentials_file" {
  description = "Profile file with credentials to the AWS account"
  type        = string
  default     = "~/.aws/credentials"
}

variable "tags" {
  description = "A map of tags to add to all resources."
  type        = map(string)
  default = {
    application = "Learning-Tutor"
    env         = "Test"
  }
}

Here you can define the region and aws_profile. We have to point to the location of credentials of the AWS account. We can also define the tags.

Let us now create the new file lambda.tf to create the lambda function.

$ touch lambda.tf

Add the below code in lambda.tf

module "profile_generator_lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "2.7.0"
  # insert the 28 required variables here
  function_name = "profile-generator-lambda"
  description   = "Generates a new profiles"
  handler       = "index.handler"
  runtime       = "nodejs14.x"
  source_path   = "${path.module}/resources/profile-generator-lambda"
  tags = {
    Name = "profile-generator-lambda"
  }
}

• source field is used to indicate the terraform module which we intend to use.

• function_name field is the unique name for our lambda function.

• handler field is the Lambda Function entry point for our code.

• runtime field is Lambda Function runtime

• source_path field is the absolute path to a local file or directory containing our Lambda source code

Let us now create a file index.js in the folder resources/profile-generator-lambda

$mkdir resources/profile-generator-lambda
$touch index.js

Add the below code in index.js

const faker = require("faker/locale/en_IND");

exports.handler = async (event, context) => {
  let firstName = faker.name.firstName();
  let lastName = faker.name.lastName();
  let phoneNumber = faker.phone.phoneNumber();
  let vehicleType = faker.vehicle.vehicle();

  let response = {
    firstName: firstName,
    lastName: lastName,
    phoneNumber: phoneNumber,
    vehicleType: vehicleType,
  };

  return {
    statusCode: 200,
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      profile: response,
    }),
  };
};

Let us initialize the folder resources/profile-generator-lambda with npm and also install the faker dependency which is used in our lambda code to get the random profiles.

$npm init && $npm install faker

2. Terraform Init

At the root of the folder, let us now initialize the directory, which will download all the providers and modules used in the configuration.

$cd ../../
$terraform init

3. Terraform plan

Let us check if a plan matches the expectation and also store the plan in output file plan-out

$terraform plan --out "plan-out"

4. Terraform apply

Once the plan is verified, apply the changes to get the desired infrastructure components.

$terraform apply "plan-out"

Now, let us verify the infrastructure created on the AWS console.

• Lambda Function

• Test the Lambda Function

Conclusion

In this blog post, we saw how the basic terraform workflow works and also, how it maintains the dependency graph. We also saw a basic example involving the Lambda function.