Docker (provisional title)

 (In development).

S3 File Editor App: Integrating Python with AWS

In this post, I illustrate how I tackled the problem of making a specific application in Python that would communicate with AWS services. 

You can find the code in this repository: https://github.com/gianlucaballa/s3-file-editor

Task: Create an application that allows users to effortlessly share and modify written information such as materials, quantities, and important notes with one another.

The application needs to meet the following criteria:

• Any user should be able to open the application on a pc with a double-click.
• Upon opening the application, users should have immediate access to a simple text file for viewing, editing, and saving (structure for the text file is not necessary – a blank text file suffices).
• The user can read, modify, and save the text file for other users to use.

This application will be specifically used by building engineers to easily share information such as material to buy, quantity and similar notes regarding construction sites. 

No overheads: ease of use is vital here.

Key criteria: accessibility, seamless file handling, collaboration. 

----------------------------------------------------------------------------------------------------------------------

My solution:

At first, I thought about using RDS in AWS and creating a simple table with 3 columns for a MySQL database. I steered away from this solution because it would have over complicated the code for the execution of SQL commands, without producing any significant benefit.

Instead, I proceeded with the following steps:

1. I created an S3 bucket in AWS using a specific user with specific permissions and credentials.
2. I uploaded an empty text file with a specific name to the S3 bucket.
3. I wrote a Python application using boto3*.

*(Boto3 is the official AWS SDK for Python. It provides an easy-to-use Python interface to interact with various AWS services such as S3. With Boto3, developers can programmatically manage AWS resources, automate tasks, and build applications that leverage AWS services without needing to manually configure API requests). 

This solution showed to be effective and satisfied the established key criteria, as it will be demonstrated below.

----------------------------------------------------------------------------------------------------------------------

Explanation:

By sharing and analysing the Python code that I wrote for the application (see point 3. above), you can better understand the operation of the application that I designed.

    The code (formatted with Black):

import boto3

import os

def download_file(bucket_name, key, local_filename):

    s3 = boto3.client(

        "s3",

        region_name="insert_region",

        aws_access_key_id="insert_id",

        aws_secret_access_key="insert_secret",

    )

    s3.download_file(bucket_name, key, local_filename)

def upload_file(bucket_name, key, local_filename):

    s3 = boto3.client(

        "s3",

        region_name="insert_region",

        aws_access_key_id="insert_id",

        aws_secret_access_key="insert_secret",

    )

    s3.upload_file(local_filename, bucket_name, key)

def edit_file(local_filename):

    os.system(f'notepad "{local_filename}"')

def main():

    bucket_name = "insert_bucket_name"

    key = "insert_file_name.txt"

    local_filename = "temp_file.txt"

    # Download the file from S3.

    download_file(bucket_name, key, local_filename)

    # Allow the user to edit the file.

    edit_file(local_filename)

    # Upload the modified file back to S3.

    upload_file(bucket_name, key, local_filename)

    # Clean up the temporary file.

    os.remove(local_filename)

    print("File has been updated and uploaded to S3.")

if __name__ == "__main__":

    main()

import boto3

import os

These import the "boto3" module and the "os" module. The "os"  module provides a way of using the operating system dependent functionality. For example, it allows the application to execute a command in the system's shell.

 

def download_file(bucket_name, key, local_filename):

    s3 = boto3.client(

        "s3",

        region_name="insert_region",

        aws_access_key_id="insert_id",

        aws_secret_access_key="insert_secret",

    )

    s3.download_file(bucket_name, key, local_filename)

 

This function called download_file takes 3 arguments. 

Once called, it creates an S3 client object, in other words, an interface through which your Python code can communicate with Amazon S3. In order to do this, it is necessary to specify the region in which the S3 exists, the AWS access key and secret access key of the AWS user (see point 1. above).

 

Hardcoding keys is never a good idea because they can be seen by anyone looking at the code! I hardcoded them to test the application.

 

Then the download_file function downloads the file from the specified S3 bucket (bucket_name) with the specified key/name of the file (key) to the local file (local_filename) on the file system using the function (usually the local file is where the file with the code is on the pc). s3.download_file(bucket_name, key, local_filename) is a specific function provided by Boto3.

 

def upload_file(bucket_name, key, local_filename):

    s3 = boto3.client(

        "s3",

        region_name="insert_region",

        aws_access_key_id="insert_id",

        aws_secret_access_key="insert_secret",

    )

    s3.upload_file(local_filename, bucket_name, key)

 

This is a similar function but now it uploads the local file (local_filename) on the file system using the function to the specified S3 bucket (bucket_name) with the specified key/name of the file (key). s3.upload_file(local_filename, bucket_name, key) is a specific function provided by Boto3.

 

def edit_file(local_filename):

    os.system(f'notepad "{local_filename}"')

 

The edit_file function opens the specified file (local_filename) in the default text editor of the system using the os.system function. Since I am testing this application in Windows, I specified "notepad" as the default text editor.

 

def main():

    bucket_name = "insert_bucket_name"

    key = "insert_file_name.txt"

    local_filename = "temp_file.txt"

 

    # Download the file from S3.

    download_file(bucket_name, key, local_filename)

    # Allow the user to edit the file.

    edit_file(local_filename)

    # Upload the modified file back to S3.

    upload_file(bucket_name, key, local_filename)

    # Clean up the temporary file.

    os.remove(local_filename)

    print("File has been updated and uploaded to S3.")

 

In Python, main() is a conventional name for the main entry point of a script or program. It's a function that typically contains the main logic or sequence of actions that the script should perform when executed.The purpose of defining a main() function is to encapsulate the main functionality of the script and to provide a clear starting point for the program's execution. This makes the code more modular and easier to understand, as the main logic is isolated within a specific function. 

 

In this case, main() is used to call the functions that I defined above in a specific order (download, edit, upload), clean up the temporary file on the pc once it is uploaded to S3 (using os.remove(local_filename)) and print a message once the process is concluded. In main(), I also establish variables that will be used as arguments for the functions: the name of the S3 bucket (see point 1. above), the name of the .txt file that I initially uploaded to S3 (see point 2. above) and the local file name (temp_file.txt works fine for this).

if __name__ == "__main__":

    main()

This final block ensures that the main() function is executed only when the script is run directly, not when it's imported as a module into another script. This allows the script to be used both as a stand-alone program and as a reusable module. 

 

Final notes:

• Ensure that the IAM user associated with the AWS credentials has the necessary permissions to access the specified S3 bucket and perform read/write operations on the objects.
• Make sure to handle sensitive information such as AWS access keys securely.
• This script is intended for educational purposes and can be modified to suit specific requirements. 

----------------------------------------------------------------------------------------------------------------------

To transform the script into a clickable .exe, I can use

pyinstaller --onefile my_script_name.py

After PyInstaller finishes, it will create a "dist" directory in the script's directory, containing the executable file. This executable can be distributed to others.

Adapting Visual References in Concept Art For Monster Design

(This is a short version of one of my academic articles that you can find here).

There is efficacy in adapting visual references (3D renders and photos) to concept art for high-budget game development and film production.

For example, embedding visual references into an artwork for time efficiency, correct use of perspective and establishment of believable textures is significant in a video game triple-A development. The search for realism enhanced by the use of visual references also shows to be advantageous in designing uncanny monsters. 

Visual references can be manipulated in software such as Photoshop to prepare not only the blueprints for the 3D modelling/sculpting stage but also to design the special effects makeup for live-action monsters and animatronics. 

There is a gap between our current understanding of the uncanny valley, as it is defined in robotics, and the process of designing characters. Investigating this subject is important for a dual reason:

1. It moves knowledge forward in the field of the uncanny valley’s applications to concept art, since this has not been investigated in depth.
2. It helps professional concept artists in shaping and controlling the uncanniness of antagonistic characters.

Analyses on industrial practice showed that:

• The use of photorealistic references to be implanted directly into concept art is not a mandatory request for all game developments and digital effects film productions but it is increasingly encouraged for certain products such as triple-A games because it reduces hiccups in the workflow: indeed, this practice spares the artist from hand painting texture details through brush strokes and minimizes perspective mistakes.

• When it comes to designing uncanny monsters, the use of these references pushes the design towards realism facilitating the triggering of the uncanny valley phenomenon at a design level – as highlighted by the current literature on the subject. The horror genre often favours rich textures (e.g., organic material such as blood, filth, rustiness, etc.) not only in characters but environments too – this adds to the subtle subversion of the concept of affinity which is at the centre of the uncanny valley and it is commonly used in horrors to make the viewer feel vulnerable. 

In the light of this, the adaptation of photorealistic visual references might represent a strong starting point for concept artists who aim to trigger the "uncanniness" through character design. This type of reference should be directly inserted into the work and then transformed through deformation tools, photo bashing and minimal painting. The use of "previs", in collaboration with the animation department, can implement the monster’s movements, since these contribute to the uncanniness of the creature. 

A study on the finalisation of the sculpting and modelling work which analyses how to convey the texture of the uncanny monster in a three-dimensional environment (e.g., UV mapping) would represent a natural development for a research which aims to formalise a specific industrial pipeline. 

Furthermore, an investigation into the role of scriptwriting and sound on making uncanny monsters for films and games would be a valuable supplement because it could make clearer how the integration of visuals, narrative and audio influences the design of the uncanny monster. Study on sound in particular would offer significant insight that can be applied in robotics.

Generative AI – like Midjourney, Stable Diffusion and OpenAI’s DALL-E – might bring a significant change to contemporary practice. In fact, the use of AI has already demonstrated to be powerful in testing ideas, establishing colour palettes and producing suggestive photographic references with a high degree of realism. Because of this and their impact on the concept art practice, generative AI applied to this subject requires further investigations.

Photo by Pavel Danilyukyuk: https://www.pexels.com/photo/close-up-shot-of-white-robot-toy-8294606/Photo by Pavel Danilyuk: https://www.pexels.com/photo/close-up-shot-of-white-robot-toy-8294606/

 

Terraform: Launching an EC2 Instance in AWS

Terraform is an infrastructure as code (IaC) tool that lets a user define both cloud and on-prem resources in human-readable configuration files to version, reuse, and share. In other words, it allows users to define and provision infrastructure resources, such as virtual machines, storage accounts, and networks, in a declarative configuration language. 

For example, on Amazon Web Services (AWS), it would be lengthy to use the console to manually build an infrastructure. Terraform makes it faster and easier by providing a consistent and reproducible way to define, provision, and manage resources on AWS through a programmatic approach.

The key components of Terraform include:

• Configuration files: These files define the desired state of your infrastructure, specifying the resources and their configurations.
• Providers: Terraform providers are plugins that interact with APIs of different cloud providers (such as AWS, Azure, Google Cloud Platform, etc.) or other services to manage resources.
• Resource types: Terraform supports a wide range of resource types, representing various infrastructure components like virtual machines, networks, databases etc.
• Execution plans: Terraform generates execution plans to show what actions it will take when you apply your configuration, giving you a preview of changes before they are implemented.
• State management: Terraform maintains a state file that keeps track of the current state of your infrastructure. This allows Terraform to understand the relationships between resources and manage updates efficiently.

The following tutorial aims to show how to create a basic infrastructure: we will provision an EC2 instance on AWS. EC2 instances are virtual machines running on AWS, and a common component of many infrastructure projects. EC2 instances can be configured with various CPU, memory, storage, and networking options to meet different workload requirements. EC2 is widely used for hosting websites, running applications, processing data etc.

Requirements. For this tutorial, it is assumed that:

1. You have installed Terraform on your machine.
2. You know a bit of AWS.
3. You have installed AWS CLI.
4. You already have an account with AWS*.

*(Create an IAM user for this exercise rather than using the Root account. Give it the right permissions, depending on what you want to create through Terraform – an EC2 instance in this case).

I used Studio Visual Code as source-code editor (with the official Terraform extension) and GitHub.

----------------------------------------------------------------------------------------------------------------------

I created a directory on my pc (one per Terraform project is advisable). Inside, I created an empty file called "main.tf" (the name is not important, but the extension is!).

The first step is to identify a supported provider for Terraform, in my case AWS. Think of it as plugins: I want to make sure that Terraform downloads the needed code to communicate with the AWS API. Indeed, Terraform does not come with all the codes to connect to all APIs when you install it; it would not be functional since there are many providers.

Instructions for the AWS provider can be found here:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs

In the instructions, there is an example about the setup:

terraform {

    required_providers {

        aws = {

            source = "hashicorp/aws"

            version = "~> 5.0"

        }

    }

}

# Configure the AWS Provider and define a Region (optional)

provider "aws" {

    region = "eu-west-2"

}

I copy this and paste it in main.tf. So, I declared to Terraform what provider I need.

Now, the problem is that Terraform needs to connect to my AWS account to create, manage or destroy resources. There are many ways to do that.

In AWS, for the user I created ad hoc for this exercise, I can 'Create New Access Key'.

The new access key comes with an 'Access Key ID' and a 'Secret Access Key' that can be seen only once (you can save them on your machine though). I could now add them in the main.tf file to instruct Terraform to connect to my AWS account through that key, as shown below:

provider "aws" {

    region = "us-west-2"

    access_key = "my-access-key"

    secret_key = "my-secret-key"

}

However, hardcoding the key in this way is risky because someone might discover it by looking at main.tf.

The alternative is using the AWS CLI (make sure to have it installed). *

*(There are more alternatives which are ignored in this post).

In my terminal, I type:

aws configure

I am asked for the AWS Access Key ID, AWS Secret Access Key, the Default region name (the one I used for my provider in main.tf), and the Default output format (JSON is fine).

This process populates a file in my home directory that can be accessed with cd .aws (I am using Git Bash). The credential file in the aws directory will be automatically used by Terraform to connect to my AWS account; this is safer than hardcoding the keys in plain sight in main.tf as suggested before.

----------------------------------------------------------------------------------------------------------------------

In the link provided above (this), there are instructions to create various resources in AWS. I searched for EC2 and found 'aws_instance' under Resource. The basic code is:

resource "aws_instance" "web" {

    ami = data.aws_ami.ubuntu.id

    instance_type = "t3.micro"

    tags = {

        Name = "HelloWorld"

    }

}

This can be copy-pasted in main.tf, under what I already wrote before. With this, I am telling Terraform to provide a resource which is an AWS instance named "web" by me (Terraform will remember this name for this resource for future uses but AWS will ignore it). 

data.aws_ami.ubuntu.id is the AMI for that instance and the type is t3.micro (better to use t2.micro since it is free in AWS). ami and instance_type are examples of config options (there could be more). Let’s ignore tags for now.

This is the code I have now in main.tf:

terraform {

    required_providers {

        aws = {

            source = "hashicorp/aws"

            version = "~> 5.0"

        }

    }

}

# Configure the AWS Provider and define a Region (optional)

provider "aws" {

    region = "eu-west-2"

}

resource "aws_instance" "web" {

    ami = data.aws_ami.ubuntu.id

    instance_type = "t2.micro"

}

That AMI has an 'id' at its end: it means that it comes from another piece of code (attribute) on main.tf (usually it is a combination of resource + provider + name + id but let’s ignore this for now). I insert a specific AMI found on AWS under Launch Instance: “ami-0e5f882be1900e43b” * (use quotations if you don’t use the id). 

*(Make sure to take a free one in the same Region you are working in. You can use Ubuntu).

Final code in main.tf (remember to save it):

terraform {

    required_providers {

        aws = {

            source = "hashicorp/aws"

            version = "~> 5.0"

        }

    }

}

# Configure the AWS Provider and define a Region (optional)

provider "aws" {

    region = "eu-west-2"

}

resource "aws_instance" "web" {

    ami = “ami-0e5f882be1900e43b”

    instance_type = "t2.micro"

}

I open the terminal and navigate where the project is. Then I type the command:

terraform init

This command initialises my working directory containing Terraform configuration files. In brief, it:

• Downloads Providers: It downloads the necessary provider plugins specified in my configuration files.
• Initialises Backend: My file is now stored locally on my machine. However, it can be stored remotely on Amazon S3, Azure Blob Storage, Terraform Cloud etc. to ease collaborations between developers.
• Installs Modules: in case these are used.
• Initializes Working Directory: It initializes the working directory for Terraform, creating necessary files and directories.

There is a well know problem that you might encounter at this stage:

'Error: Failed to query available provider packages

│ Could not retrieve the list of available versions for provider hashicorp/aws: could not connect to registry.terraform.io: failed to request

│ discovery document: Get https://registry.terraform.io/.well-known/terraform.json'

If I curl https://registry.terraform.io/.well-known/terraform.json, I receive 'curl: (35) Recv failure: Connection was reset.'

To solve this, I simply downloaded a free VPN and did the terraform init while logged into the VPN.

If the command is executed correctly (green output in the terminal), you will find a series of new files in the working directory. One of these is the "terraform.tfstate" which is fundamental to Terraform's operation, providing a snapshot of the infrastructure's desired state and serving as the basis for making changes and maintaining infrastructure. Proper management and protection of the state file are crucial for effective Terraform workflows.

Next command to type in the terminal after terraform init:

terraform plan

This command compares what has been built already with the plan I coded in Terraform. It basically allows me to review proposed changes and ensure they align with my intentions before making them live. The green plus sign stands for “creation” and the red minus sign stands for “destruction”. Every resource is evaluated in comparison to the current state.

Since nothing was created before for my AWS user, terraform plan informs me that one resource only (the EC2 instance) will be added ('Plan: 1 to add, 0 to change, 0 to destroy.')

Now I can terraform apply to launch the instance ("yes" to approve the action when prompted). If I go into my AWS account under EC2 I can verify that the t2.micro instance with AMI ami-0e5f882be1900e43b is indeed running.

terraform destroy to destroy it (remember to do it to avoid costs of running instances on AWS).