A step-by-step tutorial with a demonstration
You’re most likely accustomed to running Python scripts on your local machine. However, that machine may not be able to meet the compute or storage requirements needed to run certain operations.
For those cases, one can leverage virtual instances to meet any requirements that can’t be met on premise.
Here, we perform a quick demo to show how Python scripts can be run in Amazon’s EC2 instances.
Demo
What better way to explain how to do something than with a demonstration?
As part of a guide, we will aim to run a Python script named “random_number_generator.py” in an EC2 instance.
Pretty simple, right? The script just prints out a random number using the Numpy package.
It’s important to keep in mind the version of Python as well as the versions of the packages that are needed to run the code successfully (my current environment uses Python 3.9.13 and uses just the Numpy module).
To get the Python version used in a machine, simply enter the following command in the command line:
python --version
To get the list of packages and their versions installed with pip, use the following command in the command line.
pip list
You can install packages in your instance one by one, but that can be time-consuming. You also run the risk of getting the wrong version of a package. So, it’s recommended to store all packages in a “requirements.txt” file, which can be done with the following command.
pip freeze > requirements.txt
Step 1: Launch the EC2 Instance
First, the EC2 instance that the script will be run in needs to be launched.
I won’t delve into this step too much, since there are countless tutorials on how to create an EC2 instance on AWS (and it’s not designed to be difficult in the first place).
Feel free to choose the pricing and security configurations that fit your use case. Since this is just a simple demo, we’ll use using a free tier instance with no additional security measures.
The more important details when configuring an instance are as follows:
1. Choosing an Amazon Machine Image (AMI)
The first thing you will be asked to do when launching an instance is to choose your AMI.
The AMI you choose will determine the instance’s username, which you will need to know to connect to your instance later on. For those unsure of what their instance’s username is, the AWS documentation covers the default username assigned to instances launched with each AMI.
For our example, we will launch an instance using the Amazon Linux AMI. The username for this type of instance is “ec2-user”.
The AMI of choice will also influence the commands that will be needed to facilitate operations with the command line. For instance, the commands used for an instance launched with Amazon Linux AMI may not match an instance launched with Ubuntu AMI.
2. Key pair
You will also need to create a key pair or use an existing key (as a .pem file) for accessing your instance.
For our example, we will create a private key called “test_key.pem”
Once you have completed all sections, click “Launch instance”.
Step 2: Connect to the instance
AWS allows you to connect to your instance using secure shell (SSH).
To find the ssh command needed to connect to the instance, select your instance in the EC2 console and click the “Connect” tab on the top. After that, select the “SSH client” tab.
You will be presented with the command you will need to connect to your instance. You are also shown the public DNS of your instance (you will need this later, so keep this in mind).
In this case, the command for connecting to this instance is:
ssh -i "test_key.pem" ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com
The ssh command for connecting to the instance has to be executed from the location of your key file. On your terminal window, change the directory to the folder containing the private key and enter the provided command. Select “yes” when prompted. If successful, the output should something look like the following:
Voila! You have now connected to your instance!
Step 3: Install Python (Optional)
Your EC2 instance should already offer a version of Python3, which you can verify yourself with the python3 --version command.
If you are looking for a specific version of Python, you can install it yourself (skip this step if this doesn’t apply to you).
The Amazon Linux instance provides Python 3.7. For the sake of the demo, let’s assume that we need to match our local machine and install Python 3.9.13.
Note: The following commands are meant for instances launched with Amazon Linux AMI. They may not be suited for instances launched with other AMI (e.g., Ubuntu).
First, we install the dependencies for Python.
sudo yum -y groupinstall "Development Tools"
sudo yum -y install openssl-devel bzip2-devel libffi-devel
Next, we download the Python version of interest with the wget command.
sudo yum -y install wget
wget https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tgz
The instance should now have a file named “Python-3.9.13.tgz”.
Note: these commands are used for downloading Python version 3.9.13. To get the Python version of your choice, simply replace the version number in the commands with the one of your choosing.
After downloading the file, extract it.
tar xvf Python-3.9.13.tgz
The extracted Python folder should now be in the EC2 instance (you can confirm this using the ls command).
Move to the created Python folder and perform the installation.
cd Python-*/
./configure --enable-optimizations
sudo make altinstall
With that, the new version of Python should be successfully installed! To test this, we can enter the Python3.9 --version command.
Step 4: Copy files to the EC2 instance
Now that we have the correct version of Python, we can copy all files into the instance. In our case, the files being copied are the “random_number_generator.py” and “requirements.txt” files.
We will leave the Python folder and create a new folder named “project”, which will contain the two files. This step is optional; you can place your files wherever you’d like in the instance.
Now, open a new terminal window (let’s call this terminal 2). Terminal 2 will be used to copy the files in the local machine to the EC2 instance.
Files can be copied from one location to another with the scp (secured copy) command. The command for copying files from a local machine to an EC2 instance has the following format:
scp -i </path/key-pair-name.pem> </path/file_to_copy> <username>@<instance_public_dns>:<destination_path>
Note: To get the EC2 instance’s public DNS, go to the SSH client tab in the AWS EC2 console (see step 2).
The “random_number_generator.py” and “requirements.txt” files can be copied into the “project folder” in the instance with these commands:
scp -i test_key.pem random_number_generator.py ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com:/home/ec2-user/projectscp -i test_key.pem requirements.txt ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com:/home/ec2-user/project
If there are no issues, the command should yield the following output:
When you go back to terminal 1 (i.e. terminal with the EC2 instance), you should be able to see the files in the destination path you stated in the scp command.
Step 5: Create a virtual environment
At this stage, you might be tempted to install your packages so that you can run your code right away. However, installing packages globally can result in conflicting behavior with the system package manager.
So, it’s better to work in a virtual environment instead.
We will create a virtual environment named “test_venv” and then enter that environment.
Python3.9 -m venv test_env
source test_env/bin/activate
Once you activate the virtual environment, you should be able to see its name in parenthesis to the left of the current directory.
Now that we are in the virtual environment, we can install all the necessary packages needed to run the script. If you have your packages and versions stored in a requirements.txt file, you can perform all installations with a single command.
pip3.9 install -r requirements.txt
Once again, this command is for Python 3.9. Modify the commands based on the version of Python used.
Step 6: Run Your Code!
You’ve created your EC2 instance, connected to the instance, installed the correct Python version, transferred your files to the instance, and set up your virtual environment. Finally, you can run your Python scripts in your new instance!
Once you are done working in the virtual environment, you can leave it with the deactivate command.
Friendly Reminder: Stop Your Instance
EC2 instances use a pay-as-you-go pricing model, so always stop your instance when you are done using them! Otherwise, you will find yourself getting an unpleasant bill.
Even if you are working on a free-tier instance, it’s best to build the habit of stopping instances as soon as you are done with them early on.
Conclusion
With this demo, you’ve hopefully managed to run your scripts on AWS instances with little trouble.
Even if you primarily write programs that can be run on local machines, it’s good to have some exposure to computing services offered by cloud platforms since you’ll be prepared when your work has greater computational demand.
I wish you the best of luck in your data science endeavors!
A step-by-step tutorial with a demonstration
You’re most likely accustomed to running Python scripts on your local machine. However, that machine may not be able to meet the compute or storage requirements needed to run certain operations.
For those cases, one can leverage virtual instances to meet any requirements that can’t be met on premise.
Here, we perform a quick demo to show how Python scripts can be run in Amazon’s EC2 instances.
Demo
What better way to explain how to do something than with a demonstration?
As part of a guide, we will aim to run a Python script named “random_number_generator.py” in an EC2 instance.
Pretty simple, right? The script just prints out a random number using the Numpy package.
It’s important to keep in mind the version of Python as well as the versions of the packages that are needed to run the code successfully (my current environment uses Python 3.9.13 and uses just the Numpy module).
To get the Python version used in a machine, simply enter the following command in the command line:
python --version
To get the list of packages and their versions installed with pip, use the following command in the command line.
pip list
You can install packages in your instance one by one, but that can be time-consuming. You also run the risk of getting the wrong version of a package. So, it’s recommended to store all packages in a “requirements.txt” file, which can be done with the following command.
pip freeze > requirements.txt
Step 1: Launch the EC2 Instance
First, the EC2 instance that the script will be run in needs to be launched.
I won’t delve into this step too much, since there are countless tutorials on how to create an EC2 instance on AWS (and it’s not designed to be difficult in the first place).
Feel free to choose the pricing and security configurations that fit your use case. Since this is just a simple demo, we’ll use using a free tier instance with no additional security measures.
The more important details when configuring an instance are as follows:
1. Choosing an Amazon Machine Image (AMI)
The first thing you will be asked to do when launching an instance is to choose your AMI.
The AMI you choose will determine the instance’s username, which you will need to know to connect to your instance later on. For those unsure of what their instance’s username is, the AWS documentation covers the default username assigned to instances launched with each AMI.
For our example, we will launch an instance using the Amazon Linux AMI. The username for this type of instance is “ec2-user”.
The AMI of choice will also influence the commands that will be needed to facilitate operations with the command line. For instance, the commands used for an instance launched with Amazon Linux AMI may not match an instance launched with Ubuntu AMI.
2. Key pair
You will also need to create a key pair or use an existing key (as a .pem file) for accessing your instance.
For our example, we will create a private key called “test_key.pem”
Once you have completed all sections, click “Launch instance”.
Step 2: Connect to the instance
AWS allows you to connect to your instance using secure shell (SSH).
To find the ssh command needed to connect to the instance, select your instance in the EC2 console and click the “Connect” tab on the top. After that, select the “SSH client” tab.
You will be presented with the command you will need to connect to your instance. You are also shown the public DNS of your instance (you will need this later, so keep this in mind).
In this case, the command for connecting to this instance is:
ssh -i "test_key.pem" ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com
The ssh command for connecting to the instance has to be executed from the location of your key file. On your terminal window, change the directory to the folder containing the private key and enter the provided command. Select “yes” when prompted. If successful, the output should something look like the following:
Voila! You have now connected to your instance!
Step 3: Install Python (Optional)
Your EC2 instance should already offer a version of Python3, which you can verify yourself with the python3 --version command.
If you are looking for a specific version of Python, you can install it yourself (skip this step if this doesn’t apply to you).
The Amazon Linux instance provides Python 3.7. For the sake of the demo, let’s assume that we need to match our local machine and install Python 3.9.13.
Note: The following commands are meant for instances launched with Amazon Linux AMI. They may not be suited for instances launched with other AMI (e.g., Ubuntu).
First, we install the dependencies for Python.
sudo yum -y groupinstall "Development Tools"
sudo yum -y install openssl-devel bzip2-devel libffi-devel
Next, we download the Python version of interest with the wget command.
sudo yum -y install wget
wget https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tgz
The instance should now have a file named “Python-3.9.13.tgz”.
Note: these commands are used for downloading Python version 3.9.13. To get the Python version of your choice, simply replace the version number in the commands with the one of your choosing.
After downloading the file, extract it.
tar xvf Python-3.9.13.tgz
The extracted Python folder should now be in the EC2 instance (you can confirm this using the ls command).
Move to the created Python folder and perform the installation.
cd Python-*/
./configure --enable-optimizations
sudo make altinstall
With that, the new version of Python should be successfully installed! To test this, we can enter the Python3.9 --version command.
Step 4: Copy files to the EC2 instance
Now that we have the correct version of Python, we can copy all files into the instance. In our case, the files being copied are the “random_number_generator.py” and “requirements.txt” files.
We will leave the Python folder and create a new folder named “project”, which will contain the two files. This step is optional; you can place your files wherever you’d like in the instance.
Now, open a new terminal window (let’s call this terminal 2). Terminal 2 will be used to copy the files in the local machine to the EC2 instance.
Files can be copied from one location to another with the scp (secured copy) command. The command for copying files from a local machine to an EC2 instance has the following format:
scp -i </path/key-pair-name.pem> </path/file_to_copy> <username>@<instance_public_dns>:<destination_path>
Note: To get the EC2 instance’s public DNS, go to the SSH client tab in the AWS EC2 console (see step 2).
The “random_number_generator.py” and “requirements.txt” files can be copied into the “project folder” in the instance with these commands:
scp -i test_key.pem random_number_generator.py ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com:/home/ec2-user/projectscp -i test_key.pem requirements.txt ec2-user@ec2-34-204-68-237.compute-1.amazonaws.com:/home/ec2-user/project
If there are no issues, the command should yield the following output:
When you go back to terminal 1 (i.e. terminal with the EC2 instance), you should be able to see the files in the destination path you stated in the scp command.
Step 5: Create a virtual environment
At this stage, you might be tempted to install your packages so that you can run your code right away. However, installing packages globally can result in conflicting behavior with the system package manager.
So, it’s better to work in a virtual environment instead.
We will create a virtual environment named “test_venv” and then enter that environment.
Python3.9 -m venv test_env
source test_env/bin/activate
Once you activate the virtual environment, you should be able to see its name in parenthesis to the left of the current directory.
Now that we are in the virtual environment, we can install all the necessary packages needed to run the script. If you have your packages and versions stored in a requirements.txt file, you can perform all installations with a single command.
pip3.9 install -r requirements.txt
Once again, this command is for Python 3.9. Modify the commands based on the version of Python used.
Step 6: Run Your Code!
You’ve created your EC2 instance, connected to the instance, installed the correct Python version, transferred your files to the instance, and set up your virtual environment. Finally, you can run your Python scripts in your new instance!
Once you are done working in the virtual environment, you can leave it with the deactivate command.
Friendly Reminder: Stop Your Instance
EC2 instances use a pay-as-you-go pricing model, so always stop your instance when you are done using them! Otherwise, you will find yourself getting an unpleasant bill.
Even if you are working on a free-tier instance, it’s best to build the habit of stopping instances as soon as you are done with them early on.
Conclusion
With this demo, you’ve hopefully managed to run your scripts on AWS instances with little trouble.
Even if you primarily write programs that can be run on local machines, it’s good to have some exposure to computing services offered by cloud platforms since you’ll be prepared when your work has greater computational demand.
I wish you the best of luck in your data science endeavors!