Conversational Sentiment Analysis on Audio Data | by Avi Chawla | Jul, 2022

Analyzing sentiment in Speech

Sentiment Analysis, also known as opinion mining, is a popular task in Natural Language Processing (NLP) due to its diverse industrial applications. In the context of applying NLP techniques specifically to textual data, the primary objective is to train a model that can classify a given piece of text between different sentiment classes. A high-level overview of a sentiment classifier is shown in the image below.

For instance, the classes for a three-class classification problem can be Positive, Negative and Neutral. An example of the three-class sentiment analysis problem is the popular Twitter Sentiment Analysis dataset which is an Entity-level sentiment analysis task on multi-lingual tweets posted by various users on Twitter.

While most of the prior research and development in NLP has primarily focused on applying sentiment analysis over text, in recent times, we have seen massive adoption and popularity of speech-based interaction tools among users, veering researchers and organizations to build sentiment classifiers in the speech space.

Therefore, this post will demonstrate how to build a sentiment analysis system on conversational data using the AssemblyAI API and Python. The end-to-end system holds extensive applicability in areas involving rigorous customer support and feedback evaluation — making it an important and valuable problem to solve, especially in the speech domain. Towards the end, I’ll also demonstrate an extensive analysis to enhance the interpretability of the obtained results and draw appropriate insights from the data.

You can find the code for this article here. Moreover, the highlight of the article is as follows:

Sentiment Analysis on Conversational Audio Data
Sentiment Analysis Results
Sentiment Analysis Insights

In this section, I am going to demonstrate the use of AssemblyAI API to classify individual sentences in a given piece of pre-recorded voice conversation into three sentiment classes: Positive, Negative and Neutral.

Step 1: Installing Requirements

There are very few requirements to build the sentiment classifier. In terms of python libraries, we need therequests package only in Python. This can be done as follows:

pip install requests

Step 2: Generating your API Token

The next step is to create an account on the AssemblyAI website, which you can do for free. Once done, you will get your private API access key, which we will use to access the speech-to-text models.

Step 3: Uploading Audio File

For the purpose of this tutorial, I’ll use a pre-recorded audio conversation between two people to perform sentiment analysis on. Once you have obtained the API Key, you can proceed with the sentiment classification task on the pre-recorded audio file.

However, before doing that, you will need to upload the audio file so that it can be accessed via a URL. Options include uploading to an AWS S3 bucket, audio hosting services like SoundCloud or AssemblyAI’s self-hosting services, etc. I have uploaded the audio file to SoundCloud, which can be accessed below.

Once the status changes to completed, you will receive a response similar to the one mentioned below.

1. The status in the JSON response is marked as completed. This indicates that there were no errors in transcribing the audio.
2. The text key contains the entire transcription of the input audio conversation, and it includes 22 sentences.
3. As the audio file is composed of multiple speakers, we see all speaker keys within the words key as Not Null. The speaker key is either “A” or “B.”
4. We can see a confidence score for all the individual words and the entire transcription text. The score ranges from 0 to 1–with 0 being the lowest and 1 being the highest.
5. The results for sentiment analysis on each of the 22 individual sentences in the audio are accessible using the sentiment_analysis_results key of the JSON response.
6. Corresponding to each sentence, we get a confidence score similar to that in point 4 above.
7. The sentiment of each sentence can be retrieved using the sentiment key of a sentence’s dictionary. The sentiment analysis result for the second sentence is shown below:

JSONs are usually hard to read and interpret. Therefore, to make the data visually appealing and to conduct further analysis, let’s convert the sentiment analysis results above to a DataFrame. We will store the text, the duration of the sentence, its speaker, and the sentiment of the sentence. This is implemented below:

The DataFrame generated with the code snippet above is shown in the image below. Here, we have the 22 sentences that were spoken during the conversation along with the corresponding speaker labels (“A” and “B”), their duration in seconds, and the sentiment of the sentence as predicted by the model.

#1 Speaker distribution

The number of sentences spoken by each of the speakers can be calculated using the value_counts() method as shown below:

To view the percentage distribution of the speakers, we can pass normalize = True to the value_counts() method as follows:

Both the speakers “A” and “B” contributed equally to the conversation in terms of the number of sentences.

#2 Speaker Duration distribution

Next, let’s compute the individual contribution of each of the speakers in the conversation. This is shown below:

We use the groupby() method and compute the total duration of their speech. Speaker A is the dominant speaker in terms of duration.

#3 Sentiment Distribution

Out of the 22 sentences spoken during the conversation, only three were tagged as negative sentiment. Moreover, none of the sentences were predicted as positive sentiment.

The normalized distribution can be calculated as follows:

#4 Sentiment Distribution on Speaker-level

Finally, let’s compute the distribution of sentiment across individual speakers. Here, instead of using the groupby() method, we will use crosstab() for better visualization. This is demonstrated below:

The fraction of negative sentences spoken by Speaker “A” were more than that of Speaker “B”.

#5 Average Sentence Duration on Sentiment-level

Lastly, we shall compute the average duration of the sentences belonging to the individual sentiment classes. This is implemented below using the groupby() method:

The average duration of negative sentences are smaller than that of the neutral sentences.

To conclude, in this post, we discussed a particular NLP use case of the AssemblyAI API. Specifically, we saw how to build a sentiment classification module on a pre-recorded audio file comprising multiple speakers. Finally, we did an extensive analysis on the sentiment analysis results. The obtained results from the API highlighted the sentiments of the 22 individual sentences in the input audio file.

You can find the code for this article here.

In the upcoming posts, I will discuss more use-cases of the AssemblyAI API, such as Entity Detection, Content Moderation, and more, from both the technical and practical perspectives.

See you next time. Thanks for reading.

Analyzing sentiment in Speech

Sentiment Analysis, also known as opinion mining, is a popular task in Natural Language Processing (NLP) due to its diverse industrial applications. In the context of applying NLP techniques specifically to textual data, the primary objective is to train a model that can classify a given piece of text between different sentiment classes. A high-level overview of a sentiment classifier is shown in the image below.

For instance, the classes for a three-class classification problem can be Positive, Negative and Neutral. An example of the three-class sentiment analysis problem is the popular Twitter Sentiment Analysis dataset which is an Entity-level sentiment analysis task on multi-lingual tweets posted by various users on Twitter.

While most of the prior research and development in NLP has primarily focused on applying sentiment analysis over text, in recent times, we have seen massive adoption and popularity of speech-based interaction tools among users, veering researchers and organizations to build sentiment classifiers in the speech space.

Therefore, this post will demonstrate how to build a sentiment analysis system on conversational data using the AssemblyAI API and Python. The end-to-end system holds extensive applicability in areas involving rigorous customer support and feedback evaluation — making it an important and valuable problem to solve, especially in the speech domain. Towards the end, I’ll also demonstrate an extensive analysis to enhance the interpretability of the obtained results and draw appropriate insights from the data.

You can find the code for this article here. Moreover, the highlight of the article is as follows:

Sentiment Analysis on Conversational Audio Data
Sentiment Analysis Results
Sentiment Analysis Insights

In this section, I am going to demonstrate the use of AssemblyAI API to classify individual sentences in a given piece of pre-recorded voice conversation into three sentiment classes: Positive, Negative and Neutral.

Step 1: Installing Requirements

There are very few requirements to build the sentiment classifier. In terms of python libraries, we need therequests package only in Python. This can be done as follows:

pip install requests

Step 2: Generating your API Token

The next step is to create an account on the AssemblyAI website, which you can do for free. Once done, you will get your private API access key, which we will use to access the speech-to-text models.

Step 3: Uploading Audio File

For the purpose of this tutorial, I’ll use a pre-recorded audio conversation between two people to perform sentiment analysis on. Once you have obtained the API Key, you can proceed with the sentiment classification task on the pre-recorded audio file.

However, before doing that, you will need to upload the audio file so that it can be accessed via a URL. Options include uploading to an AWS S3 bucket, audio hosting services like SoundCloud or AssemblyAI’s self-hosting services, etc. I have uploaded the audio file to SoundCloud, which can be accessed below.

Once the status changes to completed, you will receive a response similar to the one mentioned below.

1. The status in the JSON response is marked as completed. This indicates that there were no errors in transcribing the audio.
2. The text key contains the entire transcription of the input audio conversation, and it includes 22 sentences.
3. As the audio file is composed of multiple speakers, we see all speaker keys within the words key as Not Null. The speaker key is either “A” or “B.”
4. We can see a confidence score for all the individual words and the entire transcription text. The score ranges from 0 to 1–with 0 being the lowest and 1 being the highest.
5. The results for sentiment analysis on each of the 22 individual sentences in the audio are accessible using the sentiment_analysis_results key of the JSON response.
6. Corresponding to each sentence, we get a confidence score similar to that in point 4 above.
7. The sentiment of each sentence can be retrieved using the sentiment key of a sentence’s dictionary. The sentiment analysis result for the second sentence is shown below:

JSONs are usually hard to read and interpret. Therefore, to make the data visually appealing and to conduct further analysis, let’s convert the sentiment analysis results above to a DataFrame. We will store the text, the duration of the sentence, its speaker, and the sentiment of the sentence. This is implemented below:

The DataFrame generated with the code snippet above is shown in the image below. Here, we have the 22 sentences that were spoken during the conversation along with the corresponding speaker labels (“A” and “B”), their duration in seconds, and the sentiment of the sentence as predicted by the model.

#1 Speaker distribution

The number of sentences spoken by each of the speakers can be calculated using the value_counts() method as shown below:

To view the percentage distribution of the speakers, we can pass normalize = True to the value_counts() method as follows:

Both the speakers “A” and “B” contributed equally to the conversation in terms of the number of sentences.

#2 Speaker Duration distribution

Next, let’s compute the individual contribution of each of the speakers in the conversation. This is shown below:

We use the groupby() method and compute the total duration of their speech. Speaker A is the dominant speaker in terms of duration.

#3 Sentiment Distribution

Out of the 22 sentences spoken during the conversation, only three were tagged as negative sentiment. Moreover, none of the sentences were predicted as positive sentiment.

The normalized distribution can be calculated as follows:

#4 Sentiment Distribution on Speaker-level

Finally, let’s compute the distribution of sentiment across individual speakers. Here, instead of using the groupby() method, we will use crosstab() for better visualization. This is demonstrated below:

The fraction of negative sentences spoken by Speaker “A” were more than that of Speaker “B”.

#5 Average Sentence Duration on Sentiment-level

Lastly, we shall compute the average duration of the sentences belonging to the individual sentiment classes. This is implemented below using the groupby() method:

The average duration of negative sentences are smaller than that of the neutral sentences.

To conclude, in this post, we discussed a particular NLP use case of the AssemblyAI API. Specifically, we saw how to build a sentiment classification module on a pre-recorded audio file comprising multiple speakers. Finally, we did an extensive analysis on the sentiment analysis results. The obtained results from the API highlighted the sentiments of the 22 individual sentences in the input audio file.

You can find the code for this article here.

In the upcoming posts, I will discuss more use-cases of the AssemblyAI API, such as Entity Detection, Content Moderation, and more, from both the technical and practical perspectives.

See you next time. Thanks for reading.