CREATING A CHATBOT FOR ANY WEBSITE WITH OPENAI AND LANGCHAIN
GOAL:
This project aims to create chatbot for any website by using langchain and OpenAI API. You should have the valid API from OpenAI. I used my own website as database.
LOAD THE DATA¶
In [1]:
!pip install -q langchain
In [2]:
!pip install -q --upgrade langchain
In [3]:
!pip show langchain
Name: langchain Version: 0.0.344 Summary: Building applications with LLMs through composability Home-page: https://github.com/langchain-ai/langchain Author: Author-email: License: MIT Location: /Users/selen/opt/miniconda3/lib/python3.9/site-packages Requires: tenacity, langchain-core, PyYAML, SQLAlchemy, requests, anyio, langsmith, async-timeout, aiohttp, pydantic, jsonpatch, dataclasses-json, numpy Required-by:
In [4]:
#!pip install -q langchain==0.0.133 # use selenium for latest version of langchain
In [5]:
!pip install selenium
Requirement already satisfied: selenium in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (4.15.2) Requirement already satisfied: urllib3[socks]<3,>=1.26 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from selenium) (1.26.9) Requirement already satisfied: trio-websocket~=0.9 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from selenium) (0.11.1) Requirement already satisfied: trio~=0.17 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from selenium) (0.23.1) Requirement already satisfied: certifi>=2021.10.8 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from selenium) (2022.9.24) Requirement already satisfied: sniffio>=1.3.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio~=0.17->selenium) (1.3.0) Requirement already satisfied: exceptiongroup>=1.0.0rc9 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio~=0.17->selenium) (1.2.0) Requirement already satisfied: idna in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio~=0.17->selenium) (3.3) Requirement already satisfied: sortedcontainers in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio~=0.17->selenium) (2.4.0) Requirement already satisfied: attrs>=20.1.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio~=0.17->selenium) (22.1.0) Requirement already satisfied: outcome in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio~=0.17->selenium) (1.3.0.post0) Requirement already satisfied: wsproto>=0.14 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from trio-websocket~=0.9->selenium) (1.2.0) Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from urllib3[socks]<3,>=1.26->selenium) (1.7.1) Requirement already satisfied: h11<1,>=0.9.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium) (0.14.0)
In [6]:
from langchain.document_loaders import SeleniumURLLoader
/Users/selen/opt/miniconda3/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.2.0)/charset_normalizer (2.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
In [7]:
#from langchain.document_loaders import UnstructuredURLLoader
In [8]:
urls = [
'https://arslanselen.com/',
'https://arslanselen.com/twitter-api-v2-search-with-tweepy/',
'https://arslanselen.com/x/',
'https://arslanselen.com/213-2/',
'https://arslanselen.com/267-2/',
'https://arslanselen.com/cs224n-c1/',
'https://arslanselen.com/436-2/',
'https://arslanselen.com/523-2/',
'https://arslanselen.com/551-2/'
]
In [9]:
loader = SeleniumURLLoader(urls=urls)
In [10]:
#loaders = UnstructuredURLLoader(urls=urls)
data = loader.load()
In [11]:
data
Out[11]:
[Document(page_content='My data science diary\n\nSelen Arslan\n\nWelcome to my data science diary, where you will find documentation of my data science journey. On this website, you will find a showcase of my projects and experiences in the field of data analysis and natural language processing. As a graduate in the University of Padua in the ICT for Internet and Multimedia (Cybersystems) department and Electrical and Electronics Engineering from the University of Turkish Aeronautical Association, I have developed a strong passion in these areas through my data science journey. I am excited to share with you my projects, research, and professional experiences that have shaped my understanding and approach to data science. I invite you to take a look around and discover my approach to data science.\n\nMail\n\nLinkedIn\n\nTWITTER API V2 SEARCH WITH TWEEPY\n\nSCRAPPING REDDIT DATA WITH PMAW\n\nDATA CLEANSING FOR BERTOPIC\n\nUMAP, HDBSCAN, c-TF-IDF AND BERTopic\n\nINTRODUCTION TO WORD EMBEDDINGS\n\nWORD2VEC : CBOW AND SKIPGRAM\n\nINTRODUCTION TO BERT\n\nTRANSFORMERS', metadata={'source': 'https://arslanselen.com/', 'title': 'Selen Arslan', 'description': 'My data science diary Welcome to my data science diary, where you will find documentation of my data science journey. On this website, you will find a showcase of my projects and experiences in the field of data analysis and natural language processing. As a graduate in the University of Padua in the ICT for…', 'language': 'en-US'}),
Document(page_content='TWITTER API V2 SEARCH WITH TWEEPY\n\nHow to use Tweepy to get data for a specific hashtag with Twitter API V2?\n\nTwitter API V2 for Academic Research account is used for this project. Please check for more information here.\n\nWHAT IS TWEEPY?\n\nWHY TWEEPY?\n\nTweepy is an open source Python package that gives you a very convenient way to access the Twitter API with Python. It is defined as ‘An easy-to-use Python library for accessing the Twitter API’ in the original website. You can chech the documentation for more detail from here.\n\nTweepy is not a third party API application to get data from Twitter. It is a Python library used to access real Twitter API which requires to have developer account and keys.You can easily find the all the methods you need in Tweepy documentation. Zero experience is not a problem for Tweepy user. Easy to learn and apply.\n\nSET UP\n\nSet up process for Tweepy is pretty easy. You only need one line code to install tweepy.\n\nBe sure you download the latest version to avoid errors in next steps.\n\nupgrade tweepy\n\nTweepy is set up and ready to use. Now we will connect to the Twitter API by Tweepy. Bearer Token from developer account will be used in next step.\n\nCONNECTING TWITTER API\n\nThe Twitter API enables programmatic access to Twitter in unique and advanced ways. It is quite easy to use the twitter API to collect data. Only thing you need is bearer token from your developer account. A Bearer Token is a byte array of unspecified format that you generate using a script like a curl command. You can also obtain a Bearer Token from the developer portal inside the keys and tokens section of your App’s settings. We will be ready to collect data from Twitter after following codes:\n\nPut your Bearer Token instead of\xa0\'BEARER TOKEN\'\xa0in the following code.\n\n\'BEARER TOKEN\'\n\ntweepy.Client is Tweepy’s interface for Twitter API v2. Details can be found in\xa0here. We will initialize the client with Bearer Token.\n\n= tweepy\n\n.Client\n\n(bearer_token\n\n, wait_on_rate_limit\n\nTrue\n\nCOLLECT DATA FROM TWITTER\n\nTopic\n\nI want to collect data about postpartum. We will search for tweets includes word ‘postpartum‘ or hashtag ‘#postpartum‘.\n\nTime Period\n\nI choose 2021. I want to have a homogeneous dataset which means same amount of data for each month.\n\nLanguage\n\nWe will look for English tweets.\n\nType of Tweet\n\nI want to have original tweets, so we will not search for retweets.\n\nWe will decide later which information will be kept.\n\nLet’s work on the time period. Unfortunately, Twitter API does not support clear distribution on given time period. We will try our best to have most homogeneous dataset for whole year 2021. We will collect 50 tweets from each day during 2021. So, 50 tweets * 365 days = 18250. We are expecting to see 18250 rows in our dataset. The time format of request process is %Y-%m-%dTH:M:SZ. First we should create a list of days in 2021 in this format.\n\n#start date: 2021-january 1 st\nstart_date\n\n= datetime\n\n.date\n\n2021\n\n#end date: 2022-january 1 st\n\n#end date (01.01.2022) will not be included in data collecting part.\n\n#last day will be 31.12.2021\nend_date\n\n= datetime\n\n.date\n\n2022\n\n# delta time\ndelta\n\n= datetime\n\n.timedelta\n\n(days\n\n# iterate over range of dates\n\nwhile\n\n(start_date\n\n<= end_date\n\n:\n d1\n\n= datetime\n\n.datetime\n\n.strptime\n\nstr\n\n(start_date\n\n"%Y-%m-%d"\n\n)\n new_format\n\n"%Y-%m-%dT00:00:00Z"\n date\n\n.append\n\n(d1\n\n.strftime\n\n(new_format\n\n)\n start_date\n\n+= delta\n\nBasically, code structures is:\n\nfor response in tweepy.Paginator(client.search_all_tweets,\n\nquery =\n\ntweet_fields = 8\n\nexpansions =\n\nstart_time =\n\nend_time =\n\nmax_results=\n\nlimit=1)\n\nquery is used for keywords.\n\ntweet_fields and expansions are used to select required columns.\n\nstart_time and end_time represent time period.\n\nmax_results is maximum capacity of collecting data in each request.\n\nlimit is number of request in each loop.\n\nPlease check the\xa0Twitter documentation page\xa0for more details.\n\nNote: This process will take some time.\n\n#iterates the days\n\nfor i\n\nin\n\nrange\n\nlen\n\n(date\n\n#first day of 2022 is not included\n\n#request process\n\nfor response\n\nin tweepy\n\n.Paginator\n\n(client\n\n.search_all_tweets\n\n, \n query\n\n\'(postpartum OR #postpartum) lang:en -is:retweet\'\n\n# tweets inclued \'postpartum\' or #postpartum \n tweet_fields\n\n\'entities,id,text,author_id,lang,created_at,public_metrics,referenced_tweets\'\n\n,\n expansions\n\n\'author_id,in_reply_to_user_id,referenced_tweets.id\'\n\n,\n start_time\n\n= date\n\n[i\n\n,\n end_time\n\n= date\n\n[i\n\n# (end_date)-(start_date) = 1 day\n max_results\n\n50\n\n, limit\n\n# 50 tweets in each request. we will have totaly 1 request for each day.\n \n time\n\n.sleep\n\n)\n df_tweet\n\n.append\n\n(response\n\nSo, we are searching for english original tweets which includes ‘postpartum’ or ‘#postpartum’.\n\nFinally we have the data. We should save it with column names which make sense in csv format. Following code maps the information we collect and store everything in a csv file.\n\nLet’s check the distribution of the tweets with following code.', metadata={'source': 'https://arslanselen.com/twitter-api-v2-search-with-tweepy/', 'title': 'Selen Arslan', 'description': 'TWITTER API V2 SEARCH WITH TWEEPY How to use Tweepy to get data for a specific hashtag with Twitter API V2? Twitter API V2 for Academic Research account is used for this project. Please check for more information here. WHAT IS TWEEPY? WHY TWEEPY? Tweepy is an open source Python package that gives you a…', 'language': 'en-US'}),
Document(page_content="26 January 2023\n\nSCRAPING REDDIT DATA WITH PMAW LIBRARY FOR SPECIFIC KEYWORD\n\nCollecting and Cleaning Reddit Data for Postpartum-related Submissions and Comments in 2021\n\nIn this script, we will be using the pmaw library to scrape Reddit data for a specific keyword. The data will be collectingfor the keyword “postpartum” within the year 2021. PMAW (Pushshift API Wrapper) is a Python wrapper for the Pushshift.io API. It allows for easy access to the data provided by the API, and provides a variety of useful functions for searching and filtering the data.\n\nREDDIT\n\nBefore we begin, it’s important to understand the structure of Reddit. Reddit is an online platform that allows users to submit, discuss and share content. The content can be organized in to different communities called “subreddits”. Each subreddit is focused on a specific topic, and users can submit posts or comments related to that topic. A submission on Reddit refers to a post that is submitted to a subreddit by a user. A comment, on the other hand, refers to a reply or discussion on a submission. In this script, we will be scraping the submission data (post) and comment data from Reddit.\n\nLet’s begin by importing the necessary libraries including Pandas, Numpy, datetime, and the PushshiftAPI library. We also install pmaw library.\n\nimport datetime\n\nas dt\n\nimport pandas\n\nas pd\n\nimport numpy\n\nas np\n!pip install pmaw pandas\n\nfrom pmaw\n\nimport PushshiftAPI\n\nimport matplotlib\n\n.pyplot\n\nas plt\n\nimport seaborn\n\nas sns\n\nimport ast\n\nSCRAPPING POST\n\nWe then use the PushshiftAPI to search for submissions that contain the keyword “postpartum” within the year 2021. The search_submissions() method is used to search for the submissions, and the ‘q’ parameter is used to specify the keyword to search for. The ‘after’ and ‘before’ parameters are used to specify the date range, and the ‘limit’ parameter is used to specify the maximum number of submissions to return. The data is stored in a dataframe called “df_post”.\n\n= PushshiftAPI\n\n)\nend_epoch\n\nint\n\n(dt\n\n.datetime\n\n2021\n\n12\n\n31\n\n.timestamp\n\n)\nstart_epoch\n\nint\n\n(dt\n\n.datetime\n\n2021\n\n.timestamp\n\n)\nsubmissions\n\n= api\n\n.search_submissions\n\n(q\n\n'postpartum'\n\n, after\n\n=start_epoch\n\n, before\n\n=end_epoch\n\n,limit\n\n30000\n\n)\ndf_post\n\n=pd\n\n.DataFrame\n\n(submissions\n\nNow we will start cleaning the data. We will first replace empty selftext rows with NaN. We then proceed to drop the selftext rows that contain NaN.\n\nWe also drop any duplicated selftext rows and keep the first one. This step is done because some of the posts are shared in different subreddits, which results in duplicate entries in our dataframe. By keeping only one instance of the duplicated post, we can ensure that our data is clean and accurate.\n\nWe also drop any rows that contain the selftext “[deleted]” or “[removed]” as they do not contain any relevant information.\n\nFinally, we save the cleaned dataframe to a CSV file.\n\nWith this script, we have successfully scraped Reddit data for the keyword “postpartum” within the year 2021 and cleaned the data to obtain a useful dataset for further analysis.\n\nSCRAPPING COMMENT\n\nThis code block is used to scrape comment data from Reddit that contain the keyword “postpartum” within the year 2021. We are using the same PushshiftAPI library that we used in the previous code block to scrape the submission data. The api.search_comments() method is used to search for comments that contain the keyword “postpartum”. The ‘q’ parameter is used to specify the keyword to search for, the ‘after’ and ‘before’ parameters are used to specify the date range and the ‘limit’ parameter is used to specify the maximum number of comments to return.\n\nOnce we have the comments, we are creating a dataframe from the comments and storing the data in the variable “comment_df”. Then we save the comment dataframe to a csv file. It’s similar to our first code block where we are scraping post data but this time we are scraping comments instead.\n\n= api\n\n.search_comments\n\n(q\n\n'postpartum'\n\n, after\n\n=start_epoch\n\n, before\n\n=end_epoch\n\n,limit\n\n70000\n\n)\ncomment_df\n\n=pd\n\n.DataFrame\n\n(comment\n\n)\ncomment_df\n\n.to_csv\n\n'YOUR PATH/df_comment_original.csv'\n\nMATCHING\n\nThis code block is used to match the comments with the related posts. In order to match the comments with the related posts, we first need to create a common identifier that we can use to join the two dataframes. The ‘parent_id column’ in the comment dataframe contains the id of the post that the comment is related to, but it has a prefix of “t3_” which we need to remove. We use the .str[] method to remove the prefix and store the cleaned parent id in a new column called parent_id_clean.\n\nWe make a copy of post dataframe and comment dataframe and store in “post_df1” and “comment_df1” respectively. These copies are used to avoid modifying the original dataframes. We create a dataframe from the parent_id_clean column of the comment dataframe and rename it to “id” using the .rename() method. We also create a dataframe from the id column of the post dataframe. Then we use the pd.merge() function to join these two dataframes on the ‘id’ column, so we have all the comments that match with their parent posts.We then drop the duplicate rows and reset the index of the resulting dataframe, and store it in the variable “merged_id”.\n\nThen we will create the post and comment datasets\n\n=pd\n\n.merge\n\n(post_df1\n\n,merged_id\n\n,how\n\n'inner'\n\n,on\n\n'id'\n\n)\nmerged_id\n\n'parent_id_clean'\n\n=merged_id\n\nid\n\n.copy\n\n)\nmerged_comment\n\n=pd\n\n.merge\n\n(comment_df1\n\n,merged_id\n\n,how\n\n'inner'\n\n,on\n\n'parent_id_clean'\n\n)\nmerged_comment\n\n'type'\n\n'comment'\nmerged_post\n\n'type'\n\n'post'\nmerged_comment\n\n.to_csv\n\n'YOUR PATH/matched_comment.csv'\n\n)\nmerged_post\n\n.to_csv\n\n'YOUR PATH/matched_post.csv'\n\nIn conclusion, this script uses the pmaw library to scrape submission data from Reddit for the keyword “postpartum” within a specific date range. The data is stored in a dataframe and cleaned to remove any duplicates or irrelevant data. Additionally, the script also scraps the comment data and matches it with the related post data to create two final datasets, one for posts and one for comments. These datasets can then be used for further analysis or modeling. Overall, this script provides a useful tool for collecting and cleaning data from Reddit for specific keywords and date ranges.", metadata={'source': 'https://arslanselen.com/x/', 'title': 'Selen Arslan', 'description': 'SCRAPING REDDIT DATA WITH PMAW LIBRARY FOR SPECIFIC KEYWORD Collecting and Cleaning Reddit Data for Postpartum-related Submissions and Comments in 2021 In this script, we will be using the pmaw library to scrape Reddit data for a specific keyword. The data will be collectingfor the keyword "postpartum" within the year 2021. PMAW (Pushshift API Wrapper)…', 'language': 'en-US'}),
Document(page_content='28 January 2023\n\nDATA PREPROCESSING FOR BERTOPIC\n\nStep-by-Step Guide to Cleaning and Preprocessing Data for BERT-based Topic Modeling\n\nData cleansing contains pre-steps before analysing process of the projects. These steps and order of steps are as important as the analysis process. In some cases, we can say that it is even more important. Because a dataset that has not been properly cleaned and still contains noise is not safe enough to use in the analysis process. Result will not be reliable enough. Data cleansing process basically contains couple of steps which may seems like same for all kind of projects. But, it is not true. You should be very carefull about order of steps to avoid loosing useful data.\n\nThese project represents the preprocessing steps for BERTopic. Check for more information about BERTopic.\n\nDATASET\n\nLet’s import the dataset. You can download the dataset:\n\nDataset\n\nimport pandas\n\nas pd\ndf\n\n=pd\n\n.read_csv\n\n\'YOUR_PATH/reddit_data.csv\'\n\nDataset contains Reddit posts and comments. You can see how to scrap the data from Reddit in my blog post. We will analyze the posts.\n\nTEXT TO SUBSENTENCES\n\nIf you are working with long Reddit texts and your goal is to analyze the emotions expressed in the text, dividing the texts into subsentences can be a very efficient approach. This is because emotions can change throughout a text, and by analyzing subsentences separately, you can more accurately capture the nuances of the emotions expressed.\n\nDividing the text into subsentences can also help you to identify patterns or themes in the emotional content of the text. This can allow you to gain a more detailed understanding of the emotions expressed in the text, which can be beneficial for tasks such as sentiment analysis or emotion recognition.\n\nAdditionally, by dividing the text into subsentences, you can also reduce the computational resources required to analyze the text. As the subsentences are shorter than the whole text, the model can process them more efficiently, which can be beneficial if you are working with a large dataset.\n\nIn summary, dividing long Reddit texts into subsentences can be an efficient approach for analyzing the emotions expressed in the text. It can help to capture the nuances of the emotions expressed, identify patterns or themes in the emotional content, and reduce the computational resources required to analyze the text.\n\nWe will use the Natural Language Toolkit (NLTK) library to tokenize the text in the “selftext” column of the DataFrame (df_post) into sentences. The sent_tokenize function from the NLTK library is used to split the text into a list of sentences, where each sentence is represented as a separate string.\n\nFor each text in the “selftext” column, we will tokenize it into sentences using the sent_tokenize function and append it to the list “sent_list”. This will give us a list of lists, where each inner list contains the sentences for one text.\n\nfrom nltk\n\n.tokenize\n\nimport sent_tokenize\nsent_list\n\nfor i\n\nin\n\nrange\n\nlen\n\n(df_post\n\n:\n text\n\n=df_post\n\n.selftext\n\n[i\n\n]\n sentence\n\n= sent_tokenize\n\n(text\n\n)\n sent_list\n\n.append\n\n(sentence\n\nWe will create a new DataFrame (df_post_seperated) to store the tokenized sentences and their corresponding main text. The DataFrame will have three columns: “main_text_index”, “main_text” and “sentences”.\n\nThe “main_text_index” column will contain the index of the main text in the original DataFrame, “main_text” column will contain the original text, and “sentences” column will contain the tokenized sentences.\n\nIn this block of code, we are creating an empty DataFrame called df_p_s with columns “main_text_index”, “sub_sentence_index” and “sub_sentence”. We are then using a nested for loop to iterate through each element in the sentences column of the df_post_seperated DataFrame. For each sentence, we are appending the sentence, its index, and the index of its parent text to the sub_sentence, sub_index, and main_index lists respectively. We then assign these lists as values to the corresponding columns in the df_p_s DataFrame. This DataFrame now contains the sub-sentences of the original texts with their respective indices for the main text and sub-sentence.\n\nHASHTAGS AND MENTIONS\n\nHashtag extraction needs to be done before data cleaning process, we will keep them for next analysis process.\n\nBefore cleaning the punctuation, we should remove the mentions which start with ‘@’. We don’t want to keep the mentioned usernames as the word, they will be noise in the dataset.\n\nDATA CLEANSING\n\nLet’s create a dataset to store the cleaned version of the text column in dataset.\n\nLet’s apply the functions.\n\nThis will take some time.\n\nSo, we removed all the noises from our dataset. But still we have things to do. The next step is tokenization. Here is the clean text up to now:\n\nTOKENIZATION\n\nTokenization is the process of splitting a phrase, sentence, paragraph, or an entire text document into smaller units, such as individual words or terms. Each of these smaller units are called tokens. There are different kind of tokenization techniques and different python libraries. word_tokenize from nltk library is used for this project.\n\nPunctuation is already cleaned and all letters are lowercased , we don’t need to know where the sentence starts or ends for topic analysis. We will focus on whole dataset as the big one piece of information to see what it is about.\n\n\'token\'\n\n=clean_df\n\n.clean_text\n\n.copy\n\nfor i\n\nin\n\nrange\n\nlen\n\n(clean_df\n\n:\n clean_df\n\n\'token\'\n\n.iloc\n\n[i\n\n=word_tokenize\n\n(clean_df\n\n.clean_text\n\n.iloc\n\n[i\n\nPART OF SPEECH TAGGING\n\nUniversal POS tags are part-of-speech tags used in Universal Dependencies (UD) project which has the plain and significant categories.\n\nHere are the POStags:\n\nADJ: adjective\n\nADP: adposition\n\nADV: adverb\n\nAUX: auxiliary\n\nCCONJ: coordinating conjunction\n\nDET: determiner\n\nINTJ: interjection\n\nNOUN: noun\n\nNUM: numeral\n\nPART: particle\n\nPRON: pronoun\n\nPROPN: proper noun\n\nPUNCT: punctuation\n\nSCONJ: subordinating conjunction\n\nSYM: symbol\n\nVERB: verb\n\nX: other\n\nCheck for more information here\n\nSTOP WORDS\n\nIn the Natural Language Toolkit (NLTK), a stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query. NLTK provides a list of commonly used stop words in various languages, which can be useful for natural language processing tasks such as text classification and text mining. To use NLTK’s list of stop words in your Python code, you can use the nltk.corpus.stopwords module, which contains lists of stop words for different languages.\n\nWe are going to remove the stopwords from the text in our dataframe df. To do this, we will first initialize some empty columns in the dataframe to store the stopword-removed tokens, stopword-removed text, extracted stopwords, and POS tags for the stopword-removed words.\n\nNext, we will use a for loop to iterate through each row of the dataframe. For each row, we will use a list comprehension to remove the stopwords from the token column and store the result in the token_stopword_removed column.\n\nWe will then use another list comprehension to extract the stopwords that were removed in the previous step and store the result in the extracted_stopword column.\n\nWe will also use the join() method to create a text string from the stopword-removed tokens and store it in the text_stopword_removed column.\n\nFinally, we will use a list comprehension to match the original POS tags with the stopword-removed words and store the result in the pos_stopword_removed column.\n\nWe will be using stopwords.words(‘english’) from the nltk library to get the list of english stopwords. With the help of list comprehension we will remove the stopwords from the token column and also match the original pos_tag column with the stopword removed token column.\n\nWe will also extract the stopwords that were removed from the token column and store it in extracted_stopword column and also join the stopword removed token column to form a text and store it in text_stopword_removed column.\n\n#stop words removing and matching removed version of the clean data with original postags\ndf\n\n\'token_stopword_removed\'\n\n\'\'\ndf\n\n\'text_stopword_removed\'\n\n\'\'\ndf\n\n\'extracted_stopword\'\n\n\'\'\ndf\n\n\'pos_stopword_removed\'\n\n\'\'\n\nfor i\n\nin\n\nrange\n\nlen\n\n(df\n\n#df.token[i]=ast.literal_eval(df.token[i])\n\n#df.pos_tag[i]=ast.literal_eval(df.pos_tag[i])\n\n#cleaning the stopwords\n df\n\n.token_stopword_removed\n\n[i\n\n[word\n\nfor word\n\nin df\n\n.token\n\n.iloc\n\n[i\n\nif\n\nnot word\n\nin stopwords\n\n.words\n\n\'english\'\n\n#check the extracted stopwords \n df\n\n.extracted_stopword\n\n[i\n\n[word\n\nfor word\n\nin df\n\n.token\n\n.iloc\n\n[i\n\nif word\n\nin stopwords\n\n.words\n\n\'english\'\n\n#creating text from stopword tokens\n df\n\n.text_stopword_removed\n\n[i\n\n" "\n\n.join\n\n(df\n\n.token_stopword_removed\n\n[i\n\n#matching the original pos tag with the stopword removed word \n df\n\n.pos_stopword_removed\n\n[i\n\n(word\n\n,tag\n\nfor word\n\n,tag\n\nin df\n\n.pos_tag\n\n[i\n\nif word\n\nnot\n\nin stopwords\n\n.words\n\n\'english\'\n\nTAG ANALYSIS\n\nThe cleanest version of text is preprocessed, hashtag-mention removed and stop word removed version of the text which is stored in column ‘text_stopword_removed’. pos tag information of the cleanest version of the text is stored in ‘pos_stopword_removed’ column. we will count the tags, verbs and order the tag counts. Also we will store the verbs with related postags for future analysis. You can modify this part as your wish.\n\nThis block of code counts the POS tags in the ‘pos_stopword_removed’ column and stores the count in ‘tag_count’ column. It also orders the tags by count in descending order and stores the result in ‘tag_order’ column. Additionally, it extracts all verbs from the ‘pos_stopword_removed’ column and stores them in ‘verb_tag’ column along with their count, and verb’s POS tag information in ‘verb_pos_tag’ column for future analysis. The goal is to provide a summary of the POS tags and verbs in the cleaned text for further analysis.\n\nLEMMATIZATION\n\nLemmatization is the process of reducing a word to its base or root form. It is similar to stemming, but it takes into account the word’s part-of-speech (POS) tag. This means that it uses the context of the word to determine its base form, rather than just removing common morphological affixes. For example, the lemma of the verb “running” would be “run”, and the lemma of the noun “dogs” would be “dog”.\n\nThe lemmatization process is done by comparing the word and its POS tag to a dictionary of base forms, such as WordNet. The lemmatizer then replaces the word with its corresponding base form, if it’s found in the dictionary. This process can be useful for text processing tasks such as text classification, information retrieval, and machine learning, because it reduces the dimensionality of the data and makes it easier to compare different words.\n\nIt is common practice to perform POS tagging before lemmatization because the lemmatizer needs the POS tag information to determine the base form of a word. The POS tag provides a clue to the lemmatizer about whether the word is a noun, verb, adjective or adverb. This information is used to select the right lemma from the dictionary, which can be different depending on the POS tag. For example, “running” as verb and “running” as noun have different base form, “run” and “running” respectively.\n\nHere is the final dataset:\n\nwh_mention clean_text token pos_tag token_stopword_removed text_stopword_removed extracted_stopword pos_stopword_removed tag_count tag_order verb_tag verb_pos_tag text_lemmatized lemmatized_postag lemmatized_verb_pos_tag 0 All my homies who have experienced hair change… all my homes who have experienced hair changes… [all, my, homes, who, have, experienced, hair,… [(all, DET), (my, PRON), (homes, NOUN), (who, … [homes, experienced, hair, changes, postpartum… homes experienced hair changes postpartum plea… [all, my, who, have, your] [(homes, NOUN), (experienced, VERB), (hair, NO… {‘NOUN’: 5, ‘VERB’: 2, ‘ADJ’: 1} [(NOUN, 5), (VERB, 2), (ADJ, 1)] [(VERB, 2)] [(experienced, VERB), (postpartum, VERB)] home experience hair change postpartum please … [(home, NOUN), (experience, VERB), (hair, NOUN… [(experience, VERB), (postpartum, VERB)] 1 I’m getting to the point that I’m worried that… i am getting to the point that i am worried th… [i, am, getting, to, the, point, that, i, am, … [(i, NOUN), (am, VERB), (getting, VERB), (to, … [getting, point, worried, limp, sad, hair, due… getting point worried limp sad hair due hormon… [i, am, to, the, that, i, am, that, my, is, to… [(getting, VERB), (point, NOUN), (worried, ADJ… {‘VERB’: 1, ‘NOUN’: 4, ‘ADJ’: 5, ‘ADV’: 1} [(ADJ, 5), (NOUN, 4), (VERB, 1), (ADV, 1)] [(VERB, 1)] [(getting, VERB)] get point worried limp sad hair due hormonal c… [(get, VERB), (point, NOUN), (worried, ADJ), (… [(get, VERB)] 2 My hair is still thick but so soft and very li… my hair is still thick but so soft and very li… [my, hair, is, still, thick, but, so, soft, an… [(my, PRON), (hair, NOUN), (is, VERB), (still,… [hair, still, thick, soft, limp, nearly, curly… hair still thick soft limp nearly curly handfu… [my, is, but, so, and, very, not, as, as, it, … [(hair, NOUN), (still, ADV), (thick, ADJ), (so… {‘NOUN’: 3, ‘ADV’: 5, ‘ADJ’: 2} [(ADV, 5), (NOUN, 3), (ADJ, 2)] [] [] hair still thick soft limp nearly curly handfu… [(hair, NOUN), (still, ADV), (thick, ADJ), (so… [] 3 Ten months postpartum, and I just want to know… ten months postpartum and i just want to know … [ten, months, postpartum, and, i, just, want, … [(ten, ADJ), (months, NOUN), (postpartum, NOUN… [ten, months, postpartum, want, know, alone] ten months postpartum want know alone [and, i, just, to, that, i, am, not] [(ten, ADJ), (months, NOUN), (postpartum, NOUN… {‘ADJ’: 1, ‘NOUN’: 2, ‘VERB’: 2, ‘ADV’: 1} [(NOUN, 2), (VERB, 2), (ADJ, 1), (ADV, 1)] [(VERB, 2)] [(want, VERB), (know, VERB)] ten month postpartum want know alone [(ten, ADJ), (month, NOUN), (postpartum, NOUN)… [(want, VERB), (know, VERB)] 4 Hi fellow fit pregnant friends! hi fellow fit pregnant friends [hi, fellow, fit, pregnant, friends] [(hi, ADV), (fellow, ADJ), (fit, NOUN), (pregn… [hi, fellow, fit, pregnant, friends] hi fellow fit pregnant friends [] [(hi, ADV), (fellow, ADJ), (fit, NOUN), (pregn… {‘ADV’: 1, ‘ADJ’: 2, ‘NOUN’: 2} [(ADJ, 2), (NOUN, 2), (ADV, 1)] [] [] hi fellow fit pregnant friend [(hi, ADV), (fellow, ADJ), (fit, NOUN), (pregn… [] … … … … … … … … … … … … … … … …\n\nREFERENCES\n\nhttps://towardsdatascience.com/cleaning-preprocessing-text-data-by-building-nlp-pipeline-853148add68a', metadata={'source': 'https://arslanselen.com/213-2/', 'title': 'Selen Arslan', 'description': 'DATA PREPROCESSING FOR BERTOPIC Step-by-Step Guide to Cleaning and Preprocessing Data for BERT-based Topic Modeling Data cleansing contains pre-steps before analysing process of the projects. These steps and order of steps are as important as the analysis process. In some cases, we can say that it is even more important. Because a dataset that has…', 'language': 'en-US'}),
Document(page_content='28 January 2023\n\nUMAP, HDBSCAN, c-TF-IDF AND BERTopic\n\nExploring the Intersection of BERTopic and Advanced Clustering and Dimensionality Reduction Techniques: A Comprehensive Guide to UMAP, HDBSCAN, and ClassTfidfTransformer\n\nUMAP\n\nUMAP (Uniform Manifold Approximation and Projection) is a non-linear dimensionality reduction technique that is particularly well-suited for preserving the global structure of high-dimensional data. UMAP is based on the idea that the data lies on a low-dimensional manifold that is embedded in a high-dimensional space. By modeling the manifold structure of the data, UMAP is able to capture the underlying geometry of the data, making it useful for visualizing and understanding complex datasets.\n\nUMAP is a relatively new technique compared to other dimensionality reduction techniques such as PCA (Principal Component Analysis) and t-SNE (t-Distributed Stochastic Neighbor Embedding). The main advantage of UMAP over these techniques is its ability to preserve both local and global structure. UMAP uses a technique called “manifold learning” which is able to preserve the nearest neighbor relationships in the high-dimensional space, unlike linear techniques like PCA which can only preserve global structure. This allows UMAP to effectively capture the underlying structure of the data and make it more interpretable.\n\nAnother advantage of UMAP is its ability to handle large datasets. UMAP has been shown to be much faster than t-SNE, while still producing high-quality visualizations of the data. Additionally, UMAP can be used in combination with other techniques such as feature scaling, normalization, and data imputation to further improve the results.\n\nIn relation to BERTopic, UMAP can be used to reduce the dimensionality of the feature vectors generated by the model. By projecting the high-dimensional feature vectors onto a low-dimensional space, UMAP can make it easier to visualize and understand the relationships between the different topics learned by BERTopic. This can be useful for identifying patterns and structure in the data that would otherwise be difficult to discern.\n\nDEFINING UMAP MODEL\n\nDefining a UMAP model with parameters involves instantiating the UMAP class and setting the desired parameters. The most important parameters to set are the number of dimensions to reduce the data to (n_components), the minimum distance between points in the low-dimensional space (min_dist), and the number of nearest neighbors to use when constructing the UMAP graph (n_neighbors).\n\nHere’s an example of how to define a UMAP model with 2 dimensions, a minimum distance of 0.1, and 15 nearest neighbors:\n\nAdditionally, you can also set other parameters such as the metric used to calculate distances between points (metric), the method used to construct the UMAP graph (spread), and the angular size of the neighborhood (angular_rp_forest).\n\nHere’s an example of how to set a UMAP model with additional parameters:\n\nIt is important to note that the optimal values for these parameters will depend on the specific dataset you’re working with and the goals of your analysis. It is recommended to use grid search or other optimization techniques to find the best set of parameters for your data.\n\nHDBSCAN\n\nHDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm that is particularly well-suited for discovering clusters of varying densities and shapes. HDBSCAN is based on the idea that clusters are dense regions of data separated by areas of lower density. By constructing a tree-based representation of the data, HDBSCAN is able to identify clusters of high-density regions.\n\nOne of the key advantages of HDBSCAN is its ability to discover clusters of varying densities and shapes. Unlike other clustering algorithms, such as k-means, HDBSCAN does not rely on a predefined number of clusters or a predefined shape for the clusters. This allows HDBSCAN to effectively capture the underlying structure of the data, making it useful for identifying and understanding subtopics within a given topic. Additionally, HDBSCAN has the ability to identify noise in the data and exclude it from the clustering process, which can lead to more accurate and interpretable results.\n\nHDBSCAN also has the ability to handle datasets with missing or noisy data. It can work with sparse data and high-dimensional data and it can also handle data with different density regions. HDBSCAN can also be used in combination with other techniques such as feature scaling, normalization, and data imputation to further improve the results.\n\nWhen combined with UMAP, UMAP is used to reduce the dimensionality of the data and make it more manageable for HDBSCAN to cluster. By reducing the dimensionality of the data with UMAP, it becomes easier for HDBSCAN to identify clusters and patterns, even in high-dimensional data. Furthermore, HDBSCAN can be applied to the low-dimensional data generated by UMAP to identify clusters that were not evident in the original high-dimensional data.\n\nIn summary, UMAP is used to reveal the global and local structure of the data and making it easy to identify patterns and clusters, while HDBSCAN is used to identify clusters of any shape, including clusters with variable density, and identify noise points. Combining both techniques allow to achieve better results in clustering and visualizing high-dimensional data.\n\nIn relation to BERTopic, HDBSCAN can be used to cluster the feature vectors generated by the model. By grouping similar feature vectors together, HDBSCAN can be useful for identifying and understanding the different subtopics within a given topic learned by BERTopic. This can be useful for identifying patterns and structure in the data that would otherwise be difficult to discern. Additionally, HDBSCAN can be used to identify outliers in the data, which can be useful for identifying and excluding data that may not be relevant to the topic being studied.\n\nDEFINING HDBSCAN MODEL\n\nDefining a HDBSCAN model with parameters involves instantiating the HDBSCAN class and setting the desired parameters. The most important parameters to set are the minimum number of points in a cluster (min_cluster_size) and the minimum similarity between points for them to be considered part of the same cluster (min_samples).\n\nHere’s an example of how to define a HDBSCAN model with a minimum cluster size of 15 and a minimum similarity of 5:\n\nAdditionally, you can also set other parameters such as the metric used to calculate distances between points (metric), the method used to construct the HDBSCAN tree (algorithm), and the minimum probability for a point to be considered part of a cluster (min_probability).\n\nHere’s an example of how to set a HDBSCAN model with additional parameters:\n\nIt is important to note that the optimal values for these parameters will depend on the specific dataset you’re working with and the goals of your analysis. It is recommended to use grid search or other optimization techniques to find the best set of parameters for your data.\n\nc-TF-IDF\n\nClassTfidfTransformer is a transformer class that can be used to convert a collection of raw documents to a matrix of TF-IDF features. TF-IDF (term frequency-inverse document frequency) is a statistical measure that is commonly used in text analysis and natural language processing to evaluate the importance of a word in a document. The TF-IDF score of a word is calculated by multiplying its term frequency (TF) by its inverse document frequency (IDF).\n\nThe main advantage of using TF-IDF as a feature representation for text data is that it is able to capture the importance of a word in relation to the entire corpus of documents. By giving more weight to words that are less common in the corpus and less weight to words that are more common, TF-IDF is able to identify words that are more relevant to the topic being studied. This can be useful for identifying patterns and structure in the data that would otherwise be difficult to discern.\n\nAnother advantage of ClassTfidfTransformer is its ability to handle sparse data. Since it only assigns non-zero values to words that are present in a document, it can effectively handle datasets with a large number of features. Additionally, ClassTfidfTransformer can be used in combination with other techniques such as feature scaling, normalization, and data imputation to further improve the results.\n\nIn relation to BERTopic, ClassTfidfTransformer can be used to convert the raw text data used to train the model into a matrix of TF-IDF features. This can be useful for highlighting the most important words in the text data, which can be used to identify patterns and structure in the data that would otherwise be difficult to discern. Additionally, the use of TF-IDF as a feature representation can help to improve the performance of BERTopic by providing the model with more relevant and informative features for learning the underlying topics in the data.\n\nRelation of UMAP, HDBSCAN and ClassTfidfTransformer with BERTopic\n\nUMAP, HDBSCAN and ClassTfidfTransformer are all techniques that can be used to improve the interpretability and performance of BERTopic. By reducing the dimensionality of the feature vectors generated by the model, UMAP can make it easier to visualize and understand the relationships between the different topics learned by BERTopic.\n\nHDBSCAN, on the other hand, can be used to cluster the feature vectors and identify subtopics within a given topic. ClassTfidfTransformer can be used to convert the raw text data used to train the model into a matrix of TF-IDF features, which can improve the performance of BERTopic by providing the model with more relevant and informative features for learning the underlying topics in the data.\n\nCombined, these techniques can provide a more complete understanding of the data being analyzed and can help to identify patterns and structure that would otherwise be difficult to discern. By using UMAP, HDBSCAN and ClassTfidfTransformer in combination with BERTopic, researchers and practitioners can gain a more comprehensive understanding of the topics and subtopics present in their data, which can be useful for a wide range of applications such as text classification, sentiment analysis and topic modeling.', metadata={'source': 'https://arslanselen.com/267-2/', 'title': 'Selen Arslan', 'description': 'UMAP, HDBSCAN, c-TF-IDF AND BERTopic Exploring the Intersection of BERTopic and Advanced Clustering and Dimensionality Reduction Techniques: A Comprehensive Guide to UMAP, HDBSCAN, and ClassTfidfTransformer UMAP UMAP (Uniform Manifold Approximation and Projection) is a non-linear dimensionality reduction technique that is particularly well-suited for preserving the global structure of high-dimensional data. UMAP is based on the…', 'language': 'en-US'}),
Document(page_content='14 September 2023\n\nWORD EMBEDDINGS\n\nINTRO AND WORD VECTORS\n\nMeaning of a word, WordNet, Words to Discrete Numbers, Distributional Semantics, Word Embeddings\n\nMEANING OF A WORD\n\nThe meaning of a word is the idea that is represented by a word, phrase etc. What does it mean in the sense of NLP? It can be described by a simple relation:\n\nSignifier (symbol) ↔ Signified (idea or thing)\n\nA ‘Signifier’ is a word and a ‘Signified’ is the thing that word signifies. This relation is called as denotational semantics.\n\nWORDNET\n\nWordNet is a large lexical database of English and a dictionary for synonym sets and hypernyms. Although, it is a large dataset the words listed as similar may not signed as similar in different contexts. It is impossible to keep up-to-date; new words’ meaning may be missed.\n\nWORDS TO DISCRETE NUMBERS\n\nIn NLP, the words are represented as a combination of discrete numbers.\n\nOne-Hot Vectors: Representing the words by a vector contains one 1 and 0’s.\n\nA simple example of two similar words:\n\nMotel=[000000000010000]\n\nHotel=[000000010000000]\n\nThese two words are similar and represented as vectors. Let’s think about a huge vocabulary. The size of the dimension will be a problem when we are thinking about thousands of words and vector versions of them.\n\nLet’s say we want to search for an X motel on a search engine. If there is an X hotel, we want to see it on the page too. But, there is no mathematical relation between the vectors of these two words: X motel and X hotel. How will the engine know these two words have a relation and they should be presented on the same page for the user? Grouping the words based on WordNet may be a solution but it is not reliable enough because of the problems we mentioned before. Instead of WordNet, modern deep learning techniques can be used such as distributional semantics.\n\nDISTRIBUTIONAL SEMANTICS AND WORD EMBEDDINGS\n\nA word’s meaning is given by the words that frequently appear close-by. When a word w appears in a text, its context is the set of words that appear nearby (within a fixed-size window). The main idea is using the many contexts of w to build up a representation of w. This representational vector called as word vector or word embeddings or (neural) word representations.\n\nLet’s make it more clear with an example of three sentences:\n\n…..problems turning into banking crises…..\n\n…..Europe needs unified banking regulation…..\n\n…..banking system….\n\nThe context words will represent the word banking and we will build a dense vector for each word, chosen so that it is similar to vectors of words that appear in similar contexts.\n\nThis vector may not makes everything clear for you but when we visualize it as a network we will see the similar words closer to each other.\n\nHere is a simple example created by using https://projector.tensorflow.org/\n\nThe most related words with the word ‘arena’ are shown in the figure.\n\nWhat about the word embedding models? I found a good resource that explains lots of different word embedding models. You can check the article:\n\nTwo minutes NLP — 11 word embeddings models you should know\n\nWORD2VEC\n\nWord2Vec is a framework for learning word vectors. I don’t want to explain the theoretical parts deeply. You can learn it from the reference video. The concept will be explained in the next blog post.\n\nHere is a simple examples for word vectors : https://github.com/selenarslann/DATA_ANALYSIS/blob/main/word_embedding_gensim.ipynb\n\nReferences:\n\nhttps://www.youtube.com/watch?v=rmVRLeJRkl4&list=PLoROMvodv4rOSH4v6133s9LFPRHjEmbmJ&index=2', metadata={'source': 'https://arslanselen.com/cs224n-c1/', 'title': 'Selen Arslan', 'description': 'WORD EMBEDDINGS INTRO AND WORD VECTORS Meaning of a word, WordNet, Words to Discrete Numbers, Distributional Semantics, Word Embeddings MEANING OF A WORD The meaning of a word is the idea that is represented by a word, phrase etc. What does it mean in the sense of NLP? It can be described by a simple…', 'language': 'en-US'}),
Document(page_content='18 September 2023\n\nWORD2VEC : CBOW AND SKIPGRAM\n\nWord2Vec, bag of words\n\nWORD2VEC\n\nWord embeddings can be generated using various methods like neural networks, co-occurrence matrix, probabilistic models, etc.\xa0Word2Vec\xa0consists of models for generating word embedding. These models are shallow two-layer neural networks having one input layer, one hidden layer, and one output layer.\n\nBag of words model : The position and word order is not important. The model makes the same predictions at each position. The bag of words technique is a method used in Natural Language Processing for text modeling. A bag of words is a representation of text that describes the occurrence of words within a document.\xa0This method disregards the grammatical intricacies and word order, treating the text as a collection of words without any specific sequence. It’s referred to as a “bag” of words because it doesn’t consider the arrangement or structure of words in the document. Instead, it focuses solely on determining whether known words appear in the document, without caring about their positions within the text.\n\nThe main idea: find the unique words and create a table with 0 and 1’s which represents the occurrence of a word within the text.\n\nThere are two problems with this approach:\n\nThe vector is very sparse (i.e. most values are 0)\n\nWe lose information relating the context (i.e. order of the words in the sentence)\n\nWe want a model that gives a reasonably high probability estimate to all words that occur in the context.\n\nHow do we learn good vectors?\n\nwe have a cost function J(θ) and we want to minimize it. Gradient Descent is an algorithm to minimize the cost function by changing θ.\n\nThe idea is basic: From current value of θ, calculate the gradient of J(θ) and take small step in the direction of negative gradient. The important part is selecting the step size. The small step size takes too much time. With big step size we may miss the minimum point.\n\nThe algorithm of the gradient descent:\n\nwhile True:\n\nwhile True:\n\ntheta_grad=evaluate_gradient(J,corpus,theta)\n\ntheta=theta-alpha*theta_grad\n\nalpha is the step size.\n\nThe problem is J(θ) is a function of all windows in the corpus. So, single update on gradient takes too much time. Solution is Stochastic Gradient Descent (SGD) which is using the sample windows and updating the each window (small batch).\n\nThe algorithm of the stochastic gradient descent:\n\nwhile True:\n\nwindow=sample_window(corpus)\n\nwindow=sample_window(corpus)\n\ntheta_grad=evaluate_gradient(J,window,theta)\n\ntheta=theta-alpha*theta_grad\n\nalpha is the step size.\n\nIn Word2Vec basically we have 2 vectors: the center vector and outside vector and we average both of them at the end. What about having only one vector for a word? There are two variants of Word2Vec : skip-gram and CBOW.\n\nSkip-gram predicts context words from a target word.\n\nCBOW predicts a target word from a set of context words.\n\nSKIPGRAM\n\nThe following schema shows how the algorithm works.\n\nBoth the target and context word pairs are passed to individual embedding layers from which we get dense word embeddings for each of these two words.\n\nWe then use a ‘merge layer’ to compute the dot product of these two embeddings and get the dot product value.\n\nThis dot product value is then sent to a dense sigmoid layer that outputs either 0 or 1.\n\nThe output is compared with the actual label and the loss is computed followed by backpropagation with each epoch to update the embedding layer in the process.\n\nCBOW\n\nThe CBOW architecture comprises a deep learning classification model in which we take in context words as input, X, and try to predict our target word, Y.\n\nFor example, if we consider the sentence – “Word2Vec has a deep learning model working in the backend.”, there can be pairs of context words and target (center) words. If we consider a context window size of 2, we will have pairs like\xa0([deep, model], learning), ([model, in], working), ([a, learning), deep)\xa0etc. The deep learning model would try to predict these target words based on the context words.\n\nThe following steps describe how the model works:\n\nThe context words are first passed as an input to an embedding layer (initialized with some random weights) as shown in the Figure below.\n\nThe word embeddings are then passed to a lambda layer where we average out the word embeddings.\n\nWe then pass these embeddings to a dense SoftMax layer that predicts our target word. We match this with our target word and compute the loss and then we perform backpropagation with each epoch to update the embedding layer in the process.\n\nWe can extract out the embeddings of the needed words from our embedding layer, once the training is completed.\n\nReferences:\n\nhttps://www.analyticsvidhya.com/blog/2021/07/word2vec-for-word-embeddings-a-beginners-guide/', metadata={'source': 'https://arslanselen.com/436-2/', 'title': 'Selen Arslan', 'description': 'WORD2VEC : CBOW AND SKIPGRAM Word2Vec, bag of words WORD2VEC Word embeddings can be generated using various methods like neural networks, co-occurrence matrix, probabilistic models, etc.\xa0Word2Vec\xa0consists of models for generating word embedding. These models are shallow two-layer neural networks having one input layer, one hidden layer, and one output layer. Bag of words model :…', 'language': 'en-US'}),
Document(page_content='13 November 2023\n\nCONTEXTUALIZED EMBEDDING: BERT\n\nINTRODUCTION TO BERT\n\nCONTEXTUALIZED EMBEDDING\n\nLet’s start with what we should understand by calling ‘contextualized meaning’. I will explain it an with elementary example. The following two sentences have the same word ‘apple’.\n\nI like apples.\n\nI like Apple Macbooks.\n\nThe word embedding method represents the words by vectors. The word is the same for both sentences but the meanings are different. Having the same vectors for both words means we are skipping the importance of the meaning of the word. The contextualized language models allow us to represent the same words which have different meanings in different vectors.\n\nBidirectional Encoder Representations from Transformer (BERT)… Okay, but what does it mean?\n\nBIDIRECTIONAL\n\nAs opposed to directional models, which read the text input sequentially (left-to-right or right-to-left), the Transformer encoder reads the entire sequence of words at once. Therefore it is considered bidirectional, this characteristic allows the model to learn the context of a word based on all of its surroundings (left and right of the word).\n\nTRANSFORMER AND ENCODER\n\nTransformers use an attention mechanism to observe relationships between words. Machine Learning models need to learn how to pay attention only to the things that matter and not waste computational resources processing irrelevant information. Transformers create differential weights signaling which words in a sentence are the most critical to further process.\n\nTransformer includes two separate mechanisms — an encoder that reads the text input and a decoder that produces a prediction for the task. Since BERT’s goal is to generate a language model, only the encoder mechanism is necessary.\n\nMASKED LANGUAGE MODELING AND NEXT SENTENCE PREDICTION\n\nMany models predict the next word in a sequence which is a a directional approach. BERT uses two training strategies:\n\n1-Masked Language Modeling (MLM)\n\nMLM enables/enforces bidirectional learning from text by masking (hiding) a word in a sentence and forcing BERT to bidirectionally use the words on either side of the covered word to predict the masked word. Actually, we do the same thing as a human. Imagine, you are listening to a course and you miss a word said by your professor. How would you predict the missed word?\n\n‘blablablabla …… blablablabla.’\n\nThe correct answer is by checking the both sides of the missed word. This is the same thing that BERT does.\n\nA random 15% of tokenized words are hidden during training and BERT’s job is to correctly predict the hidden words.\n\n2-Next Sentence Prediction\n\nNSP (Next Sentence Prediction) is used to help BERT learn about relationships between sentences by predicting if a given sentence follows the previous sentence or not.\n\nIn training, 50% correct sentence pairs are mixed in with 50% random sentence pairs to help BERT increase next sentence prediction accuracy.\n\nReferences:\n\nhttps://huggingface.co/blog/bert-101#2-how-does-bert-work\n\nhttps://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270', metadata={'source': 'https://arslanselen.com/523-2/', 'title': 'Selen Arslan', 'description': "CONTEXTUALIZED EMBEDDING: BERT INTRODUCTION TO BERT CONTEXTUALIZED EMBEDDING Let's start with what we should understand by calling 'contextualized meaning'. I will explain it an with elementary example. The following two sentences have the same word 'apple'. I like apples. I like Apple Macbooks. The word embedding method represents the words by vectors. The word is…", 'language': 'en-US'}),
Document(page_content='16 November 2023\n\nTRANSFORMERS\n\nINTRODUCTION TO TRANSFORMERS\n\nTRANSFORMERS\n\nThe transformers work with 3 mechanisms: encoder-decoder, attention mechanism and transfer learning.\n\nENCODER-DECODER FRAMEWORK\n\nBefore transformers, NLP mainly used structures like LSTMs, which were considered top-notch. These structures have a loop that allows information to move from one step to the next. This makes them great for handling data in a sequence, like text.\n\nIn an RNN, you give it input (like a word or character), it processes it, and gives you a hidden state vector. At the same time, it sends some information back to itself through a loop, which it then uses in the next step. The RNN shares its state information at each step with the next operation in the sequence. This helps the RNN remember information from previous steps and use it to predict the output. RNNs were crucial in the advancement of machine translation systems, particularly in the task of converting a sequence of words from one language to another. For such tasks, an encoder-decoder or sequence-to-sequence architecture is commonly employed. This architecture is well suited for scenarios where both the input and output consist of sequences of varying lengths.\n\nThe encoder’s role is to convert information from the input sequence into a numerical representation commonly referred to as the last hidden state. This state is then forwarded to the decoder, which produces the output sequence.\n\nTypically, the encoder and decoder components can be any neural network architecture capable of modeling sequences. This is demonstrated with a pair of RNNs in the figure below. The input words are processed sequentially through the encoder, and the output words are generated one at a time, from top to bottom. A drawback of this architecture is the creation of an information bottleneck by the final hidden state of the encoder (C in the figure). This state must encapsulate the meaning of the entire input sequence since it’s the only information available to the decoder during output generation. This becomes particularly challenging for lengthy sequences, where information from the beginning might get lost when compressing everything into a single, fixed representation. Fortunately, there’s a solution to this bottleneck by enabling the decoder to access all of the encoder’s hidden states. This mechanism is known as attention, and it serves as a fundamental component in numerous contemporary neural network architectures.\n\nATTENTION MECHANISM\n\nThe central concept behind attention is that instead of generating a single hidden state for the input sequence, the encoder produces a hidden state at each step that the decoder can access. However, using all these states simultaneously would overwhelm the decoder, so a mechanism is necessary to prioritize which states to utilize. This is where attention becomes crucial: it enables the decoder to assign varying amounts of weight, or ‘attention,’ to each of the encoder states at each decoding timestep.\n\nBy concentrating on which input tokens are most relevant at each timestep, these models that employ attention are capable of acquiring intricate relationships between the words in a translated output and those in the source sentence\n\nSo, I explained why we use the attention mechanism basically, we will focus on types of attention in future blog posts.\n\nIn numerous real-world NLP applications, obtaining substantial labeled text data for training models is often impractical. This is where we need the transfer learning.\n\nTRANSFER LEARNING\n\nIn terms of architecture, transfer learning involves dividing the model into a body and a head, where the head represents a task-specific network. During training, the weights of the body learn broad features from the source domain, and these weights are utilized to initialize a new model for the target task. Compared to traditional supervised learning, this approach typically yields high-quality models that can be trained much more efficiently across various downstream tasks, requiring significantly less labeled data.\n\nOkay, let’s take a deep breath and ask the question: ‘What will we do with these 3 mechanisms?’\n\nIn 2018, two transformers were introduced, which integrated self-attention with transfer learning: GPT and BERT\n\nGPT:\n\nUses only the decoder part of the Transformer architecture, following the same language modeling approach as ULMFiT. GPT was pretrained on the BookCorpus, a collection of 7,000 unpublished books spanning various genres, including Adventure, Fantasy, and Romance.\n\nBERT:\n\nUtilizes the encoder part of the Transformer architecture and employs a unique form of language modeling known as masked language modeling. The objective of masked language modeling is to predict randomly masked words in a text. For instance, given a sentence like ‘I looked at my [MASK] and saw that [MASK] was late,’ the model must predict the most likely candidates for the masked words denoted by [MASK]. BERT was pretrained on the BookCorpus and English Wikipedia.” (Check for more information.)\n\nIn summary, transformers have redefined natural language processing through three key components: the encoder-decoder framework, attention mechanism, and transfer learning. They address challenges like information bottlenecks and limited labeled data by enabling efficient training on diverse tasks.\n\nThe introduction of transformers like GPT and BERT in 2018 marked a pivotal moment, combining self-attention with transfer learning. GPT, focused on the decoder, and BERT, leveraging the encoder, demonstrate the adaptability of transformers in language modeling, pretrained on extensive datasets.\n\nThe transformative impact of transformers continues to revolutionize language processing, promising innovative solutions for various linguistic challenges. Future blog posts will explore specific attention mechanisms in more detail.', metadata={'source': 'https://arslanselen.com/551-2/', 'title': 'Selen Arslan', 'description': 'TRANSFORMERS INTRODUCTION TO TRANSFORMERS TRANSFORMERS The transformers work with 3 mechanisms: encoder-decoder, attention mechanism and transfer learning. ENCODER-DECODER FRAMEWORK Before transformers, NLP mainly used structures like LSTMs, which were considered top-notch. These structures have a loop that allows information to move from one step to the next. This makes them great for handling data in…', 'language': 'en-US'})]
In [12]:
from langchain.text_splitter import CharacterTextSplitter
In [13]:
text_splitter = CharacterTextSplitter(separator='\n',
chunk_size=3065, # Adjust to a larger value
chunk_overlap=200) # Adjust to a larger value
In [14]:
text_chunks=text_splitter.split_documents(data)
In [15]:
len(text_chunks)
Out[15]:
24
In [16]:
text_chunks[0]
Out[16]:
Document(page_content='My data science diary\nSelen Arslan\nWelcome to my data science diary, where you will find documentation of my data science journey. On this website, you will find a showcase of my projects and experiences in the field of data analysis and natural language processing. As a graduate in the University of Padua in the ICT for Internet and Multimedia (Cybersystems) department and Electrical and Electronics Engineering from the University of Turkish Aeronautical Association, I have developed a strong passion in these areas through my data science journey. I am excited to share with you my projects, research, and professional experiences that have shaped my understanding and approach to data science. I invite you to take a look around and discover my approach to data science.\nMail\nLinkedIn\nTWITTER API V2 SEARCH WITH TWEEPY\nSCRAPPING REDDIT DATA WITH PMAW\nDATA CLEANSING FOR BERTOPIC\nUMAP, HDBSCAN, c-TF-IDF AND BERTopic\nINTRODUCTION TO WORD EMBEDDINGS\nWORD2VEC : CBOW AND SKIPGRAM\nINTRODUCTION TO BERT\nTRANSFORMERS', metadata={'source': 'https://arslanselen.com/', 'title': 'Selen Arslan', 'description': 'My data science diary Welcome to my data science diary, where you will find documentation of my data science journey. On this website, you will find a showcase of my projects and experiences in the field of data analysis and natural language processing. As a graduate in the University of Padua in the ICT for…', 'language': 'en-US'})
openAI and HuggingFace Embeddings¶
In [17]:
!pip -q install openai
In [18]:
!pip install --upgrade openai
Requirement already satisfied: openai in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (1.3.6) Requirement already satisfied: distro<2,>=1.7.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (1.8.0) Requirement already satisfied: sniffio in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (1.3.0) Requirement already satisfied: anyio<4,>=3.5.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (3.5.0) Requirement already satisfied: httpx<1,>=0.23.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (0.25.2) Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (1.10.13) Requirement already satisfied: tqdm>4 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (4.66.1) Requirement already satisfied: typing-extensions<5,>=4.5 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from openai) (4.8.0) Requirement already satisfied: idna>=2.8 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from anyio<4,>=3.5.0->openai) (3.3) Requirement already satisfied: httpcore==1.* in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from httpx<1,>=0.23.0->openai) (1.0.2) Requirement already satisfied: certifi in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from httpx<1,>=0.23.0->openai) (2022.9.24) Requirement already satisfied: h11<0.15,>=0.13 in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)
In [19]:
!pip show openai
Name: openai Version: 1.3.6 Summary: The official Python library for the openai API Home-page: Author: Author-email: OpenAI <support@openai.com> License: Location: /Users/selen/opt/miniconda3/lib/python3.9/site-packages Requires: pydantic, distro, typing-extensions, httpx, sniffio, tqdm, anyio Required-by:
In [20]:
api_key='your api key'
In [ ]:
#from openai import OpenAI
#client = OpenAI(api_key=api_key) # Pass your API key to the OpenAI client
In [25]:
from langchain.embeddings import HuggingFaceEmbeddings
In [26]:
embeddings=HuggingFaceEmbeddings()
In [27]:
query_result = embeddings.embed_query("Hello world")
len(query_result)
Out[27]:
768
Convert the Text Chunks into Embeddings and Create a Knowledge Base¶
In [28]:
!pip install faiss-cpu
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) Requirement already satisfied: faiss-cpu in /Users/selen/opt/miniconda3/lib/python3.9/site-packages (1.7.4)
In [29]:
from langchain.vectorstores import FAISS
In [30]:
vectorstore=FAISS.from_documents(text_chunks, embeddings)
In [31]:
vectorstore
Out[31]:
<langchain.vectorstores.faiss.FAISS at 0x2bc7796a0>
In [32]:
embeddings
Out[32]:
HuggingFaceEmbeddings(client=SentenceTransformer(
(0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
(2): Normalize()
), model_name='sentence-transformers/all-mpnet-base-v2', cache_folder=None, model_kwargs={}, encode_kwargs={}, multi_process=False)
Create a Large Language Model (LLM) Wrapper¶
In [22]:
from langchain.chat_models import ChatOpenAI
In [23]:
llm = ChatOpenAI(openai_api_key=api_key)
In [24]:
llm
Out[24]:
ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x1767f64c0>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x1769ff640>, openai_api_key='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', openai_proxy='')
In [43]:
from langchain.chains import RetrievalQAWithSourcesChain
In [44]:
# Create RetrievalQAWithSourcesChain
chain = RetrievalQAWithSourcesChain.from_chain_type(llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever())
In [76]:
# Generate a response to a query using the chain
d_response = chain({"question": "What is distributional semantics?"})
# Display the answer and sources
print("Response:")
print(d_response["answer"])
#print("Sources:")
#for source in d_response["sources"].split(", "):
# print("- " + source)
In [77]:
d_response["answer"]
Out[77]:
'Distributional semantics is a concept in natural language processing (NLP) that refers to the meaning of a word being derived from the words that frequently appear close to it in a given context. This is achieved by building word vectors or word embeddings, which are dense representations of words based on their contextual usage. These word vectors are created using techniques such as the distributional semantics approach, where the context words surrounding a target word are used to build its representation. Word2Vec is one such model that generates word embeddings using shallow two-layer neural networks. It has two variants: skip-gram, which predicts context words from a target word, and CBOW (continuous bag-of-words), which predicts a target word from a set of context words. These word embeddings help capture the semantic relationships between words and can be used in various NLP tasks. BERT (Bidirectional Encoder Representations from Transformers) is another contextualized word embedding model that uses a transformer-based architecture and techniques like masked language modeling and next sentence prediction to learn contextualized representations of words. Overall, distributional semantics and word embeddings play a crucial role in capturing the meaning of words in NLP tasks.\n'
In [ ]:
