WORD EMBEDDINGS
INTRO AND WORD VECTORS
Meaning of a word, WordNet, Words to Discrete Numbers, Distributional Semantics, Word Embeddings
MEANING OF A WORD
The meaning of a word is the idea that is represented by a word, phrase etc. What does it mean in the sense of NLP? It can be described by a simple relation:
Signifier (symbol) ↔ Signified (idea or thing)
A ‘Signifier’ is a word and a ‘Signified’ is the thing that word signifies. This relation is called as denotational semantics.
WORDNET
WordNet is a large lexical database of English and a dictionary for synonym sets and hypernyms. Although, it is a large dataset the words listed as similar may not signed as similar in different contexts. It is impossible to keep up-to-date; new words’ meaning may be missed.
WORDS TO DISCRETE NUMBERS
In NLP, the words are represented as a combination of discrete numbers.
One-Hot Vectors: Representing the words by a vector contains one 1 and 0’s.
A simple example of two similar words:
Motel=[000000000010000]
Hotel=[000000010000000]
These two words are similar and represented as vectors. Let’s think about a huge vocabulary. The size of the dimension will be a problem when we are thinking about thousands of words and vector versions of them.
Let’s say we want to search for an X motel on a search engine. If there is an X hotel, we want to see it on the page too. But, there is no mathematical relation between the vectors of these two words: X motel and X hotel. How will the engine know these two words have a relation and they should be presented on the same page for the user? Grouping the words based on WordNet may be a solution but it is not reliable enough because of the problems we mentioned before. Instead of WordNet, modern deep learning techniques can be used such as distributional semantics.
DISTRIBUTIONAL SEMANTICS AND WORD EMBEDDINGS
A word’s meaning is given by the words that frequently appear close-by. When a word w appears in a text, its context is the set of words that appear nearby (within a fixed-size window). The main idea is using the many contexts of w to build up a representation of w. This representational vector called as word vector or word embeddings or (neural) word representations.
Let’s make it more clear with an example of three sentences:
…..problems turning into banking crises…..
…..Europe needs unified banking regulation…..
…..banking system….
The context words will represent the word banking and we will build a dense vector for each word, chosen so that it is similar to vectors of words that appear in similar contexts.

This vector may not makes everything clear for you but when we visualize it as a network we will see the similar words closer to each other.
Here is a simple example created by using https://projector.tensorflow.org/

The most related words with the word ‘arena’ are shown in the figure.
What about the word embedding models? I found a good resource that explains lots of different word embedding models. You can check the article:
Two minutes NLP — 11 word embeddings models you should know
WORD2VEC
Word2Vec is a framework for learning word vectors. I don’t want to explain the theoretical parts deeply. You can learn it from the reference video. The concept will be explained in the next blog post.
Here is a simple examples for word vectors : https://github.com/selenarslann/DATA_ANALYSIS/blob/main/word_embedding_gensim.ipynb
References:
