📘
📘
📘
📘
Notes
Natural Language Processing
Search…
⌃K
Links

Information Retrieval

Information Retrieval and Related Applications. TF/IDF, Cosine Similarity.

TF/IDF

Term Frequency (TF)

TF: number of times term t occurs in document (or alternative: number of terms divided by length of document)
​
TF(t,d)=Count(times of term t appearing in d)TF(t, d)=Count(times\ of\ term\ t\ appearing\ in\ d)
​

Inverse Document Frequency (IDF)

IDF: logarithm of number of documents (in corpus) divided by number of documents containing term t
​
IDF(t)=log⁡Count(documents in total)Count(documents containing term t)IDF(t)=\log{{Count(documents\ in\ total)}\over{Count(documents\ containing\ term\ t)}}
​

TF-IDF

​
TF_IDF(t,d)=TF(t,d)∗IDF(t)TF\_IDF(t, d)=TF(t, d) * IDF(t)
​

Cosine Similarity

Cosine of the Angle Between the Vectors. Range is [0, 1]. The higher the value, the more similar the vectors.
​
Cosine(v1,v2)=v1⋅v2v12⋅v22Cosine(v1, v2) = \frac{v_1 \cdot v_2}{\sqrt{{v_1}^2} \cdot \sqrt{{v_2}^2}}
​

Example

​
v1=[0,5,0,5,0]v1 = [0, 5, 0, 5, 0]
​
​
v2=[0,7,0,9,0]v2 = [0, 7, 0, 9, 0]
​
​
Cosine(v1,v2)=0∗0+5∗7+0∗0+5∗9+0∗002+52+02+52+02+02+72+02+92+02=0.992Cosine(v1, v2) = {{0*0+5*7+0*0+5*9+0*0}\over{\sqrt{0^2+5^2+0^2+5^2+0^2}+\sqrt{0^2+7^2+0^2+9^2+0^2}}} = 0.992
​