📘
📘
📘
📘
Notes
Natural Language Processing
Search
K
Links

Information Retrieval

Information Retrieval and Related Applications. TF/IDF, Cosine Similarity.

TF/IDF

Term Frequency (TF)

TF: number of times term t occurs in document (or alternative: number of terms divided by length of document)
TF(t,d)=Count(times of term t appearing in d)TF(t, d)=Count(times\ of\ term\ t\ appearing\ in\ d)

Inverse Document Frequency (IDF)

IDF: logarithm of number of documents (in corpus) divided by number of documents containing term t
IDF(t)=logCount(documents in total)Count(documents containing term t)IDF(t)=\log{{Count(documents\ in\ total)}\over{Count(documents\ containing\ term\ t)}}

TF-IDF

TF_IDF(t,d)=TF(t,d)IDF(t)TF\_IDF(t, d)=TF(t, d) * IDF(t)

Cosine Similarity

Cosine of the Angle Between the Vectors. Range is [0, 1]. The higher the value, the more similar the vectors.
Cosine(v1,v2)=v1v2v12v22Cosine(v1, v2) = \frac{v_1 \cdot v_2}{\sqrt{{v_1}^2} \cdot \sqrt{{v_2}^2}}

Example

v1=[0,5,0,5,0]v1 = [0, 5, 0, 5, 0]
v2=[0,7,0,9,0]v2 = [0, 7, 0, 9, 0]
Cosine(v1,v2)=00+57+00+59+0002+52+02+52+02+02+72+02+92+02=0.992Cosine(v1, v2) = {{0*0+5*7+0*0+5*9+0*0}\over{\sqrt{0^2+5^2+0^2+5^2+0^2}+\sqrt{0^2+7^2+0^2+9^2+0^2}}} = 0.992