Table of Contents
- Why NLP Matters for Data Engineers
- Why Text Is Hard for Computers
- Our Example — Classifying Support Tickets
- Step 1 — Cleaning and Normalizing Text
- Step 2 — Tokenization
- Step 3 — Stop Words
- Step 4 — Stemming vs Lemmatization
- Bag of Words — Turning Text into Numbers
- TF-IDF — Rewarding the Words That Matter
- N-grams — Capturing Short Phrases
- Text Classification — Baselines First
- The 100% Accuracy Trap — Near-Duplicate Leakage
- Where Bag-of-Words Breaks Down
- Word Embeddings — Meaning as Coordinates
- Contextual Embeddings and Transformers
- Text Similarity and Semantic Search
- Topic Modeling — Finding Themes Without Labels
- Other Core NLP Tasks
- Tokens and LLMs — What Data Engineers Should Know
- NLP at Scale with Spark
- Classic NLP vs Azure AI Language vs LLMs
- Common Mistakes
- Interview Questions
- Wrapping Up
Most of the world’s business data is not in neat columns. It is in support tickets, emails, chat logs, product reviews, contracts, call transcripts, and free-text comment fields. Natural Language Processing (NLP) is the set of techniques that turns that text into something a computer can count, compare, classify, and search.
This post builds NLP from the ground up with one realistic project: automatically routing customer support tickets to the right team. We clean text, tokenize it, compare stemming and lemmatization, turn words into numbers with bag-of-words and TF-IDF, train classifiers, uncover a leakage trap that made a model look perfect, see exactly where keyword-based methods fail, and then move to embeddings, semantic search, topic modeling, Spark, and LLM tokens. Every result in this post comes from actually running the code.
Analogy — A mailroom sorting letters. Imagine a company mailroom receiving 5,000 letters a day. A new clerk sorts them by spotting keywords: “invoice” goes to finance, “broken” goes to support. An experienced clerk does better — they notice phrases (“not working”), know that “parcel” and “package” mean the same thing, and understand that “bank” means something different in “river bank.” NLP is the journey from the new clerk’s keyword spotting to the experienced clerk’s understanding, and this post walks that path step by step.
Why NLP Matters for Data Engineers
Text data you are likely to handle:
Support tickets and chats Route, prioritize, measure topics over time
Product reviews Sentiment trends, complaints by feature
Emails and forms Extract fields, classify requests
Application and pipeline Group error messages, detect new failure types
logs
Documents and contracts Search, summarize, extract clauses
Survey free-text answers Themes that multiple-choice questions miss
What usually lands on your plate:
Ingesting and storing raw text safely (encoding, PII)
Cleaning and normalizing it consistently
Building features (TF-IDF vectors, embeddings) at scale
Running classification or embedding jobs over millions of rows
Feeding vector search and RAG pipelines
Writing results back as columns: category, sentiment, topic, languageWhy Text Is Hard for Computers
Computers need numbers. Text brings five problems:
1. Variable length "Help!" vs a 2,000-word complaint
2. Huge vocabulary Tens of thousands of distinct words, typos,
product codes, slang, emoji
3. Synonyms "package", "parcel", "delivery", "shipment"
4. Ambiguity "bank" (river or money), "charge" (fee or battery)
5. Word order matters "not good" vs "good, not bad"
Every technique in this post is a different trade-off for handling
these problems.Our Example — Classifying Support Tickets
We generated 2,240 realistic support tickets across four teams — billing, technical, account, and shipping — with the messiness real tickets have: greetings, sign-offs, dropped words, typos, order numbers, and some genuinely mixed tickets (“I was charged for an order that never arrived”).
Sample tickets:
"hey, my was declined when paying the invoice please fix asap" billing
"good morning, my invoice shows the wrogn amount for order 85965" billing
"cannot log in to download my invoice" account
"hello team, please delete my account all my personla data" account
Dataset: 2,240 tickets, 560 per category
Each category has 8 underlying ways customers phrase their problem,
which turns out to matter a great deal later.Step 1 — Cleaning and Normalizing Text
import re
def clean(text):
text = text.lower() # "Invoice" = "invoice"
text = re.sub(r"\b\d{5}\b", " ordernum ", text) # all order numbers -> one token
text = re.sub(r"[^a-z0-9\s-]", " ", text) # drop punctuation and symbols
return re.sub(r"\s+", " ", text).strip() # collapse whitespace
print(clean("URGENT: Order 85965 arrived DAMAGED!!!"))
# urgent ordernum arrived damagedCommon cleaning steps (choose deliberately, not all by default):
Lowercasing "Refund" and "refund" become the same word
Replacing IDs Order numbers, emails, URLs -> placeholder tokens
so 2,000 different order numbers do not become
2,000 useless "words"
Removing punctuation Usually helpful; keep it for sentiment ("!!!")
Unicode normalization "café" vs "cafe", smart quotes vs straight quotes
Masking PII Replace names, emails, phone numbers BEFORE
storing or sharing text (compliance)
Fixing encoding Mojibake like "don’t" means the text was decoded
with the wrong character set upstreamAnalogy — Washing vegetables before cooking. You rinse off dirt and remove stickers, but you do not peel away the parts that carry flavor. Cleaning text works the same way: remove what adds noise (random order numbers, formatting), keep what carries meaning. Over-cleaning — such as stripping “not” or “!” from reviews — throws away the flavor. Our Python regular expressions guide covers the patterns used here in depth.
Step 2 — Tokenization
Tokenization splits text into units — usually words — that the model counts or embeds.
Input: "the app crashes, error 500!"
Whitespace split: ["the", "app", "crashes,", "error", "500!"]
↑ punctuation stuck to words
Regex word tokenizer: ["the", "app", "crashes", "error", "500"]
Subword (LLMs): ["the", " app", " crash", "es", ",", " error", " 500", "!"]
↑ unknown or long words split into reusable piecesAnalogy — Cutting a sentence into LEGO bricks. Word tokens are standard bricks: easy to count, but a word you have never seen is a brick you do not have. Subword tokens are smaller bricks that can build any word — “unfollowable” becomes “un” + “follow” + “able” — which is why modern language models never meet a truly unknown word.
Step 3 — Stop Words
Stop words are very common words — “the”, “is”, “to”, “my” — that often carry little meaning on their own.
"please help, how do i change the email on my account"
After removing English stop words:
"please help change email account"
When removing them helps:
Topic modeling, keyword extraction, search indexes, word clouds
When removing them HURTS:
Sentiment: "not good" → "good" (meaning reversed!)
Short phrases: "how do i" signals a how-to question
Classification with n-grams: "my account", "log in" lose their structureIn our project, the strongest signals for account tickets included “my account” and “how do” — phrases made entirely of common words. The Spark version below removes stop words and scores lower, which is likely part of the reason. Treat stop-word removal as a choice to test, not a default.
Analogy — Filler in conversation. “Um”, “like”, and “you know” rarely change what someone means — until they do. “I like it” and “I, like, hate it” differ precisely in a word you might have filtered out.
Step 4 — Stemming vs Lemmatization
Both reduce words to a base form so “fails”, “failed”, and “failing” count as one concept.
from nltk.stem import PorterStemmer, WordNetLemmatizer
stemmer, lemmatizer = PorterStemmer(), WordNetLemmatizer()
for word, pos in [("running", "v"), ("studies", "n"), ("better", "a"),
("pipelines", "n"), ("university", "n"), ("universe", "n")]:
print(word, stemmer.stem(word), lemmatizer.lemmatize(word, pos))Real output:
Word Stemmer Lemmatizer
────────── ───────── ──────────
running run run
studies studi study
better better good ← lemmatizer knows "better" → "good"
pipelines pipelin pipeline
university univers university
universe univers universe ← stemmer merged two different words!
Stemming: Chops word endings by rules. Fast, crude, creates non-words,
and can merge unrelated words (university / universe).
Lemmatization: Looks words up in a dictionary with their part of speech.
Slower, accurate, returns real words.Analogy — A lumberjack versus a librarian. The stemmer is a lumberjack: it chops every word at a fixed point, fast, without checking what it cut. The lemmatizer is a librarian: it looks each word up and files it under its proper dictionary entry. For search indexes over huge volumes, the lumberjack’s speed is often fine. When meaning matters, ask the librarian.
Bag of Words — Turning Text into Numbers
Bag of words represents each document as counts of the words it contains, ignoring order.
Vocabulary: [refund, payment, failed, package, damaged, my]
"refund my payment" → [1, 1, 0, 0, 0, 1]
"payment failed again" → [0, 1, 1, 0, 0, 0]
"package arrived damaged" → [0, 0, 0, 1, 1, 0]
Each document becomes a row; each vocabulary word becomes a column.
Real vocabularies have thousands of columns, almost all zeros for any
one document -- stored efficiently as sparse matrices.Analogy — A shopping receipt. A receipt lists what you bought and how many, but not the order you picked items off the shelves. Bag of words is a receipt for a document: “two refunds, one payment” — useful for telling a grocery trip from a hardware-store trip, even without the sequence.
TF-IDF — Rewarding the Words That Matter
Raw counts overweight words that appear everywhere. TF-IDF (Term Frequency × Inverse Document Frequency) boosts words that are frequent in one document but rare across the collection.
TF = how often the word appears in THIS document (count / length)
IDF = ln(total documents / documents containing the word)
TF-IDF = TF × IDF
Worked example with three tickets:
doc 1: "refund my payment"
doc 2: "payment failed again"
doc 3: "package arrived damaged"
Word In how many docs IDF = ln(3/df) TF in doc 1 TF-IDF in doc 1
─────── ──────────────── ────────────── ─────────── ───────────────
refund 1 1.099 0.333 0.366
payment 2 0.405 0.333 0.135
package 1 1.099 0 0
"refund" scores almost 3× higher than "payment" in doc 1: both appear
once, but "refund" is rare across tickets, so it says more about this
ticket. (Libraries add small smoothing terms, but the idea is the same.)Analogy — Recognizing someone at a party by their accent. Everyone at the party says “hello” — it tells you nothing about who is speaking. But only one guest has a distinctive accent. TF-IDF listens for the accent, not the “hello.”
N-grams — Capturing Short Phrases
Single words lose order. N-grams add sequences of n words as extra features.
"the reset link is not arriving"
Unigrams (1): the, reset, link, is, not, arriving
Bigrams (2): the reset, reset link, link is, is not, not arriving
Why it matters:
"log in" (account) vs "log" (technical logs)
"not working" vs "working"
"credit card" vs "card" (could be a gift card, a SIM card...)In our project, adding bigrams raised accuracy on unseen ticket wordings from 70.5% to 79.9% — the single biggest improvement from any one change.
Analogy — Reading two words at a time. “Hot dog” means something entirely different from “hot” plus “dog.” Bigrams let the model notice when two words together mean more than each word alone.
Text Classification — Baselines First
As with every model in this series, start with simple baselines.
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
model = Pipeline([
("tfidf", TfidfVectorizer(preprocessor=clean, ngram_range=(1, 2),
min_df=2, sublinear_tf=True)),
("clf", LogisticRegression(max_iter=1000, C=5)),
])
model.fit(train.text, train.label)
predictions = model.predict(test.text)Approaches compared:
Always predict the most common category 25.0% (4 equal classes)
Keyword rules ("refund" → billing...) 74.9% on unseen wordings
TF-IDF unigrams + logistic regression 70.5% on unseen wordings
TF-IDF unigrams+bigrams + logistic reg. 79.9% on unseen wordings
Hand-written keyword rules beat the unigram model! Rules written by
someone who knows the business are a strong baseline -- and a common,
perfectly reasonable first version in production.TF-IDF with logistic regression is the classic NLP workhorse: fast to train (seconds), cheap to run on millions of rows, and easy to explain because you can read the most important words per class.
Top words the model learned per category:
account account, my account, how do, log
billing card, my card, subscription, refund, payment
shipping order, ordernum, arrived, the package, received
technical page, error, update, are notThe 100% Accuracy Trap — Near-Duplicate Leakage
Our first experiment used a standard random train/test split. Every model scored 100% accuracy.
Random split results:
Bag of words + naive Bayes 100.0%
TF-IDF unigrams + logistic reg. 100.0%
TF-IDF unigrams+bigrams 100.0%
Then we tried four brand-new tickets written differently:
"the charge on my card is wrong" → billing ✓ (0.91 confidence)
"app freezes when i click export" → technical ✓ (0.82)
"reset link never came" → account ✓ (0.58)
"my parcel never showed up" → billing ✗ should be shippingWhy 100% was a lie: customers phrase the same problem in similar ways, so a random split put near-identical tickets (“my order 12345 has not arrived” and “my order 67890 has not arrived”) on both sides of the split. The test was really asking “have you seen this sentence before?” — not “can you understand a new one?”
The honest test holds out entire phrasings: we removed two of the eight ways customers describe each problem from training, and tested only on those unseen wordings. That is where the realistic numbers above — 70–80% — come from.
How to split text data honestly:
Group by template, customer, or conversation Keep near-duplicates on
one side (GroupShuffleSplit)
Split by time Train on last year, test on
this month's new tickets
Deduplicate first Remove exact and near-exact
copies before splittingAnalogy — Memorizing last year’s exam answers. A student who memorized last year’s exam scores 100% if this year’s exam reuses the same questions — and fails when the teacher rewrites them. Near-duplicate text is the reused question; holding out whole phrasings is the rewritten exam. (Our model evaluation guide covers splitting strategies in general.)
Where Bag-of-Words Breaks Down
Looking at the mistakes on unseen wordings shows exactly what word counting cannot do:
Real misclassifications (true → predicted):
"i need a copy of my receipt for order 50213 expenses"
billing → shipping The word "order" pulled it to shipping;
"receipt" was never seen in training
"how do i change the email address on my account"
account → shipping A phrasing never seen before; the
model had no learned words to rely on
"login shows a blank screne in chrome"
technical → billing A typo ("screne") and an unseen phrasing
"my parcel never showed up"
shipping → billing "parcel" never appeared in training --
the model has no idea it means "package"
Per-category results on unseen wordings:
technical F1 0.988 Distinctive words (error, page, crash)
account F1 0.826
shipping F1 0.735 Absorbs anything mentioning "order"
billing F1 0.667 Receipt requests misrouted to shippingEvery one of these failures has the same root cause: bag-of-words knows which words appeared, not what they mean. It cannot tell that “parcel” means “package”, or that a receipt request is about money even when it mentions an order. That is the problem embeddings solve.
Word Embeddings — Meaning as Coordinates
An embedding represents a word or sentence as a list of numbers — a point in space — placed so that similar meanings land close together.
Analogy — GPS coordinates for meaning. Every word gets coordinates on a giant map. “Parcel”, “package”, and “shipment” are neighbors in the same district. “Refund” and “reimbursement” live on the same street. “Package” and “password” are on opposite sides of the city. A model can now reason “this new word is right next to words I know mean shipping.”
The classic illustration (toy 3-number vectors, real ones have hundreds):
king = [0.9, 0.8, 0.1] dimensions loosely mean:
man = [0.1, 0.8, 0.1] [royalty, maleness, femaleness]
woman = [0.1, 0.1, 0.8]
queen = [0.9, 0.1, 0.8]
king - man + woman = [0.9, 0.1, 0.8] = queen
How Word2Vec-style embeddings are learned:
"You shall know a word by the company it keeps."
The model reads billions of sentences and learns to predict each word
from its neighbors. Words that appear in similar contexts ("my parcel
arrived", "my package arrived") end up with similar vectors.Analogy — Judging people by their friends. You may not know a newcomer at a party, but if they spend the evening talking with the data engineers, you can guess what they do. Word2Vec learns a word’s meaning from the words it hangs around with.
Contextual Embeddings and Transformers
Classic word embeddings give each word one vector, regardless of context.
The problem:
"I deposited money at the bank" ← financial institution
"We sat on the river bank" ← edge of a river
Word2Vec: the SAME vector for "bank" in both sentences.
Contextual embeddings (BERT and modern transformer models):
Every word's vector depends on the whole sentence around it, using
the self-attention mechanism. "bank" gets different vectors in the
two sentences above.
Sentence embeddings:
Models trained to produce ONE vector for a whole sentence or
paragraph, so whole tickets, documents, or questions can be compared
by meaning. These power semantic search and RAG.The transformer architecture behind this is explained in our neural networks and deep learning post.
Text Similarity and Semantic Search
Cosine similarity measures how closely two vectors point in the same direction: 1.0 means identical direction, 0 means unrelated.
Analogy — Two compass needles. Cosine similarity ignores how long the needles are and only asks whether they point the same way. A long document and a short question can still point in the same direction if they are about the same thing.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
sentences = ["reset my password", "i forgot my login",
"send me a password reset link", "my package is late"]
vectors = TfidfVectorizer(stop_words="english").fit_transform(sentences)
print(cosine_similarity(vectors[0], vectors).round(2))
# [[1. 0. 0.62 0. ]]TF-IDF similarity to "reset my password":
1.00 reset my password
0.00 i forgot my login ← same problem, ZERO similarity
0.62 send me a password reset link
0.00 my package is late
TF-IDF can only match shared words. "I forgot my login" shares none
with "reset my password", so keyword search would never connect them.
Embedding similarity (illustrative vectors):
cos("reset my password", "i forgot my login") ≈ 0.996 very similar
cos("reset my password", "my package is late") ≈ 0.214 unrelated
This is exactly why RAG and modern search use embeddings, and why
hybrid search combines keywords (exact terms, codes, names) with
vectors (meaning).Our RAG pipelines post shows how embeddings, chunking, and hybrid search come together in Azure AI Search.
Topic Modeling — Finding Themes Without Labels
What if nobody has labeled the tickets? Topic modeling finds groups of words that tend to appear together, without any labels. Here we ran NMF (non-negative matrix factorization) on TF-IDF vectors of all 2,240 tickets, asking for four topics.
from sklearn.decomposition import NMF
vectorizer = TfidfVectorizer(preprocessor=clean, stop_words="english", min_df=5)
X = vectorizer.fit_transform(tickets.text)
nmf = NMF(n_components=4, random_state=42, max_iter=500).fit(X)Topics discovered with NO labels (top words, and which real
category dominates each topic):
Topic 0: account, address, change, team, package, delivered account 47%
Topic 1: order, arrived, charged, damaged, crushed, box shipping 74%
Topic 2: page, error, crashes, billing, time, open technical 92%
Topic 3: invoice, card, log, download, paying, declined billing 75%
Three of the four topics clearly match a real team. Topic 0 mixes
"change my address" requests from account AND shipping tickets --
"address" means different things to different teams.Analogy — Sorting a pile of unlabeled mail. Without being told the departments, you could still notice that letters mentioning “invoice” and “card” cluster together, and letters mentioning “crushed box” form another pile. Topic modeling does this automatically — and, like a human, it gets confused when one word (“address”) belongs to two departments. Use it to explore themes and to bootstrap labels, then validate with people who know the business.
Other Core NLP Tasks
Sentiment analysis Positive / negative / neutral, often per aspect
("food great, service slow")
Named entity Find people, organizations, products, locations,
recognition (NER) dates, amounts in text
Language detection Route text to the right model or translator
Keyword extraction Pull the main phrases from a document
Summarization Shorten long text (extractive or generative)
Translation Convert between languages
PII detection Find and mask personal information before storage
For most of these, prebuilt services (Azure AI Language) or LLMs are
faster to adopt than training your own models. Our AI-103 post on
text analysis and speech covers the Azure services in detail.See the Azure text analysis and speech post for sentiment, NER, PII detection, and language understanding with Azure AI Language.
Tokens and LLMs — What Data Engineers Should Know
Large language models do not read words; they read tokens — subword pieces from a fixed vocabulary.
Rough rules of thumb for English text:
1 token ≈ 4 characters ≈ 0.75 words
100 tokens ≈ 75 words
A 1,500-word article ≈ 2,000 tokens
Why tokens matter to data engineers:
Cost LLM APIs bill per input and output token
Limits Context windows are measured in tokens; long documents
must be chunked for embeddings and RAG
Throughput Rate limits (tokens per minute) cap how fast a batch
job can process text
Other Code, numbers, and non-English text often use more
languages tokens per word, so they cost more
Back-of-envelope cost check before running a job:
2,000,000 tickets × ~60 tokens each ≈ 120 million input tokens
→ estimate cost and runtime BEFORE sending them to an LLM;
a TF-IDF classifier might do the routing for a fraction of it.Analogy — A taxi meter. Every token is a tick of the meter. Short trips are cheap, but a batch job sending millions of long documents runs up a large fare. Knowing the token count is how you estimate the fare before you get in.
NLP at Scale with Spark
For millions of tickets, Spark ML runs the same steps — cleaning, tokenizing, TF-IDF, classification — in parallel across a cluster. This pipeline was run on the same tickets and held-out wordings:
from pyspark.ml import Pipeline
from pyspark.ml.feature import (RegexTokenizer, StopWordsRemover, NGram, HashingTF,
IDF, StringIndexer, IndexToString, SQLTransformer)
from pyspark.ml.classification import LogisticRegression
label_indexer = StringIndexer(inputCol="label", outputCol="label_idx").fit(train)
pipeline = Pipeline(stages=[
SQLTransformer(statement="""SELECT *, regexp_replace(lower(text), '[0-9]{5}', ' ordernum ')
AS clean_text FROM __THIS__"""),
RegexTokenizer(inputCol="clean_text", outputCol="tokens", pattern="[^a-z0-9-]+"),
StopWordsRemover(inputCol="tokens", outputCol="words"),
NGram(n=2, inputCol="tokens", outputCol="bigrams"),
SQLTransformer(statement="SELECT *, concat(words, bigrams) AS terms FROM __THIS__"),
HashingTF(inputCol="terms", outputCol="tf", numFeatures=2**16),
IDF(inputCol="tf", outputCol="features", minDocFreq=2),
label_indexer,
LogisticRegression(featuresCol="features", labelCol="label_idx", maxIter=100, regParam=0.01),
IndexToString(inputCol="prediction", outputCol="predicted_label", labels=label_indexer.labels),
])
model = pipeline.fit(train)
scored = model.transform(spark.table("support.silver.new_tickets"))Result on unseen wordings: 67.2% accuracy
(vs 79.9% for the scikit-learn version)
Why lower? Two differences from the scikit-learn version, likely both
contributing:
StopWordsRemover Removes "my", "how", "do" from the single-word
features -- yet words like these formed some of the
strongest account signals in scikit-learn
HashingTF Hashes terms into a fixed number of slots instead of
building a vocabulary: scalable, but occasional
collisions merge unrelated terms
The lesson: preprocessing choices are part of the model. Test them on
an honest held-out set rather than copying a standard recipe.
Production notes:
Save the whole PipelineModel -- cleaning and features travel with the
classifier, preventing training-serving skew
Log it with MLflow and score new tickets in a scheduled job
Write predicted_label and probability back to a Delta tableThe MLOps with MLflow post covers registering and scheduling models like this one.
Classic NLP vs Azure AI Language vs LLMs
| Approach | Best for | Cost at scale | Accuracy on new wordings | Setup effort |
|---|---|---|---|---|
| Keyword rules | Quick first version, clear-cut categories | Near zero | Brittle | Hours |
| TF-IDF + linear model | High-volume classification with labeled data | Very low | Good, misses synonyms | Days |
| Embeddings + classifier | Classification that must handle new phrasings | Low–moderate | Strong | Days |
| Azure AI Language (prebuilt) | Sentiment, NER, PII, language detection without training | Pay per text record | Strong for standard tasks | Hours |
| LLM prompting | Complex, low-volume, or changing tasks; extraction; summarization | Highest per item | Strongest, flexible | Hours (plus evaluation) |
A practical path for ticket routing:
1. Keyword rules → working baseline in a day
2. TF-IDF + logistic regression → beats rules once you have labels
3. Embeddings or an LLM for the tickets the cheap model is unsure about
(route only low-confidence cases to the expensive model)
4. Measure everything on an honest held-out setCommon Mistakes
Random splits on near-duplicate text. Customers repeat the same phrasings, so a random split leaks almost identical tickets into the test set. Our models scored a meaningless 100% this way. Split by template, customer, conversation, or time, and deduplicate first.
Removing stop words by default. Negations (“not”) and short phrases (“my account”, “how do”) often carry the signal. In our Spark pipeline, removing stop words before building features lowered accuracy. Test it rather than assuming it helps.
Turning IDs into vocabulary. Order numbers, ticket IDs, and email addresses create thousands of useless one-off “words.” Replace them with placeholder tokens, or remove them during cleaning.
Cleaning differently in training and production. If training text was lowercased and stripped of punctuation but production text is not, predictions quietly degrade. Package cleaning inside the model pipeline so it always runs the same way.
Assuming keyword features understand meaning. Bag-of-words cannot connect “parcel” with “package.” Review misclassified examples regularly, and add embeddings when new phrasings are common.
Storing raw text with personal information. Tickets and emails contain names, phone numbers, and account details. Detect and mask PII before text lands in analytics tables or is sent to external services.
Sending everything to an LLM by default. For high-volume, well-defined classification, a TF-IDF model can be far cheaper and faster with comparable accuracy. Estimate token costs first and reserve LLMs for the cases simpler models cannot handle.
Ignoring encoding problems. Garbled characters like “’” mean text was decoded with the wrong character set upstream. Fix encoding at ingestion; models cannot recover meaning from mojibake.
Interview Questions
Q: What are the main steps in an NLP preprocessing pipeline? A: Typically cleaning (lowercasing, removing or replacing IDs, URLs, and noise, masking PII, fixing encoding), tokenization into words or subwords, optional stop-word removal, optional stemming or lemmatization, and then vectorization with bag-of-words, TF-IDF, or embeddings. Each step is a choice that should be validated on held-out data, because removing stop words or punctuation can delete important signal like negations.
Q: What is the difference between stemming and lemmatization? A: Stemming chops word endings using rules: it is fast but crude, can produce non-words (“studies” → “studi”), and can merge unrelated words (“university” and “universe” both become “univers”). Lemmatization uses a dictionary and the word’s part of speech to return the proper base form (“studies” → “study”, “better” → “good”). Lemmatization is more accurate; stemming is cheaper at very large scale.
Q: Explain TF-IDF. A: TF-IDF weights each word by how often it appears in a document (term frequency) multiplied by the log of how rare it is across all documents (inverse document frequency). Words common everywhere, like “the”, get low weights; words frequent in one document but rare overall get high weights. It is a simple, fast, and effective representation for search and classification, but it only matches exact words and cannot capture synonyms.
Q: Why might a text classifier score extremely high in testing but perform poorly in production? A: The most common reason is leakage from near-duplicate text: similar phrasings, templates, or messages from the same conversation end up in both training and test sets, so the test measures memorization rather than generalization. Splitting by group or time and deduplicating before splitting gives an honest estimate. Other causes include different cleaning in production and new vocabulary the model never saw.
Q: What are word embeddings, and how do they differ from TF-IDF? A: Embeddings represent words or sentences as dense vectors learned from large text corpora, so similar meanings have nearby vectors — “parcel” lands near “package.” TF-IDF vectors are sparse counts of exact words with no notion of meaning, so two sentences with no shared words always have zero similarity. Contextual embeddings from transformers go further by giving the same word different vectors depending on its sentence.
Q: How does cosine similarity work, and where is it used? A: Cosine similarity measures the angle between two vectors, ignoring their length, from 1 (same direction) to 0 (unrelated) for typical text vectors. It is used to compare documents, find duplicates, cluster text, and power semantic search and RAG, where a question’s embedding is compared with document chunk embeddings to retrieve the most relevant passages.
Q: What is topic modeling, and when would you use it? A: Topic modeling, using methods such as LDA or NMF, discovers groups of words that frequently occur together across documents, without labels. It is useful for exploring large unlabeled text collections, monitoring how themes change over time, and creating starting labels for supervised models. Results need human review, because words used by several groups can blur topics.
Q: When would you choose TF-IDF with a linear model over an LLM? A: When the task is a well-defined classification over high volumes of text and you have labeled examples: TF-IDF models train in seconds, cost almost nothing per prediction, run easily in Spark, and are explainable. LLMs are better for complex, low-volume, or frequently changing tasks, extraction, and summarization. A common pattern is to use the cheap model first and send only low-confidence cases to an LLM.
Wrapping Up
NLP turns unstructured text into data you can count, classify, and search. Cleaning and tokenization prepare the text, bag-of-words and TF-IDF turn it into numbers, n-grams capture short phrases, and linear models give fast, explainable classifiers — the workhorse for high-volume routing. Our project also showed their limits honestly: a random split produced a meaningless 100%, unseen phrasings revealed the real 70–80%, and every failure came from the same gap — counting words is not understanding them. Embeddings close that gap by placing meaning on a map, which is why they power semantic search, RAG, and modern language models.
For data engineers, the practical skills are clear: clean consistently inside the pipeline, mask PII early, split text honestly, scale feature extraction with Spark, estimate token costs before calling an LLM, and choose the simplest approach that meets the accuracy you need.
Related posts: – Neural Networks and Deep Learning – Feature Engineering – Model Evaluation – MLOps with MLflow – Python Regular Expressions – RAG Pipelines on Azure – Text Analysis and Speech on Azure
Naveen Vuppula is a Senior Data Engineer and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.