Build Your First AI Text Classifier with Python

Artificial intelligence can feel abstract until you build something that makes a real decision. In this tutorial, you will create a small but complete text classifier that labels a message as either support or promotion.

The project uses classic machine learning, so it runs on an ordinary laptop, requires no paid API, and gives you a clean foundation for more advanced natural-language processing.

What you will build

Our classifier will learn from example messages such as:

• “I cannot reset my password” → support
• “Claim your free discount today” → promotion

Then it will predict the category of a message it has never seen before. Along the way, you will learn how to:

1. Convert text into numbers with TF-IDF.
2. Train a logistic-regression model.
3. Measure the model with cross-validation.
4. Return both a prediction and a confidence score.
5. Think about data quality and responsible AI.

How text classification works

A machine-learning model cannot read words directly. TF-IDF turns each message into a numeric vector. Words that are useful and distinctive receive more weight, while extremely common words receive less.

Logistic regression then learns which weighted word patterns are associated with each label. Despite its simple name, it is a strong baseline for many text-classification tasks because it is fast, interpretable, and works well with sparse text features.

Start with a simple baseline you can understand. Complexity is useful only when it solves a measured problem.

Install the dependencies

Create a new folder, open a terminal inside it, and run:

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install scikit-learn

Now create a file named classifier.py.

Complete Python code

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline

# A tiny teaching dataset. Real projects need many more reviewed examples.
texts = [
    "I cannot log in to my account",
    "Please help me reset my password",
    "The video lesson will not load",
    "My payment was deducted but access is locked",
    "How can I download my certificate",
    "The quiz result is not updating",
    "I need help changing my email address",
    "The course page shows an error",
    "Claim your free discount today",
    "Limited offer buy now and save",
    "Congratulations you have won a prize",
    "Get fifty percent off this weekend",
    "Exclusive deal available for you",
    "Click here to receive your reward",
    "Special promotion ends tonight",
    "Unlock bonus savings right now",
]

labels = [
    "support", "support", "support", "support",
    "support", "support", "support", "support",
    "promotion", "promotion", "promotion", "promotion",
    "promotion", "promotion", "promotion", "promotion",
]

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        lowercase=True,
        ngram_range=(1, 2),
        min_df=1,
    )),
    ("classifier", LogisticRegression(max_iter=1000)),
])

# Estimate performance without permanently holding out our tiny dataset.
folds = StratifiedKFold(n_splits=4, shuffle=True, random_state=42)
scores = cross_val_score(model, texts, labels, cv=folds, scoring="accuracy")
print(f"Cross-validation accuracy: {scores.mean():.2f}")

# Train on all examples after evaluation.
model.fit(texts, labels)


def classify(message: str) -> tuple[str, float]:
    probabilities = model.predict_proba([message])[0]
    best_index = probabilities.argmax()
    label = model.classes_[best_index]
    confidence = probabilities[best_index]
    return label, float(confidence)


examples = [
    "Please help, my course payment is not showing",
    "Exclusive reward available today only",
]

for message in examples:
    label, confidence = classify(message)
    print(f"{message!r} -> {label} ({confidence:.1%})")

Run the program:

python classifier.py

Your exact accuracy and confidence values may vary slightly. The important result is that the first new message should lean toward support, while the second should lean toward promotion.

Why use a pipeline?

The Pipeline keeps text preparation and model training together. This matters because the same TF-IDF rules used during training must also be applied during prediction. It also makes evaluation safer: each cross-validation fold learns its vocabulary only from its training portion, reducing accidental data leakage.

Improve the classifier

This sample is intentionally small. A production-ready model needs better data and testing.

1. Collect representative examples

Use messages that resemble the language your real users write. Include spelling mistakes, short messages, mixed casing, and difficult edge cases. Remove private information before using customer messages for training.

2. Add a validation set

Keep a final set of labeled messages that is never used while tuning the model. This gives you a more honest estimate of real-world performance.

3. Inspect mistakes

Do not look only at total accuracy. Print misclassified examples and ask why the model failed. You may discover unclear labels, duplicated examples, or a missing category.

4. Measure each class

For uneven datasets, review precision, recall, and the confusion matrix:

from sklearn.metrics import classification_report

predictions = model.predict(texts)
print(classification_report(labels, predictions))

5. Save the trained model

Once you are satisfied with testing, save the complete pipeline:

import joblib

joblib.dump(model, "text_classifier.joblib")
loaded_model = joblib.load("text_classifier.joblib")

Never load a model file from an untrusted source. Serialized model files can execute unsafe code when opened.

Responsible AI checklist

Before using a classifier in a real workflow, ask:

• Did people consent to their data being used?
• Did we remove names, email addresses, payment details, and other sensitive data?
• Are some languages, dialects, or writing styles underrepresented?
• Can a human review low-confidence or high-impact decisions?
• Are predictions logged safely so mistakes can be investigated?
• Can users appeal or correct an incorrect classification?

A confidence score is not a guarantee that a prediction is correct. For important decisions, use the model to assist a person rather than silently replacing human judgment.

Try this mini challenge

Extend the project with a third category named feedback. Add at least eight examples, retrain the model, and test messages such as “The instructor explained the topic clearly.” Then display a “needs review” result whenever the best confidence score is below 65%.

Where to go next

You have built a complete AI workflow: data, features, training, evaluation, prediction, and responsible-use checks. The next step is not necessarily a larger model—it is a better dataset and a clearer understanding of the mistakes that matter.

Explore more hands-on programming lessons in the MaheiLambi course catalog, or use the browser playground to practise smaller Python ideas before turning them into a full project.

Understanding the central idea

An AI feature is a complete product flow, not a single model call. Useful systems define the input, protect private data, give the model precise context, validate the response, and show a safe fallback when the result is unavailable.

The purpose of this article is to connect that idea to a complete working flow. Individual commands matter, but the lasting skill is understanding why each part exists and how information moves from the user's action to a trustworthy result.

Begin with the nouns and verbs in the problem. The nouns usually become data—such as a user, transaction, note, file, or task—while the verbs become operations such as create, validate, calculate, update, and report. This simple translation gives the project a shape before framework or library choices distract from the core behaviour.

It also helps to separate facts from derived values. Store facts that arrived from a trusted input and calculate summaries from those facts when possible. Duplicating calculated totals in several places creates inconsistencies because one copy can change while another remains stale.

How the pieces work together

The browser sends a limited request to the application server. The server authenticates the user, checks size and format, calls the model with a controlled instruction, validates the returned structure, and sends only the approved fields back to the interface.

Build the smallest successful path first. Keep input handling, core logic, storage, and presentation distinct even when they live in one file. This makes the project easier to explain today and easier to split into modules when it grows.

Validation belongs close to the boundary where new data enters. The core logic can then work with values that already satisfy basic rules. Persistence should receive a complete valid change, while presentation should translate the outcome into language the user understands. This order prevents a partially processed request from leaking into saved data.

Naming is part of the design. A function such as calculate_monthly_total communicates more than process, and a value such as normalised_category shows that a transformation has already happened. Clear names reduce the amount of state a beginner must remember while reading the code.

A realistic flow from start to finish

A notes assistant can accept a short lesson note and return a summary, key terms, and revision points as structured JSON. Saving the original note separately means the learner never loses work when the AI request times out or produces an unusable answer.

Follow one record through the whole system and inspect its value after every meaningful transformation. This is more instructive than copying a finished code listing because it reveals where assumptions enter the program and where an incorrect value would first become visible.

For the first implementation, use a tiny dataset that can be checked by hand. Three or four records are usually enough to expose ordering, totals, duplicates, and empty-state behaviour. Once the hand-calculated result agrees with the program, add a larger or messier input and observe which assumptions no longer hold.

Keep the successful flow visible in the interface or console output. The result should confirm what changed and include the identifier or summary needed for the next action. A generic message such as “done” hides useful evidence and makes later debugging unnecessarily difficult.

Reliability and common failure points

API keys belong on the server. Add input limits, rate limits, timeouts, content guidance, and a clear statement that generated material may be incorrect. Log operational failures without storing sensitive learner text unnecessarily.

Treat error handling as part of the user experience. A useful error message says what failed, what remained safe, and what action can be taken next. During development, keep technical detail in logs while presenting concise recovery guidance to the reader or end user.

Test failures at the same layer that owns the rule. Input-format tests belong near validation, calculation examples belong near the core logic, and save-and-reload checks belong near persistence. This makes a failed test point toward one responsibility instead of forcing the learner to inspect the entire application.

Retries also need care. A retry should not create a duplicate record or repeat a payment-like action. Stable request identifiers, uniqueness rules, or an explicit check before writing make repeated actions safe. Even a beginner project benefits from understanding that users double-click buttons and networks repeat requests.

What a complete result demonstrates

The finished feature should remain useful when the model is slow or wrong. It should make generated content identifiable, preserve the learner's original input, and provide a retry path without duplicate charges or duplicate records.

At that point, improvements such as a richer interface, more automation, or cloud deployment become controlled extensions rather than substitutes for an unfinished core. The result is a project that teaches transferable reasoning as well as syntax.

Document the final flow in a short README with setup steps, one realistic example, expected output, and known limitations. This turns the project into something another person can run and review. It also reveals missing assumptions that were obvious only on the original developer's computer.

The best next improvement is the one supported by evidence from actual use. A confusing message may matter more than a new chart, and protecting saved data may matter more than adding another button. This prioritisation habit is one of the most valuable lessons an end-to-end project can teach.

Worked case study: from problem to evidence

This is an illustrative case study designed to make the engineering decisions concrete. It does not claim results from a named organisation; every conclusion follows from the described inputs and observable behaviour.

Starting situation

A study assistant summarises lesson notes. Its first prototype sends unrestricted text directly from the browser and displays any returned string as trusted content. Slow responses lose the learner's draft, and malformed output breaks the page.

Intervention

The revised flow preserves the original note first, sends a size-limited request through the server, requests structured fields, validates the response, and displays generated material with a visible AI label and retry state.

Evidence collected

The interface remains usable when the model times out, rejects invalid response shapes without losing the note, and never exposes the provider key in browser code. Valid responses consistently contain the agreed summary sections.

Practical lesson

The model is one uncertain dependency inside a dependable product. Validation, privacy, fallbacks, and clear authorship are what turn a demonstration into a responsible learning tool.

A useful case study separates observation from opinion. The starting state records the problem, the intervention records what changed, and the evidence shows whether the change produced the intended behaviour. This structure helps readers evaluate an approach instead of accepting a success claim without support.

Test cases and expected behaviour

The following cases act as an executable specification. They are not questions for the reader; they state the conditions, expected outcomes, and reason each check matters.

Run the smallest test first and keep its input stable while repairing a failure. When it passes, add boundary and recovery cases. Changing code and test data simultaneously makes the source of improvement difficult to identify.

For automated tests, use the same arrange-act-assert pattern throughout the project. Arrange creates a known starting state, act performs one behaviour, and assert compares the observable result with the documented expectation. A good assertion checks the outcome that matters to the user, not an internal implementation detail that may change during refactoring.

Interpreting test failures

A failed test is evidence of a mismatch between the implemented behaviour and the written expectation. First confirm that the expectation represents the intended product rule. Next reduce the failure to the smallest input that still reproduces it, inspect the boundary between stages, and change one cause at a time.

Failures often reveal missing product decisions rather than typing mistakes. An empty value, repeated request, unavailable service, or partial save forces the application to choose a behaviour. Recording that decision in both the article and the test suite prevents future changes from silently reintroducing the same uncertainty.

The final test report should state the revision tested, environment, cases executed, results, and any untested limitation. That short record turns “it worked for me” into evidence another learner or reviewer can evaluate.