Fixing Python Pandas fillna That Silently Skips Columns With Categorical Dtype

You call df.fillna('Unknown') on your DataFrame, check the result, and everything looks fine — until you notice a column still has NaN sitting in it. No error, no warning, just silent failure. If that column has a Categorical dtype, that's exactly what you're dealing with.

This behavior catches a lot of people off guard because fillna works perfectly on numeric and string columns in the same call. The Categorical column is the odd one out, and Pandas doesn't tell you about it.

What You'll Learn

• Why Pandas fillna silently skips columns with Categorical dtype
• How to reproduce the bug so you can confirm the root cause
• Three concrete fixes, each with trade-offs explained
• How to safely handle mixed DataFrames where only some columns are categorical
• Pitfalls to avoid so the fix doesn't introduce new problems

Prerequisites

You'll need Pandas installed (any version from 1.0 onward exhibits this behavior). The examples use Python 3.8+ syntax. No external libraries beyond Pandas and NumPy are required.

The Problem: fillna Looks Fine but NaN Persists

Here's a minimal example that shows the failure. You have a DataFrame with a mix of column types, one of which is Categorical.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', None],
    'score': [88, np.nan, 72],
    'grade': pd.Categorical(['A', np.nan, 'C'], categories=['A', 'B', 'C'])
})

df_filled = df.fillna('Unknown')
print(df_filled)
print(df_filled.dtypes)

The output looks like this:

    name  score grade
0  Alice   88.0     A
1    Bob    NaN   NaN
2  Unknown  72.0     C

name      object
score    float64
grade   category
dtype: object

The name column got filled. The grade column did not. score also didn't fill because 'Unknown' isn't a valid float — but that's a different problem. The point is that grade failed without any indication at all.

Why Categorical Dtype Breaks fillna

A Categorical column maintains a fixed list of allowed values, called categories. When you create a Categorical series with categories=['A', 'B', 'C'], Pandas encodes each value as an integer index into that list. NaN is stored as a sentinel value (-1) outside the category codes.

When fillna tries to replace NaN with 'Unknown', it checks whether 'Unknown' is a valid category. It isn't in the list ['A', 'B', 'C'], so Pandas silently refuses to make the substitution rather than raising a ValueError or even a warning. This is arguably a design flaw — silent failures are harder to debug than loud ones.

The same constraint applies if you try to assign a value not in the category list directly: series[0] = 'Unknown' would raise an error. But fillna swallows the problem instead of surfacing it.

This is a similar class of problem to the one described in Pandas GroupBy silently dropping columns with mixed types — a dtype mismatch causes data to disappear without a trace.

Reproducing the Bug in Isolation

Before applying a fix, it helps to confirm the exact behavior on a minimal series so you understand what you're working with.

s = pd.Categorical(['A', np.nan, 'C'], categories=['A', 'B', 'C'])
cat_series = pd.Series(s)

result = cat_series.fillna('Unknown')
print(result)
print(result.isna().sum())  # Still 1

You'll see the NaN is still there and isna().sum() returns 1. Contrast this with an object-dtype series: pd.Series(['A', None, 'C']).fillna('Unknown') fills correctly without issue.

Fix 1: Add the Fill Value to the Category List First

The cleanest fix when you need to keep the Categorical dtype is to add your fill value to the list of allowed categories before calling fillna. Pandas exposes cat.add_categories() for exactly this purpose.

df['grade'] = df['grade'].cat.add_categories(['Unknown']).fillna('Unknown')
print(df['grade'])
print(df['grade'].dtype)  # Still category

This works because 'Unknown' is now a valid code in the category index, so fillna can assign it. The column stays Categorical, which preserves any memory efficiency you were counting on.

If you want to clean up unused categories afterward (for example, if some fill values ended up never being used in other rows), chain a cat.remove_unused_categories() call:

df['grade'] = (
    df['grade']
    .cat.add_categories(['Unknown'])
    .fillna('Unknown')
    .cat.remove_unused_categories()
)

This approach is ideal when the fill value itself is meaningful (a real category, not just a technical placeholder) and you want to preserve the ordered or unordered category structure.

Fix 2: Convert to Object Dtype, Fill, Then Re-Categorize

If the column's categories aren't important downstream, or you just want a quick reliable fill, convert the column to object dtype first, fill, and optionally convert back.

df['grade'] = (
    df['grade']
    .astype(object)
    .fillna('Unknown')
    .astype('category')
)
print(df['grade'])
print(df['grade'].cat.categories)  # Now includes 'Unknown'

Converting to object removes all dtype constraints, so fillna works as expected. Converting back with .astype('category') infers the new category list from the actual values, which now includes 'Unknown'.

One caveat: if your original column had a specific ordered category list (for example, pd.CategoricalDtype(['Low', 'Medium', 'High'], ordered=True)), the round-trip through object loses that ordering. You'd need to re-apply the dtype explicitly:

original_dtype = df['grade'].dtype  # Save before conversion

df['grade'] = (
    df['grade']
    .astype(object)
    .fillna('Unknown')
)

# Re-apply ordered dtype if needed, adding the fill value
new_cats = list(original_dtype.categories) + ['Unknown']
new_dtype = pd.CategoricalDtype(categories=new_cats, ordered=original_dtype.ordered)
df['grade'] = df['grade'].astype(new_dtype)

Fix 3: Use fillna With a Per-Column Dictionary and Pre-Extend Categories

When you're working on a whole DataFrame and want to apply different fill values to different columns — some of which are categorical — use a dictionary with fillna and pre-process the categorical columns first.

fill_values = {
    'name': 'Unknown',
    'score': 0,
    'grade': 'Unknown'
}

# Extend categories on all categorical columns
for col, val in fill_values.items():
    if col in df.columns and hasattr(df[col], 'cat'):
        if val not in df[col].cat.categories:
            df[col] = df[col].cat.add_categories([val])

df = df.fillna(fill_values)
print(df)

This pattern is robust in pipeline code because it dynamically checks whether a column is categorical before attempting to extend it. It won't fail if you later remove the Categorical dtype from a column — the hasattr(df[col], 'cat') guard handles that gracefully.

Handling Mixed DataFrames With Some Categorical Columns

In real data pipelines, you often don't know at write time which columns will be categorical and which won't. A safe utility function handles this automatically:

def safe_fillna(df: pd.DataFrame, fill_value) -> pd.DataFrame:
    """Fill NaN in all columns, correctly handling Categorical dtype."""
    df = df.copy()
    for col in df.columns:
        if pd.api.types.is_categorical_dtype(df[col]):
            if fill_value not in df[col].cat.categories:
                df[col] = df[col].cat.add_categories([fill_value])
        df[col] = df[col].fillna(fill_value)
    return df

df_filled = safe_fillna(df, 'Unknown')
print(df_filled.isna().sum())  # All zeros

Note that pd.api.types.is_categorical_dtype() was deprecated in Pandas 2.1 in favor of isinstance(df[col].dtype, pd.CategoricalDtype). If you're on Pandas 2.x, use the newer form:

def safe_fillna(df: pd.DataFrame, fill_value) -> pd.DataFrame:
    df = df.copy()
    for col in df.columns:
        if isinstance(df[col].dtype, pd.CategoricalDtype):
            if fill_value not in df[col].cat.categories:
                df[col] = df[col].cat.add_categories([fill_value])
        df[col] = df[col].fillna(fill_value)
    return df

This function is safe to drop into any pipeline. It makes a copy of the DataFrame rather than mutating the original, which avoids the SettingWithCopyWarning you'd get when modifying a slice. Speaking of silent data issues, Pandas value_counts silently excluding NaN is another gotcha worth reading about if you're auditing your pipeline for missing-data blind spots.

Common Pitfalls to Watch For

Filling with a numeric value on a string-category column

If your Categorical column contains strings and you try to fill with 0 or another numeric type, the category extension will work but you'll end up with mixed-type categories, which can break downstream operations like sorting or grouping. Keep fill values type-consistent with the rest of the column.

Forgetting to check after the fill

Always assert after a fill operation in pipeline code. A one-liner makes this easy:

assert df_filled.isna().sum().sum() == 0, "fillna left NaN values in the DataFrame"

This catches any dtype-related surprises before they propagate downstream. This is the same mindset you should carry when debugging other silent Pandas failures, such as Pandas dropna removing rows unexpectedly on partial NaN.

Using inplace=True with Categorical columns

fillna(inplace=True) is unreliable with Categorical columns in older Pandas versions and doesn't play well with the cat.add_categories() chaining pattern. Stick to assignment (df['col'] = df['col'].fillna(...)) for predictable behavior.

Ordered categories losing their order

When you add a new category via cat.add_categories(), the new value is appended to the end of the category list. For ordered categoricals (like 'Low' < 'Medium' < 'High'), placing 'Unknown' at the end means it would sort above 'High'. If order matters, reconstruct the dtype explicitly with the fill value placed where you want it in the ordering.

Applying fillna to a copy you don't assign back

Pandas operations on Series return new objects; they don't modify in place unless you set inplace=True (which has its own problems). If you write df['grade'].cat.add_categories(['Unknown']) without assigning the result back, nothing changes. Always use the full assignment form shown in the examples above.

If you find yourself debugging a chain of transformations where values keep disappearing, Pandas apply silently ignoring errors on axis=1 covers a related family of silent-failure patterns worth knowing.

Wrapping Up

The core issue is simple: Pandas won't fill a Categorical column with a value that isn't already in its category list, and it won't tell you it skipped the fill. Once you know that, the fix is straightforward.

Here are the concrete steps to take from here:

• Run df.isna().sum() after any fillna call in existing code and verify the counts are what you expect.
• For categorical columns, use cat.add_categories([fill_value]) before fillna, or convert to object, fill, and re-categorize.
• Drop the safe_fillna utility function into your shared utilities module so all pipelines in your project get the fix automatically.
• Add an assertion (assert df.isna().sum().sum() == 0) after critical fill steps in any ETL or data-cleaning script.
• If you're on Pandas 2.x, use isinstance(col.dtype, pd.CategoricalDtype) instead of the deprecated is_categorical_dtype helper.