Fixing Python Pandas sort_values That Puts NaN at Wrong Position by Default

You sort a DataFrame ascending and expect the smallest values at the top — but NaN rows end up at the bottom, not the top. Or you sort descending and your NaN rows suddenly jump to the top when you expected them last. Neither behavior is accidental, but it will absolutely break a ranking pipeline or a report if you don't know it's happening.

The root cause is simple: Pandas has a fixed default for NaN placement that ignores your ascending argument. Once you understand the rule, fixing it takes one extra keyword.

What you'll learn

• Exactly where Pandas places NaN rows when you call sort_values
• How to use na_position to place NaN at the start or end deliberately
• How to sort multiple columns with different NaN rules per column
• How to use the key= parameter for custom NaN handling
• The gotchas that bite even experienced Pandas users

Prerequisites

You need Python 3.8 or later and Pandas 1.3 or later. The examples below also assume a basic familiarity with DataFrames and Series. No external libraries are required beyond Pandas itself.

How Pandas handles NaN in sort_values by default

When you call df.sort_values(by="col"), Pandas always places NaN rows at the end of the result, regardless of whether you're sorting ascending or descending. That is the documented default, but many developers don't notice it until an edge case surfaces in production.

Here's a minimal example that exposes the behavior:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "score": [3.0, np.nan, 1.0, np.nan, 2.0]
})

print(df.sort_values("score", ascending=True))
print("---")
print(df.sort_values("score", ascending=False))

Output:

   score
2    1.0
4    2.0
0    3.0
1    NaN
3    NaN
---
   score
0    3.0
4    2.0
2    1.0
1    NaN
3    NaN

Notice: NaN ends up last in both directions. If your pipeline treats the last row as the "worst" value, this is correct for ascending order but wrong for descending order — where NaN should logically represent an unknown that could be the worst case and therefore sit at the bottom, yet it still appears there by coincidence. The real problem surfaces when you expect NaN first on an ascending sort (e.g., you want to flag incomplete records at the top of a report).

The na_position parameter: your primary fix

The sort_values method accepts a na_position argument that takes exactly two string values: "last" (the default) or "first". This is the cleanest and most readable fix for the majority of cases.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name": ["Alice", "Bob", "Carol", "Dave", "Eve"],
    "score": [88.0, np.nan, 72.0, np.nan, 95.0]
})

# NaN rows rise to the top — useful for flagging incomplete records
print(df.sort_values("score", ascending=True, na_position="first"))

Output:

    name  score
1    Bob    NaN
3   Dave    NaN
2  Carol   72.0
0  Alice   88.0
4    Eve   95.0

And to explicitly push NaN to the end on a descending sort (making the behavior unambiguous rather than relying on the default):

print(df.sort_values("score", ascending=False, na_position="last"))

    name  score
4    Eve   95.0
0  Alice   88.0
2  Carol   72.0
1    Bob    NaN
3   Dave    NaN

Always pass na_position explicitly even when you want the default behavior. It makes the intent clear to the next person reading the code — including future you.

Sorting multiple columns with mixed NaN rules

Things get trickier when you sort by multiple columns and each column has a different NaN strategy. As of Pandas 1.3, na_position accepts a list of strings that maps one-to-one with the by list. This is one of the most under-documented features of sort_values.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "department": ["Eng", "Eng", "HR", "HR", "Eng"],
    "salary":     [90000, np.nan, 70000, np.nan, 85000]
})

# Sort by department ascending (NaN last), then salary descending (NaN first)
result = df.sort_values(
    by=["department", "salary"],
    ascending=[True, False],
    na_position=["last", "first"]
)
print(result)

  department   salary
1        Eng      NaN
0        Eng  90000.0
4        Eng  85000.0
3         HR      NaN
2         HR  70000.0

Each position in the na_position list corresponds to the same position in by and ascending. If your list lengths don't match, Pandas raises a ValueError immediately — so you'll catch that at runtime rather than silently getting wrong output.

This multi-column control is especially valuable in reporting pipelines where some nullable columns should surface missing data prominently while others should hide it at the bottom. Understanding how Pandas drops columns with mixed types during GroupBy is a related concern when your pre-sort data contains heterogeneous column dtypes.

Using key= to push NaN into a custom position

Sometimes na_position isn't granular enough. The key= parameter (available since Pandas 1.1) lets you transform column values before sorting. You can use this to assign NaN a specific numeric sentinel that places it exactly where you want.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "priority": [2.0, np.nan, 1.0, 3.0, np.nan]
})

# Treat NaN as priority 0 — higher urgency than everything else
def nan_as_zero(series):
    return series.fillna(0)

print(df.sort_values("priority", ascending=True, key=nan_as_zero))

   priority
1       NaN
4       NaN
2       1.0
0       2.0
3       3.0

The key= function receives a Series and must return a Series of the same length. The transformation is used only for comparison purposes — the original values, including NaN, are preserved in the output. This means you can use any numeric sentinel without corrupting your data. Be careful with this approach: if your data legitimately contains the sentinel value (e.g., actual zeros), you'll create a tie between NaN and that value.

If your DataFrame might have NaN introduced during earlier preprocessing steps, it helps to know how Pandas fillna silently skips categorical dtype columns — filling NaN before sorting is sometimes a cleaner fix than fighting with sort order.

Verifying NaN placement after sorting

Once you've applied your fix, it's worth asserting the result rather than eyeballing it. Two quick checks cover most scenarios.

Check 1: NaN is in the expected half

sorted_df = df.sort_values("score", ascending=True, na_position="first")

nan_indices = sorted_df["score"].isna()
first_valid = sorted_df["score"].first_valid_index()
nan_positions = sorted_df.index.get_loc(sorted_df[sorted_df["score"].isna()].index[0])

# Assert all NaN rows come before the first non-NaN row
assert sorted_df["score"].isna().sum() == 0 or \
    sorted_df["score"].iloc[:nan_indices.sum()].isna().all(), \
    "NaN rows are not all at the front"

Check 2: Non-NaN values are themselves sorted correctly

non_nan = sorted_df["score"].dropna().reset_index(drop=True)
assert non_nan.is_monotonic_increasing, "Non-NaN values are not sorted ascending"

Adding these assertions to your data pipeline catches regressions immediately. A future Pandas upgrade or a schema change that introduces a new nullable column won't silently reorder your output without triggering a test failure.

Common pitfalls when sorting with missing data

Pitfall 1: Relying on the default and calling it intentional

The most frequent mistake is assuming that because NaN ends up last, you did something intentional. If a teammate changes the sort direction later and doesn't know about the default, the NaN rows suddenly appear at the top relative to their expectations. Always be explicit.

Pitfall 2: Forgetting inplace=True resets nothing

When you use df.sort_values("col", inplace=True), the index is reordered in place but not reset. If downstream code uses positional iloc access expecting a 0-to-n index, you may silently read the wrong rows. Call .reset_index(drop=True) after sorting if positional access matters.

df.sort_values("score", ascending=True, na_position="first", inplace=True)
df.reset_index(drop=True, inplace=True)

Pitfall 3: Integer columns can't hold NaN without special dtype

If your column is a standard int64, it cannot contain NaN. Pandas will have silently upcast it to float64 when NaN was introduced. After sorting, converting back with astype(int) will raise or silently corrupt values. This is closely related to a problem covered in detail in the guide on how Pandas astype silently coerces NaN to zero in integer columns. Use pd.Int64Dtype() (the nullable integer type) if you need integer semantics with NaN.

Pitfall 4: sort_values on a Series vs. a DataFrame

When you call series.sort_values(), the na_position parameter works identically to the DataFrame version. The gotcha is that Series.sort_values returns a Series with the original index preserved — reassigning it back to a DataFrame column can misalign rows if the DataFrame index was reset at some point but the Series was not.

Pitfall 5: NaN in a sort key column after a merge

Joins frequently introduce NaN in key columns when there is no match on one side. Sorting immediately after a merge without inspecting for new NaN values in the sort column leads to silent misplacement. Always run df["sort_col"].isna().sum() before sorting post-merge to confirm your assumptions. See how Pandas value_counts silently excludes NaN for a similar pattern of NaN being ignored at a critical analysis step.

Wrapping up

The fix for NaN appearing at the wrong position in a sorted DataFrame is almost always one of two things: passing na_position="first" or na_position="last" explicitly, or using a key= function to map NaN to a sentinel value before comparison. Neither requires restructuring your pipeline.

Here are the concrete next steps to take right now:

1. Audit every sort_values call in your codebase and add an explicit na_position argument, even when the default is what you want.
2. If you sort by multiple columns, confirm that na_position is a list with one entry per sort key.
3. After any sort that feeds a report or model, assert that non_nan_values.is_monotonic_increasing (or decreasing) to catch future regressions.
4. For integer columns that contain NaN, switch to pd.Int64Dtype() before sorting to avoid float precision surprises.
5. Run df[sort_col].isna().sum() immediately after merges, before any sort, so you know exactly how many NaN rows will be repositioned.