Fixing Python Paramiko SFTP Uploads That Silently Fail on Large Files

You run a Paramiko SFTP upload, get no exception, and assume everything is fine. Then you check the remote server and the file is 47 MB instead of 340 MB, or it's zero bytes, or it's there but corrupt. No stack trace, no warning — just a broken file.

Silent failures like this are frustrating because the normal debugging reflex (look at the exception) gives you nothing. The problem usually lives in connection timeouts, socket buffering, or the way you're calling Paramiko's transfer methods. All of them are fixable once you know where to look.

What you'll learn

• Why Paramiko SFTP uploads silently truncate or drop large files
• How to add a progress callback to catch mid-transfer failures
• The right way to chunk large files so the connection stays alive
• How to verify the upload completed successfully on the remote end
• Common configuration mistakes and how to avoid them

Prerequisites

This guide assumes you're using paramiko 2.x or later and Python 3.8+. You should already have an SFTP server you can connect to and a basic working SSHClient setup. Install Paramiko with pip install paramiko if you haven't already.

Why Large Uploads Fail Without Raising an Exception

Paramiko's SFTPClient.put() method writes the file in internal chunks and manages the SSH channel internally. When a TCP connection times out mid-transfer, the SSH channel can close in a way that Paramiko doesn't translate into a Python exception on your side. The method returns normally, the local file handle closes, and your code moves on.

There are a few specific causes worth knowing about:

• Server-side idle timeouts: Many SFTP servers (and the firewalls in front of them) kill connections that haven't sent data for a defined period. Large files take time, and if Paramiko's internal buffering stalls, the connection drops.
• Keepalive not configured: By default, Paramiko doesn't send SSH keepalive packets. A 10-minute transfer over a connection that times out at 5 minutes will fail silently.
• No post-transfer size check: Paramiko's put() doesn't verify that the remote file size matches the local file size after writing. That's your job.
• Channel window size exhaustion: On very large files, the SSH channel's flow control window can fill up if the server is slow to acknowledge. This can stall the transfer and eventually cause it to be dropped.

Enabling Keepalives to Prevent Idle Disconnects

The first thing to fix is keepalives. Before you open the SFTP session, configure the underlying transport to send periodic keepalive packets.

import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(
    hostname="sftp.example.com",
    username="youruser",
    password="yourpassword",
    timeout=30
)

# Send a keepalive every 60 seconds
transport = ssh.get_transport()
transport.set_keepalive(60)

sftp = ssh.open_sftp()

The set_keepalive(60) call tells the transport to send a keepalive packet every 60 seconds of inactivity. Adjust the interval based on your server's idle timeout. If the server drops idle connections after 5 minutes, use an interval of 120 seconds to be safe.

Adding a Progress Callback

Paramiko's put() method accepts a callback parameter. The callback receives two arguments: bytes transferred so far and total bytes. This gives you real-time visibility into whether the transfer is actually progressing.

import os
import sys

def make_progress_callback(filename):
    def progress_callback(bytes_transferred, total_bytes):
        pct = (bytes_transferred / total_bytes) * 100
        sys.stdout.write(f"\r{filename}: {bytes_transferred}/{total_bytes} bytes ({pct:.1f}%)")
        sys.stdout.flush()
        if bytes_transferred == total_bytes:
            print()  # newline after completion
    return progress_callback

local_path = "/path/to/local/largefile.tar.gz"
remote_path = "/uploads/largefile.tar.gz"

sftp.put(
    local_path,
    remote_path,
    callback=make_progress_callback("largefile.tar.gz")
)

This won't prevent failures, but it lets you see exactly where a transfer stalls. If your progress output freezes at 60% and the method eventually returns without finishing, you've confirmed the transfer dropped mid-stream rather than completing.

Verifying the Upload Completed Successfully

Never trust that put() returning without an exception means the transfer succeeded. Always check the remote file size against the local file size immediately after the call.

import os

def verified_put(sftp, local_path, remote_path):
    local_size = os.path.getsize(local_path)
    sftp.put(local_path, remote_path)
    remote_attrs = sftp.stat(remote_path)
    remote_size = remote_attrs.st_size

    if remote_size != local_size:
        raise RuntimeError(
            f"Upload size mismatch: local={local_size} bytes, "
            f"remote={remote_size} bytes. Transfer may be incomplete."
        )
    print(f"Upload verified: {remote_path} ({remote_size} bytes)")

verified_put(sftp, "/path/to/largefile.tar.gz", "/uploads/largefile.tar.gz")

This is a lightweight check that catches truncated uploads immediately. For mission-critical transfers, go further and compare checksums using a remote md5sum command over SSH, but size comparison catches the vast majority of silent failures.

Manually Chunking the Transfer for Reliability

If put() is still causing problems on very large files — say, files over a few gigabytes — consider writing the transfer yourself in explicit chunks. This gives you full control over error handling, retry logic, and progress tracking at each chunk boundary.

import paramiko
import os

CHUNK_SIZE = 32 * 1024 * 1024  # 32 MB per chunk

def chunked_upload(sftp, local_path, remote_path, chunk_size=CHUNK_SIZE):
    local_size = os.path.getsize(local_path)
    bytes_written = 0

    with open(local_path, "rb") as local_file:
        with sftp.open(remote_path, "wb") as remote_file:
            remote_file.set_pipelined(True)  # improves throughput
            while True:
                chunk = local_file.read(chunk_size)
                if not chunk:
                    break
                remote_file.write(chunk)
                bytes_written += len(chunk)
                pct = (bytes_written / local_size) * 100
                print(f"\rProgress: {bytes_written}/{local_size} ({pct:.1f}%)", end="")
    print()

    # Verify after upload
    remote_size = sftp.stat(remote_path).st_size
    if remote_size != local_size:
        raise RuntimeError(
            f"Chunked upload incomplete: expected {local_size}, got {remote_size}"
        )
    print(f"Upload complete and verified: {remote_path}")

The set_pipelined(True) call on the remote file handle tells Paramiko to pipeline write requests rather than waiting for an acknowledgment after each chunk. This significantly improves throughput over high-latency connections.

Adding Retry Logic for Transient Failures

Network transfers fail for transient reasons: a brief routing blip, a temporary server overload, a NAT device timing out. Rather than letting a single failure kill your process, wrap the upload in a retry loop with exponential backoff.

import time
import paramiko

def upload_with_retry(ssh_params, local_path, remote_path, max_attempts=3):
    for attempt in range(1, max_attempts + 1):
        try:
            ssh = paramiko.SSHClient()
            ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            ssh.connect(**ssh_params, timeout=30)
            transport = ssh.get_transport()
            transport.set_keepalive(60)
            sftp = ssh.open_sftp()
            chunked_upload(sftp, local_path, remote_path)
            sftp.close()
            ssh.close()
            return  # success
        except (RuntimeError, paramiko.SSHException, OSError) as exc:
            print(f"Attempt {attempt} failed: {exc}")
            try:
                sftp.close()
                ssh.close()
            except Exception:
                pass
            if attempt < max_attempts:
                wait = 2 ** attempt
                print(f"Retrying in {wait}s...")
                time.sleep(wait)
            else:
                raise RuntimeError(
                    f"Upload failed after {max_attempts} attempts: {local_path}"
                ) from exc

ssh_params = {
    "hostname": "sftp.example.com",
    "username": "youruser",
    "password": "yourpassword",
}
upload_with_retry(ssh_params, "/data/bigfile.tar.gz", "/uploads/bigfile.tar.gz")

Notice that we reconnect from scratch on each retry. Reusing a broken SSH connection won't work — it's better to tear everything down and start fresh.

Common Pitfalls

Not closing the remote file handle before checking size

When you use sftp.open() to write chunks, the data may be buffered until you explicitly close the remote file handle. Calling sftp.stat() on the remote path before the handle is closed can return a stale or partial size. Make sure the with sftp.open(...) as remote_file: block exits fully before you verify.

Using AutoAddPolicy in production

paramiko.AutoAddPolicy() accepts any host key without verification, which exposes you to man-in-the-middle attacks. In a production environment, load the known host key explicitly or use RejectPolicy with a pre-populated known hosts file. Using AutoAddPolicy in examples is common, but don't carry that habit into production code.

Assuming timeouts apply to the whole transfer

The timeout parameter in ssh.connect() applies to the initial connection handshake, not to data transfers. A successful connection can still stall and drop mid-transfer due to socket-level timeouts set elsewhere. Configure keepalives and handle socket.timeout exceptions separately if you need transfer-level timeout enforcement.

Treating file size as a complete integrity check

Size matching is a necessary check, not a sufficient one. Two files can be the same size with different contents if chunks were written in the wrong order or if corruption occurred at the byte level. For archival or security-sensitive transfers, compare MD5 or SHA-256 checksums between local and remote. You can run md5sum on the remote side via ssh.exec_command() and compare it to a locally computed hash.

Wrapping Up

Silent SFTP failures are fixable with a small amount of defensive code that you should add to every transfer anyway. Here are the concrete steps to take right now:

1. Enable keepalives on the transport with transport.set_keepalive(60) before opening the SFTP session.
2. Add a size verification step immediately after every put() call using sftp.stat().
3. Switch to manual chunked writes for files larger than a few hundred megabytes, and call set_pipelined(True) on the remote file handle.
4. Wrap uploads in a retry loop that reconnects from scratch on failure and uses exponential backoff.
5. Replace AutoAddPolicy with proper host key verification before deploying to production.

Once you have size verification and retry logic in place, silent failures become loud, catchable exceptions — which is exactly where you want them.