tessl/pypi-parallel-ssh

Asynchronous parallel SSH client library that enables developers to execute SSH commands across many servers simultaneously with minimal system load on the client host.

—

Pending

Overview

Eval results

Files

File Transfer Operations

Name: tessl/pypi-parallel-ssh
Author: tessl

High-performance file copying between local and remote hosts using both SFTP and SCP protocols. Supports recursive directory operations, per-host file naming, and both parallel and single-host transfers.

Capabilities

SFTP File Operations

Secure file transfer protocol operations for reliable file copying with support for directory recursion and custom file naming.

def copy_file(self, local_file, remote_file, recurse=False, copy_args=None):
    """
    Copy local file to remote hosts via SFTP.

    Parameters:
    - local_file (str): Path to local file or directory
    - remote_file (str): Remote destination path
    - recurse (bool, optional): Recursively copy directories (default: False)
    - copy_args (list, optional): Per-host copy arguments

    Returns:
    list[gevent.Greenlet]: List of greenlets for copy operations (ParallelSSHClient)
    None: For SSHClient (operation completes synchronously)
    """

def copy_remote_file(self, remote_file, local_file, recurse=False,
                    suffix_separator='_', copy_args=None, encoding='utf-8'):
    """
    Copy remote files to local host via SFTP.

    Parameters:
    - remote_file (str): Remote file path
    - local_file (str): Local destination path
    - recurse (bool, optional): Recursively copy directories (default: False)
    - suffix_separator (str, optional): Separator for per-host files (default: '_')
    - copy_args (list, optional): Per-host copy arguments
    - encoding (str, optional): File encoding (default: 'utf-8')

    Returns:
    list[gevent.Greenlet]: List of greenlets for copy operations (ParallelSSHClient)
    None: For SSHClient (operation completes synchronously)
    """

Usage examples:

from pssh.clients import ParallelSSHClient, SSHClient
from gevent import joinall

# Parallel SFTP upload
hosts = ['web1.example.com', 'web2.example.com']
client = ParallelSSHClient(hosts)

# Upload single file to all hosts
greenlets = client.copy_file('/local/config.txt', '/etc/app/config.txt')
joinall(greenlets, raise_error=True)

# Upload directory recursively
greenlets = client.copy_file('/local/website/', '/var/www/html/', recurse=True)
joinall(greenlets, raise_error=True)

# Download files from all hosts (creates host-specific local files)
greenlets = client.copy_remote_file('/var/log/app.log', '/local/logs/app.log')
joinall(greenlets)
# Creates: /local/logs/app.log_web1.example.com, /local/logs/app.log_web2.example.com

# Single host SFTP
single_client = SSHClient('server.example.com')
single_client.copy_file('/local/backup.tar.gz', '/remote/backups/backup.tar.gz')
single_client.copy_remote_file('/remote/data.csv', '/local/data.csv')

SCP File Operations

Secure copy protocol operations providing the best performance for file transfers, with support for recursive directory copying.

def scp_send(self, local_file, remote_file, recurse=False, copy_args=None):
    """
    Send files to remote hosts via SCP.

    Parameters:
    - local_file (str): Path to local file or directory
    - remote_file (str): Remote destination path  
    - recurse (bool, optional): Recursively copy directories (default: False)
    - copy_args (list, optional): Per-host copy arguments

    Returns:
    list[gevent.Greenlet]: List of greenlets for copy operations (ParallelSSHClient)
    None: For SSHClient (operation completes synchronously)

    Note: 
    SCP does not overwrite existing remote files and raises SCPError instead.
    Recursive copying requires server SFTP support for directory creation.
    """

def scp_recv(self, remote_file, local_file, recurse=False, copy_args=None,
            suffix_separator='_'):
    """
    Receive files from remote hosts via SCP.

    Parameters:
    - remote_file (str): Remote file path
    - local_file (str): Local destination path
    - recurse (bool, optional): Recursively copy directories (default: False)  
    - copy_args (list, optional): Per-host copy arguments
    - suffix_separator (str, optional): Separator for per-host files (default: '_')

    Returns:
    list[gevent.Greenlet]: List of greenlets for copy operations (ParallelSSHClient)
    None: For SSHClient (operation completes synchronously)
    """

Usage examples:

# Parallel SCP upload (highest performance)
hosts = ['server1.example.com', 'server2.example.com']
client = ParallelSSHClient(hosts)

# Send large file to all hosts
greenlets = client.scp_send('/local/large_file.bin', '/remote/large_file.bin')
joinall(greenlets, raise_error=True)

# Send directory recursively
greenlets = client.scp_send('/local/app/', '/opt/app/', recurse=True)
joinall(greenlets, raise_error=True)

# Receive files from all hosts
greenlets = client.scp_recv('/var/log/system.log', '/local/logs/system.log')
joinall(greenlets)

# Single host SCP with error handling
from pssh.exceptions import SCPError

single_client = SSHClient('server.example.com')
try:
    single_client.scp_send('/local/file.txt', '/remote/file.txt')
    print("File sent successfully")
except SCPError as e:
    print(f"SCP error: {e}")

Per-Host File Operations

Customize file operations for individual hosts using copy arguments for different file names, paths, or operations per host.

# Per-host copy arguments for different destinations
hosts = ['web1.example.com', 'web2.example.com', 'db.example.com']
client = ParallelSSHClient(hosts)

copy_args = [
    {'local_file': '/local/web_config.txt', 'remote_file': '/etc/nginx/site.conf'},  # web1
    {'local_file': '/local/web_config.txt', 'remote_file': '/etc/apache2/site.conf'}, # web2
    {'local_file': '/local/db_config.txt', 'remote_file': '/etc/mysql/my.cnf'}        # db
]

greenlets = client.copy_file(copy_args=copy_args)
joinall(greenlets, raise_error=True)

# Per-host download with custom local naming
copy_args = [
    {'remote_file': '/var/log/nginx/access.log', 'local_file': '/logs/web1_access.log'},
    {'remote_file': '/var/log/apache2/access.log', 'local_file': '/logs/web2_access.log'}, 
    {'remote_file': '/var/log/mysql/slow.log', 'local_file': '/logs/db_slow.log'}
]

greenlets = client.copy_remote_file(copy_args=copy_args)
joinall(greenlets)

Error Handling

File transfer operations provide specific error handling for various failure scenarios:

from pssh.exceptions import SFTPError, SFTPIOError, SCPError
from gevent import joinall

try:
    # SFTP operations
    greenlets = client.copy_file('/nonexistent/file.txt', '/remote/file.txt')
    joinall(greenlets, raise_error=True)
except SFTPError as e:
    print(f"SFTP initialization error: {e}")
except SFTPIOError as e:
    print(f"SFTP I/O error: {e}")

try:
    # SCP operations  
    greenlets = client.scp_send('/local/file.txt', '/remote/existing_file.txt')
    joinall(greenlets, raise_error=True)
except SCPError as e:
    print(f"SCP error (file may already exist): {e}")

Protocol Comparison

SFTP vs SCP

SFTP (SSH File Transfer Protocol):

More reliable with better error handling
Supports resuming interrupted transfers
Can overwrite existing files
Slightly more overhead than SCP
Better for general-purpose file transfers

SCP (Secure Copy Protocol):

Highest performance for large file transfers
Does not overwrite existing files (raises error instead)
Requires SFTP support for recursive directory creation
Best for one-time file deployments
Minimal protocol overhead

Usage Recommendations

# Use SFTP for:
# - Regular file synchronization  
# - When files might already exist
# - When you need detailed error information
greenlets = client.copy_file('/local/config/', '/etc/app/', recurse=True)

# Use SCP for:
# - Large file transfers requiring maximum performance
# - Initial deployment to clean remote directories
# - When you want to prevent accidental overwrites
greenlets = client.scp_send('/local/release.tar.gz', '/opt/releases/release.tar.gz')

Large File Transfer Best Practices

# For large files or many files, consider:

# 1. Compress before transfer
import tarfile
with tarfile.open('/tmp/archive.tar.gz', 'w:gz') as tar:
    tar.add('/large/directory', arcname='directory')

greenlets = client.scp_send('/tmp/archive.tar.gz', '/remote/archive.tar.gz')
joinall(greenlets)

# Extract on remote hosts
output = client.run_command('cd /remote && tar -xzf archive.tar.gz')
client.join()

# 2. Use appropriate pool_size for I/O intensive operations
client = ParallelSSHClient(hosts, pool_size=50)  # Reduce for file transfers

# 3. Monitor transfer progress for very large operations
from gevent import spawn
import time

def monitor_transfers(greenlets):
    while not all(g.ready() for g in greenlets):
        completed = sum(1 for g in greenlets if g.ready())
        print(f"Transfers completed: {completed}/{len(greenlets)}")
        time.sleep(5)

greenlets = client.scp_send('/large/file.bin', '/remote/file.bin')
monitor = spawn(monitor_transfers, greenlets)
joinall(greenlets + [monitor])

Install with Tessl CLI