Copying data from STDOUT to a remote computer using SFTP
To back up large database partitions on a remote machine using SFTP, I would like to use the dump command on the databases and send it directly using SFTP at the remote location.
This is useful when you need to dump large datasets if you don't have enough local disk space to create a backup file, then copy it to a remote location.
I've tried using python + paramiko which provides this functionality, but performance is much worse than using the native openssh / sftp binary to transfer files.
Does anyone have any ideas on how to do this either with a native sftp client on linux or in some library like paramiko? (but the one that works alongside the native sftp client)?
a source to share
If you have remote shell (ssh) access, you can do something like the following:
fancy-sql-dump-command --to-stdout | ssh me@remotehost "cat > my-dql-dump.sql"
Google "pipe over ssh" for more examples, eg. this example using tar .
a source to share
I would recommend sshfs , which runs over SFTP.
Some OS distributions have this package, for others that you will need to compile, for example, on RedHat Enterprise Linux 5.4+ or its clones such as CentOS:
sudo yum install fuse-devel glib-devel
sudo usermod -a -G fuse "$USER"
cd /tmp
tar xzf sshfs-fuse-2.2.tar.gz
cd sshfs-fuse-2.2
./configure
make
sudo make install
# relogin
mkdir /tmp/servername
sshfs servername:directory /tmp/servername/
ls /tmp/servername/
a source to share