Christian Biggins Design, Technology, SEO, General Ramblings. Something for everybody.

Using lftp to transfer a large site

I own and run a fairly large site (www.skylinesaustralia.com) and have recently decided to give up the dedicated server it has been running on and move to a managed solution with mosso.

The problem for me was that mosso do not yet have ssh access due to their network config and this meant that I would need to transfer SAU over FTP from my server. (6.5GB uncompressed database and 60GB file structure meant that this wasn’t going to be pretty).

For anybody that has dabbled in Command Line FTP transfers, its not very pleasant. FTP doesnt natively support recursive transfers (this is performed at the client level) so transferring my site non recursively would have been painful for even the most boring-job lovers. This coupled with the fact that it does not offer any form of synchronising incase the connection drops (which occurs a lot in ftp) or if we needed to transfer over several sessions. Enter lftp. This extremely cool little ssh based app is like an ftp client on steroids.

Now, these instructions are for a rhel based OS (I am on CentOS 4.5) but wouldn’t be hard to make them work on other OS’s.

So, to start, we will need lftp installed;

$ yum -y install lftp

Now, lftp has an awesome ‘mirror’ feature which does just that, mirrors one directory to another. We can also specify to only transfer newer files so we can launch the transfer as many times as we want and it will only transfer changed / new files (like rsync).

So, we launch lftp like so;

$ lftp -u username,password –e “set net:timeout 0” –e “mirror --reverse –verbose -n /local/directory remote/dir” ftp.yourftpserver.com

This does the following;

• -u specifies the username and password for the server you are connecting to
• -e allows you to ‘queue’ commands. So, instead of waiting for lftp to be ready for the next command, we can write it all in one statement and it will just run them in order.
• The first command is “set net:timeout 0” – this is just to disable the timeouts with lftp. Some of my directories contained hundreds and hundreds of subdirectories and it would timeout when reading them.
• The second command is the good one, its for mirroring two directories and its commands are;
  • -revserse just switches the order of the directories (ie local remote, not remote local)
  • -verbose just displays what lftp is doing
  • -n copies only new files
• The final argument is the remote ftp server.

This has worked flawlessly for me several times now and it is the only command line ftp method I use. The standard ftp client on *nix boxes is ridiculous.

Let me know how you go with running this.