Whether you are responsible for uploading files regularly to a remote web server, or syncing configuration files among a number of machines, most of the time you’ll find yourself using the FTP protocol. But what about automating the process? Could you simply add a shell script to a Crontab job and let it take care of all the FTP login process, file transfers, and essentially has the job done for you while you’re away from your desk? If this sounds interesting, read along.

What is FTP?

FTP stands for File Transfer Protocol. The protocol specifications were written by Abhay Bhushan. In April 1971, the Request for Comments (RFC) 114 was published to describe the newly developed protocol.

How FTP Works?

Using interactive mode
OK, so before we delve in, let’s have a quick refresher on FTP commands and usage for those who are accustomed to using GUI applications like FileZilla. In its simplest form, you can connect to a remote server using FTP using one of the following ways:
1. Type ftp then type open hostname, where hostname is the IP address or the hostname of the remote server you’re connecting to. You’ll be asked to enter your username and password. An example of this is in the following figure:
2. Type ftp hostname to avoid having to type open after starting the FTP session
3. Type ftp username@hostname where username is the username used to connect to the server. This will let you jump right into the password prompt.

Working with files

So now we’re connected to the remote server, what’s next? The following is a list of most common commands you’ll need in your FTP session:

Command	Usage
ls	List files on the remote server
get file	Download file
mget file*	Download multiple files. You can use wildcards (*) to download all files with specific filename pattern. For example mget *.log to will download all files with extension .log
put file	Upload file
mput file*	Upload multiple files. You can use wildcards (*) the same way they’re used with mget
cd	Navigate to a path on the remote server. Example: cd /tmp/uploads
lcd	Navigate to a path on the local server (your machine). Example: lcd /home/myuser
bin	Choose binary mode for file transfer
ascii	Choose ascii mode for file transfer
bye	Exits the current session

Tip: You can always execute a command on your local machine while in an FTP session with the remote server by prefixing it with an exclamation mark (!). Example: !ls will list the files and directories on your local machine’s current path.

Automating the process

So now you have a basic idea about how to connect to a remote server using FTP, and execute basic commands. Let’s see how you can automate the process.

What are we trying to do?

The rest of the article describes how to automate logging into a remote server using FTP and working with files. To accomplish this, you’ll login to the FTP server in a slightly different manner than the previous methods. We are trying to automate the following example:

The FTP command switches

In the previous example, we used a longer way to connect to FTP. First, we specified the -n switch to disable the auto-login feature (FTP expects the login user to be the current user), we manually specified the hostname using the open command, then we specified the username and password using the user command. A couple of additional important switches can be used in this task:
-i: Turns off interactive prompting during multiple file transfers. (The normal FTP behavior is to ask you before uploading or downloading files if they are multiple, a behavior you definitely don’t want in a background job)
-v: for verbose mode. This is not strictly needed, but it’s good for debugging purposes and troubleshooting.

The heredocs

We are going to use a bash feature called “here documents” or heredocs. The main use of this feature is to enable entering multiple lines of input into an interactive prompt. It starts with any string preceded by << (Example <<EOF), then you add the commands line by line, finally you end the block by adding the same string you used, but on a new line and without the << (Example EOF).
To give you a more illustrated idea about heredocs, consider this example:
The echo command may be fine if you want to output a single line of text to the screen. But what if you want to output a number of lines? You can use cat with heredocs as follows:
And the output of this would be:
The idea here is feeding the command with a block of lines and forcing it to treat them as a single argument. So how can this be useful to our FTP automation task? It’s the same concept: FTP is expecting a command as an argument (like user username password), that will be followed by a number of get, put and other FTP commands.

The script

Ok now you have all the tools you need to automate an FTP job under your belt, let’s start writing the script.
Pretty simple right?
1. You start by specifying that you are going to use bash as your shell language.
2. Assign the hostname to variable HOST.
3. Assign the username to variable USER.
4. Assign the password to variable PASSWORD.
5. Issue the FTP command in non-interactive, verbose mode, instructing the program not to use auto-login. Also you start a heredocs block to start feeding the FTP program your command block.
6. Enter the user and password.
7. Navigate to the destination directory.
8. Upload all files with extension .html to the destination directory
9. End the FTP session
10. Mark the end of the heredocs
Now save this script as ftpScript.sh for example, and add the appropriate permissions by issuing chmod 700 ftpScript.sh. Put this script in a crontab job and you’re done.

But do you see any problem in this code?

Yes, you guessed right: the password is stored in clear text! Of course you can set very strict permissions on the script so that only you have read access. But imagine you want to change the files you want to upload/download from .html to .php, and while doing so, your new colleague is sitting beside you to learn FTP automation. In such a case, your password will be accidentally visible to your colleague and to any one who happens to have your screen in his/her sight.
A not very secure way to encrypt your password
Ok let’s agree first that the following method is NOT by any means a cryptographically secure way of string encryption. This is just a way to “obfuscate” your password so that it is not visible in clear text.
Having said that, let’s get to know the base64 command. Basically, this command converts the string you supply to and from base64 format. So assuming that you have P@ssw0rd as your FTP password, you can encrypt it as follows:
As you can see the base64 representation of your password is UEBzc3cwcmQK. So how can we decrypt it to get back our text?
Now let’s make a slight modification to our script:
The changed part here is line 4. We used base64 decryption method to get the password string and assign it to the password variable. To make it work, we enclosed the command in [code]]czowOlwiXCI7e1smKiZdfQ==[[/code] to instruct bash to execute the command and not treat it as literal text.
This way, if somebody had an accidental look at your file, the base64-encoded password will be much more difficult to memorize than the clear text one.
Conclusion
In this article we had a quick overview of the FTP command and how to use it in the bash shell instead of using GUI applications. Then we explained how to use command line switches and heredocs to automate FTP file transfer operations. Finally, we showed a very basic way to encrypt clear text passwords. Although not considered “secure enough” by any security standard, it is much better than letting the password stay in clear text.
I hope I saved somebody’s time and effort through this post. Thanks for reading and see you in later posts.