How to Download Files Using the Ansible get_url Module

As a Linux user, you are probably familiar with tools that allow you to download files from remote servers using HTTP, HTTPS, and FTP protocols, such as wget and cURL. When performing automation tasks, you will occasionally need to download files and packages to remote hosts. While you can use wget and cURL on the remote hosts to perform this function, you can also bypass the required installation to use these tools using the Ansible get_url module.

This tutorial shows you how to use the Ansible get_url command to download files from remote servers.

Before we begin, the Ansible get_url command requires the following prerequisites:

1. The remote server from which the file(s) are to be downloaded should have direct access to the host server.
2. The host server should support both HTTP and HTTPS protocols.

About the get_url Module

Before you learn about how to implement the get_url module, first, there are a few things worth understanding about this module. The get_url module supports the following features:

• Checksum download and validation from server
• HTTP, HTTPS, and FTP servers
• Proxy servers, which can be specified by the _proxy directive
• Setting a timeout for get requests
• Web crawling
• Basic web authorization

Next, we will show you how to use the get_url module to download files with various protocols and configurations.

Using the get_url Module to Obtain Files

The following are some examples that you can implement using the get_url module when downloading files from a remote server.

Download Files from HTTP/HTTPS Server with Direct URL

Consider the following playbook that creates a directory in the ~/.local and uses the get_url module to download the Debian MySQL package.

NOTE: You can find MYSQL server packages in the resource provided below:

https://linkfy.to/mysql-packages

- hosts: all
- name: Download Debian MySQL server using HTTP/HTTPS
tasks:
- name: Make directory mysql-server in /home/user/.local
file:
path: ~/.local/mysql-server
state: directory
mode: 0777
- name: GET MySQL-server packages
get_url:
url: “https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar”
dest: ~/.local/mysql-server
mode: 0777
checksum: md5:5568e206a187a3b658392520540f556e

The above playbook starts by creating a directory in $HOME/.local/mysql-server, where the downloaded package should be stored. If you are creating a directory in a privileged directory, remember to use the 'become' directive.

Next, the playbook calls the get_url module and specifies the URL from which to download the package, followed by the destination directory to which to store the file. The final section of the playbook specifies the md5 checksum to check the validity of the file.

NOTE: In the above playbook, we hardcoded the checksum, but you can specify the URL at which the checksum is hosted.

After you run the playbook, you will obtain an output indicating the success or failure of the tasks.

$ ansible-playbook download_mysql_server.yml PLAY [Download Debian MySQL server using HTTP/HTTPS] ************************************************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [35.222.210.12]
TASK [Make directory mysql-server in /home/user/.local] ***************************************************************************************************************************************************************
35.222.210.12 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Once the process is completed, you can log into the host and verify whether the file exists.

Download Files with sha256 Checksum

You can also download files and verify them using the sha256 checksum, as shown in the example playbook below:

- hosts: all
- name: Download Debian MySQL server using HTTP/HTTPS
tasks:
- name: Make directory mysql-server in /home/user/.local
file:
path: ~/.local/mysql-server
state: directory
mode: 0777
- name: GET MySQL-server packages
get_url:
url: https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar
dest: ~/.local/mysql-server
mode: 0777
checksum: sha256:b5bb9d8014a0f9b1d61e21e796d78eefdf1352f23cd32812f4850b878ae4944c

To use an URL instead of hardcoding the checksum, consider the following example:

- hosts: all
- name: Download Debian MySQL server using HTTP/HTTPS
tasks:
- name: Make directory mysql-server in /home/user/.local
file:
path: ~/.local/mysql-server
state: directory
mode: 0777
- name: GET MySQL-server packages
get_url:
url: https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar
dest: ~/.local/mysql-server
mode: 0777
checksum: sha256:https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar.sha265

Download Files with Timeout on Failure to Respond

In some cases, you may have a specified resource URL that can take longer to respond or is unavailable. This can cause the connection to close before the server responds, as the default timeout is usually 10 seconds. To explicitly specify the timeout value, use the timeout: directive.

Consider the following playbook:

- hosts: all
- name: Download Debian MySQL server with timeout
tasks:
- name: Make directory mysql-server in /home/user/.local
file:
path: ~/.local/mysql-server
state: directory
mode: 0777
- name: GET MySQL-server packages
get_url:
url: https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar
dest: ~/.local/mysql-server
mode: 0777
checksum: sha256:https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar.sha265
timeout: 30

The above playbook specifies the timeout as 30 seconds, and the connection will be dropped if the server does not respond within the set time.

Download Files with Authentication

To download a file to a server that requires authentication, you will need to specify the login values.

Consider the following playbook:

- hosts: all
- name: Download File with authentication
become: yes
get_url:
url: http://102.15.192.120/backups/database.tar.gz
dest: /backups
username: user
password: 'pass'
mode: 0777
timeout: 5

Download Files from Local File Path

To download a file from a local file path, you can use the file:// URI scheme, followed by the path to the file.

Consider the following playbook:

- hosts: webservers
- name: Download File from local file path
become: yes
get_url:
url: file:///backups/secure/config.tar.gz
dest: /dev/null

Download FTP Files

Downloading FTP files is very similar to the process described in the previous section. All you need to do is specify ftp:// as the protocol to the server.

To download a secure file, you must also add the login information, as shown previously.

Consider the following playbook:

- hosts: all
tasks:
- name: Download file from FTP server
become: yes
get_url:
url: ftp://192.168.11.101
dest: /backups
mode: 0777

Ansible get_url Module Options

The get_url module also supports various options that you can use to specify and monitor file download and management. The options available with the get_url module include the following:

• Backup: The backup option, a Boolean of yes and no, allows you to specify whether to create a backup copy of the file to be downloaded.
• Group: The group option specifies the group that has ownership of the downloaded file; this option is like the chown command in Unix.
• Headers: The headers option is used to specify custom HTTP headers in a dictionary in hash format.
• http_agent: The http_agent option specifies the HTTP agent.
• Owner: The owner option specifies the owner of the download files.
• Seuser: The seuser option sets the user in the SELinux file context.
• use_proxy: The use_proxy option sets whether a proxy should be used. If this option is set to false, all proxies will be ignored, even if you have one specified in the target host.

Conclusion

This article discussed the get_url module in Ansible in detail and showed you how to use this module to download files from various sources. You can use this article to reference how to download files in Ansible using the get_url module.