Jannah Theme License is not validated, Go to the theme options page to validate the license, You need a single license for each domain name.
DevOpsLinux

How to Master the Linux Curl Command: Tips and Tricks?

Introduction

With one of the numerous accessible protocols, for example, HTTP, HTTPS, FTP, FTP, and SFTP, users can send and receive data with servers employing Curl, a freely available open-source program. You can utilize Curl to create scripts for automating repetitive processes, for instance, because it is a command line tool. Curl command in Linux has a wide range of use cases it can handle. 

Tools that support the required network protocols are required when sending data from or to a server. The most well-known programs for this kind of work on Linux are curl and wget.

You will learn how to execute the Linux curl command in the following sections and get a complete rundown of all the options.

What is the Curl Command?

What is the Curl Command?

Using several network protocols, data can get transferred using the command curl in Linux(which stands for “Client URL”). By indicating the appropriate URL along with the data that needs to be transmitted or received, it can communicate with a web and application server.

Libcurl, a mobile client-side URL transfer library, powers curl. It can be used in a script or straight on the command line. Curl is frequently used in the following situations:

  • getting files off the internet
  • testing at the endpoint
  • Detecting and fixing errors

curl Syntax

Following is the fundamental curl syntax:

curl [options/URLs]

Examples include

  • curl https://www.gnu.org/gnu/gnu.html

When the curl command in Linux gets used, the system produces the HTML data retrieved at the supplied URL.

The file can get downloaded to your local machine using curl when you provide a URL that points to it:

curl [url] > [local-file]

The download activity bar displays the number of files that have been downloaded thus far.

The protocol determines the syntax of URLs used as part of commands. When writing multiple URLs simultaneously that change in one place, braces get used:

  • http://example.{first, second, third}.com

With brackets, the alphabetic series get written:

  • ftp://ftp.url.com/file[1-100].txt

Because multiple patterns are permitted but nested sequences are not,

  • http://url.com/archive[2010-2020]/vol[1-4]/part{a,b,c}.html

the curl protocol

For data transport, curl enables a wide range of protocols. The whole list is provided below.

Protocol       Description
DICT a network protocol for dictionaries that allows users to ask the servers about word definitions.
FILEa URL pattern that curl can use to get a file from a local file system.
FTP, FTPS File transport Protocol, which is used to move files across clients and servers. A second SSL/TLS security surface gets added to FTPS, a variation of the same protocol.
GOPHER, GOPHERSA prior version of HTTP that was a legacy protocol for finding, distributing, as well as searching online documents. Using an additional SSL/TLS security layer, GOPHERS is a variation of the same protocol.
HTTP, HTTPS Transferring data across the internet and the web requires the Hypertext Transfer Protocol. A second SSL/TLS security element gets added to HTTP in HTTPS, the protocol’s version.
IMAP, IMAPS Accessing and managing email get done using the Internet Message Access Protocol. With an additional SSL/TLS security level, the protocol is now known as IMAPS.
LDAP, LDAPS For the maintenance and access of dispersed directories, there is a protocol called Lightweight Directory Access Protocol. Adding an extra SSL/TLS security layer, the protocol is now known as LDAPS.
MQTT Message Queuing Telemetry Transport is a protocol used to transfer data between tiny gadgets, typically Internet of Things platforms.
POP3, POP3S Email retrieval from the server gets done via the Post Office Protocol version 3 protocol. Incorporating an additional SSL/TLS security layer, POP3S is a variation of the same protocol.
RTMP Real-Time Messaging Protocol is a streaming platform for data, including audio and video.
RTSP Management of streaming media servers gets done through the Real Time Streaming Protocol.
SCPA mechanism for replicating files across an SSH server get called Secure Copy.
SFTPSSH File Transfer Protocol is an SSH connection-based variant of the File Transfer Protocol.
SMB, SMBSThe Server Message Block protocol controls the shared usage of files and computer devices. Having an additional SSL/TLS security layer, the protocol is now known as SMBS.
SMTP, SMPTS A mechanism for sending emails that makes it simple get called the Simple Mail Transfer mechanism. With an additional SSL/TLS security layer, SMTPS is a variation of the same protocol.
TELNETis a bidirectional, interactive, text-based application layer protocol.
TFTPFor transferring file information to or from a remote site, utilize the trivial file transfer protocol.

For your convenience, we have compiled a collection of 15 Linux curl commands below.

1. Check out the curl version

In addition to the curl Linux version, the -V or –version parameters will also provide the supporting protocols and features for the present version.

$ curl --version
Check out the curl version

2. Save a File

Curl’s -O and -o options can be used for downloading files. While the second option lets you choose a new filename and location, the first will save the file with the same name regardless of the remote location in the present working directory.

  • $ curl -O http://yourdomain.com/yourfile.tar.gz # Save as yourfile.tar.gz
  • $ curl -o newfile.tar.gz http://yourdomain.com/yourfile.tar.gz # Save as newfile.tar.gz
Save a File using Linux Curl Command

3. Continue a Download That Got Interrupted

It is fairly simple to restart a download that got stopped for whatever reason (for instance, by pressing Ctrl + c). When -C – (dash C, space dash) gets used, curl gets instructed to pick up where it left off with the download.

$ curl -C - -O http://yourdomain.com/yourfile.tar.gz

4. Save several files

You can download info.html and about.html from http://yoursite.com or http://mysite.com simultaneously by issuing the command below.

$ curl -O http://yoursite.com/info.html -O http://mysite.com/about.html 
Save several files using Linux Curl Command

5. Access a File’s URLs

Downloading files via a collection of URLs in a file is possible when curl and xargs get used together.

$ xargs -n 1 curl -O < listurls.txt
Access a File's URLs using curl and xargs

6. Employ a proxy, either with or without authentication

If you can listen on port 8080 on proxy.yourdomain.com while being behind a proxy server, then you should.

  • $ curl -x proxy.yourdomain.com:8080 -U user: password -O http://yourdomain.com/yourfile.tar.gz

When your proxy doesn’t need authentication, you can omit the -U user: password option.

7. HTTP Headers for Query

The remote web server can communicate more details about itself together with the query itself using HTTP headers. This gives the customer information about how the request is being processed.

You can do the following to get a website’s HTTP headers:

$ curl -I www.serverwala.com
HTTP Headers for Query

Your browser’s tools for developers also have access to this information.

8. Submit an HTTP POST request with the parameters

The firstName along with lastName parameters, in addition to their respective values, will get sent to https://yourdomain.com/info.php by the command that follows.

$ curl --data "firstName=John&lastName=Doe" https://yourdomain.com/info.php

The behavior of a typical HTML form can be replicated using this advice.

9. Using authentication or not, download files via an FTP server

The command that follows will download your file.tar.gz in the present working directory when a remote FTP server anticipates connections at ftp://yourftpserver.

$ curl -u username:password -O ftp://yourftpserver/yourfile.tar.gz

When the FTP server permits anonymous logins, you can omit the -u username:password command.

Also Read: Learn to use SCP Command in Linux

10. Is authentication required or not while uploading files to an FTP server

To use curl to upload a local file with the name mylocalfile.tar.gz to ftp://yourftpserver, follow these steps:

$ curl -u username:password -T mylocalfile.tar.gz ftp://yourftpserver

11. Indicate the User Agent

The data supplied with an HTTP request includes the user agent. This identifies the browser that the client employed to submit the request. The default setting for our current version of curl will be shown below, and we’ll modify it to “I am a new web browser” afterward:

$ curl -I http://localhost –user-agent “I am a new web browser”

12. Keep cookies from websites

If you visit https://www.cnn.com, do you need to know which cookies get downloaded to your computer? To store them in cnncookies.txt, execute the following command. After that, you can view the file using the cat command.

$ curl --cookie-jar cnncookies.txt https://www.cnn.com/index.html -O

13. Transmit Website Cookies

The cookies you obtained in the previous step can be used in any other requests you make to the same website.

$ curl --cookie cnncookies.txt https://www.cnn.com

14. Adjust the Name Resolution

When you’re a web developer and wish to test a local copy of yourdomain.com before publishing it, you can instruct curl to redirect http://www.yourdomain.com to your localhost as follows:

$ curl --resolve www.yourdomain.com:80:localhost http://www.yourdomain.com/

As a result, curl will get instructed to use localhost to access the website rather than DNS or the /etc/hosts file by the query to http://www.yourdomain.com.

15. Set a Download Rate Limitation

You can set a download rate restriction of 100 KB/s for curl to avoid using your Internet.

$ curl --limit-rate 100K http://yourdomain.com/yourfile.tar.gz -O

Also Read: 14 Tar Command in Linux

Conclusion

You must know how to utilize the Linux curl command and all of its options after reviewing this article. 

The fundamental application of Curl was demonstrated in this post. The most widely used HTTP methods, GET and POST, are covered in our instructions. We learned how to submit various sorts of data in a POST request, how to download a file, and how to modify a request’s header parameters. In this instance, we introduced Curl, a really important utility.

Arpit Saini

He is the Director of Cloud Operations at Serverwala and also follows a passion to break complex tech topics into practical and easy-to-understand articles. He loves to write about Web Hosting, Software, Virtualization, Cloud Computing, and much more.

Related Articles