ESP32 Arduino Tutorial 33. HTTP/2: Adding headers to GET request

Introduction

In this tutorial we will check how to set the headers of a HTTP/2 GET request, sent by the ESP32 and using the Arduino core.

We will be using the sh2lib wrapper from IDF, which works on top of the NGHTT2 library. For an explanation on how to install the sh2lib wrapper to be used as an Arduino library, please check this previous tutorial.

In order to validate if we are setting the HTTP headers correctly, we are going to contact this endpoint, which returns as response the headers sent by the client. Figure 1 illustrates the result of contacting this endpoint from a web browser (in my case, I’m using Google Chrome).

Figure 1 – Testing headers in HTTP/2 GET request.

Note that the headers that we receive in the body of the response are not the response headers but rather the request headers echoed back to us. Naturally, the HTTP/2 response has it’s own headers, which are not shown in the image, but can also be checked in the developer’s console.

It’s also important to take in consideration that the HTTP/2 pseudo headers (the ones whose names start with colons in figure 1) are also not echoed back.

Note also that a Via and a Host headers are included in the response, but were not part of the request headers. Although it’s not indicated in the website we are contacting why these additional headers are included in the response, it may be caused by the use of some proxy, since the Via header is generally added by proxies [1].

The tests were performed using a DFRobot’s ESP32 module integrated in a ESP32 development board.

If you prefer a video version of this tutorial, please check my YouTube channel below.

The code

As usual, we start our code by the library includes. We will need the WiFi.h library, to connect the ESP32 to the WiFi network, and the sh2lib.h, which exposes the functions we need to connect to the HTTP/2 server and make the request.

In order to be able to establish the connection to a WiFi network, we need its credentials (network name and password), which we will store in two global variables, making them easy to change later.

To finalize, we will also declare a Boolean variable, which will be used as flag to indicate when the HTTP/2 GET request is finished. Naturally, we will initialize this flag with the value false, so we can later set it to true to indicate the conclusion of the request. Note that we used the same technique on this previous tutorial.

#include "WiFi.h" extern "C" { #include "sh2lib.h" } const char* ssid = "yourNetworkName"; const char* password =  "yourNetworkPassword"; bool request_finished = false;

After taking care of the includes and the global variables declaration, we will move on to the Arduino setup function. There, we will take care of opening a serial connection, to output the results of our program, and connecting the ESP32 to the WiFi network.

After that, we will create a FreeRTOS task which will take care of the HTTP/2 related function calls, like we have been doing in the previous tutorials. Note that we are keeping the task configuration parameters used in the IDF HTTP/2 example.

The full setup function can be seen below.

void setup() {  Serial.begin(115200);  WiFi.begin(ssid, password);  while (WiFi.status() != WL_CONNECTED) {    delay(1000);    Serial.println("Connecting to WiFi..");  }  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL); }

Now we will check how to implement the FreeRTOS task function. The first thing we will do is declaring a sh2lib_handle struct. This struct will hold the TLS and HTTP/2 session handles and it will be used by the sh2lib functions we will call below.

struct sh2lib_handle hd;

Next we will connect to the HTTP/2 server. We will do this by calling the sh2lib_connect function, passing as first input the address of our sh2lib_handle variable and as second the URI of the server to which we want to connect.

We will do an error check on this function call, which returns ESP_OK in case the connection is successfully established.

if (sh2lib_connect(&hd, "https://nghttp2.org") != ESP_OK) {    Serial.println("Error connecting to HTTP2 server");    vTaskDelete(NULL); }

In order to send the HTTP/2 GET request with headers defined by us, we will use the sh2lib_do_get_with_nv function, which allows us to pass as one of its inputs a data structure containing the headers.

Note that this function is called in the implementation of the sh2lib_do_get function, which we have been using in the previous tutorials. Nonetheless, the h2lib_do_get has a simpler signature, which doesn’t allow us to specify the headers of the request.

This is why, in this tutorial, we will need to use the more complex sh2lib_do_get_with_nv function.

Note that the nv that is present in the name of the sh2lib_do_get_with_nv function stands for name-value, since the headers are specified as name-value pairs in the NGHTTP2 library.

So, the sh2lib_do_get_with_nv function will have as one of its parameters an array of variables of type nghttp2_nv, which will contain the headers of the request.

Although the nghttp2_nv struct contains more fields than the name and the value, the sh2lib.h has a macro called SH2LIB_MAKE_NV that helps creating this struct.

So, this macro simply receives as input the name and the value of the header and builds the nghttp2_nv struct, meaning that we don’t need to worry about the other fields it contains.

Before we start specifying the headers we want to add to the request, we need to take in consideration that, in HTTP/2, a request is started by a client sending a HEADERS frame to open a stream [2].

This frame contains not only the regular headers that we are going to define, but it should also contain the HTTP/2 pseudo headers [2]:

• :method – The request HTTP method [3];
• :path – The request path [2];
• :scheme – The request scheme, which is usually HTTP or HTTPS [1] but, in some cases, may be different [3];
• :authority – The authority portion of the target URL, which is similar to the host header in the HTTP/1 protocol [2].

Note that the colon “:” should be included in the name of the pseudo-headers.

As mentioned before, the h2lib_do_get function doesn’t receive headers and calls the sh2lib_do_get_with_nv function in its implementation, which means that its code contains the definition of the name-value array with the pseudo headers, as shown in figure 2.

Figure 2 – Declaration of the pseudo headers in the h2lib_do_get function.

As can be seen, the :method pseudo header corresponds to “GET”, the :scheme corresponds to “https”, the :authority corresponds to the hostname of the server and the :path corresponds to the route of the server we want to contact.

So, we will include in our name-value array the previously mentioned pseudo headers and then we will add the accept-encoding header and a non-standard custom one called test-header.

These two additional headers are arbitrary and just for testing effects, so you can test with other values. Nonetheless, take in consideration that these additional headers cannot contain the colons like the pseudo headers do, or the request will be considered wrongly formatted.

const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),                           SH2LIB_MAKE_NV(":scheme", "https"),                           SH2LIB_MAKE_NV(":authority", hd.hostname),                           SH2LIB_MAKE_NV(":path", "/httpbin/headers"),                           SH2LIB_MAKE_NV("accept-encoding", "gzip"),                           SH2LIB_MAKE_NV("test-header", "test")                          };

After this, we will finally call the sh2lib_do_get_with_nv function, to setup the HTTP/2 GET request.

As first input, this function receives the address of our sh2lib_handle variable. As second input, it receives the array with name-values representing the headers.

As third argument, the function receives the number of elements in the array, which we can calculate in a more dynamic way by dividing the size of the array (in bytes) by the size of an element (also in bytes). These sizes can be obtained using the sizeof function.

As fourth and final argument, this function receives a callback function that will be called for processing the response sent by the server. We will define it below.

sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

Since this function only takes care of the setup of the request, we will need to call the sh2lib_execute function periodically for the actual exchange of data between the client and the server to occur.

This function receives as input the address of our handle and returns ESP_OK in case of success. So, we will use the return value for error checking.

Since, as mentioned, this function needs to be called periodically until the request is finished, then we will do it in an infinite loop, which will break when the request_finished flag is set to true.

Naturally, the response callback function will be responsible for setting this flag to true once the request is finished and the HTTP/2 stream is closed.

while (1) {   if (sh2lib_execute(&hd) != ESP_OK) {      Serial.println("Error in send/receive");      break;   }   if (request_finished) {      break;   }   vTaskDelay(10); }

Once the loop breaks we know that the request is finished, so we can close the connection to the server by calling the sh2lib_free function, which also receives as input the address of our handle.

Then, we can delete the FreeRTOS task with a call to the vTaskDelete function, passing as input the value NULL, to indicate the task is deleting itself.

sh2lib_free(&hd); Serial.println("Disconnected"); vTaskDelete(NULL);

The complete FreeRTOS task function code can be seen below.

void http2_task(void *args) {  struct sh2lib_handle hd;  if (sh2lib_connect(&hd, "https://nghttp2.org") != ESP_OK) {    Serial.println("Error connecting to HTTP2 server");    vTaskDelete(NULL);  }  Serial.println("Connected");  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),                             SH2LIB_MAKE_NV(":scheme", "https"),                             SH2LIB_MAKE_NV(":authority", hd.hostname),                             SH2LIB_MAKE_NV(":path", "/httpbin/headers"),                             SH2LIB_MAKE_NV("accept-encoding", "gzip"),                             SH2LIB_MAKE_NV("test-header", "test")                           };  sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);  while (1) {    if (sh2lib_execute(&hd) != ESP_OK) {      Serial.println("Error in send/receive");      break;    }    if (request_finished) {      break;    }    vTaskDelay(10);  }  sh2lib_free(&hd);  Serial.println("Disconnected");  vTaskDelete(NULL); }

The implementation of the handling function will be equal to the one we have covered here. Basically we will check if data is received and, in case it is, we will print it to the serial port.

Additionally, when DATA_RECV_RST_STREAM flag is received, indicating the stream is closed, we will set the request_finished flag to true, thus indicating the request is complete.

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags) {    if (len > 0) {        Serial.printf("%.*s\n", len, data);    }    if (flags == DATA_RECV_RST_STREAM) {        request_finished = true;        Serial.println("STREAM CLOSED");    }    return 0; }

The final code can be seen below. Since we don’t have any computation to perform in the Arduino main loop, we will simply delete the corresponding task.

#include "WiFi.h" extern "C" { #include "sh2lib.h" } const char* ssid = "yourNetworkName"; const char* password =  "yourNetworkPassword"; bool request_finished = false; int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags) {    if (len > 0) {        Serial.printf("%.*s\n", len, data);    }    if (flags == DATA_RECV_RST_STREAM) {        request_finished = true;        Serial.println("STREAM CLOSED");    }    return 0; } void http2_task(void *args) {  struct sh2lib_handle hd;  if (sh2lib_connect(&hd, "https://nghttp2.org") != ESP_OK) {    Serial.println("Error connecting to HTTP2 server");    vTaskDelete(NULL);  }  Serial.println("Connected");  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),                             SH2LIB_MAKE_NV(":scheme", "https"),                             SH2LIB_MAKE_NV(":authority", hd.hostname),                             SH2LIB_MAKE_NV(":path", "/httpbin/headers"),                             SH2LIB_MAKE_NV("accept-encoding", "gzip"),                             SH2LIB_MAKE_NV("test-header", "test")                           };  sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);  while (1) {    if (sh2lib_execute(&hd) != ESP_OK) {      Serial.println("Error in send/receive");      break;    }    if (request_finished) {      break;    }    vTaskDelay(10);  }  sh2lib_free(&hd);  Serial.println("Disconnected");  vTaskDelete(NULL); } void setup() {  Serial.begin(115200);  WiFi.begin(ssid, password);  while (WiFi.status() != WL_CONNECTED) {    delay(1000);    Serial.println("Connecting to WiFi..");  }  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL); } void loop() {  vTaskDelete(NULL); }

Testing the code

To test the code, simply compile it and upload it to your ESP32. When the procedure is finished, open the Arduino IDE serial monitor. You should have an output similar to figure 3.

As can be seen, both the accept-encoding and the test-header we have sent were echoed back as response of the request, thus confirming that those headers were correctly set.

As we have seen in the introductory section, the additional Host and Via headers are also returned back by the server even though we have not included them in the request, which may be caused by the existence of some proxy.

Figure 3 – Request headers sent back by the server.