ESP32 Arduino Tutorial: HTTP/2 POST request with headers

In this esp32 tutorial we will check how to send a HTTP/2 POST request with headers defined in our program, using the ESP32 and the Arduino core. The tests were performed using a DFRobot’s ESP32 module integrated in a ESP32 development board.

Introduction

In this tutorial we will check how to send a HTTP/2 POST request with headers defined in our program, using the ESP32 and the Arduino core.

We will be using the sh2lib wrapper from IDF. This wrapper operates on top of the NGHTTP2 library and offers a higher level API to simplify our code. You can check here a detailed video tutorial on how to install it.

Note that we have already covered how to do a simple HTTP/2 POST request with the ESP32 on this previous tutorial. Nonetheless, the functions used on the mentioned tutorial did not allow us to setup additional headers that we might need to include in our request. So, in this tutorial, we will check how to do  it.

We will send the POST request to a HTTP endpoint that will return back to us a JSON structure which includes the body we previously sent.

Note that the endpoint that we are going to reach will only echo back to us the body content of the request if we set the Content-Length header properly. Thus, this will be the header that we are going to add, in addition to the HTTP/2 pseudo headers that must be part of the request.

The tests were performed using a DFRobot’s ESP32 module integrated in a ESP32 development board.

The code

Includes and global variables

As usual, we will start our code by including the WiFi.h library, so we can connect the device to a WiFi network, and the sh2lib.h library, which will allow us to perform the HTTP/2 POST request.

#include "WiFi.h"
 
extern "C"{
#include "sh2lib.h"
}

We will also need the WiFi network credentials (the network name and password), so we can connect to it. We will store both in two global variables, so they are easy to change.

const char* ssid = "yourNetworkName";
const char* password = "yourPassword";

We will also declare a Boolean variable that will be used to signal when the request is finished, like we were doing in previous tutorials. Thus, we will initialize it with the value false and later set it to true when the request is finished.

bool request_finished = false;

To finalize the global variables declaration, we will declare a string with the content that will be posted to the server. It will be a simple “Hello World” message, as can be seen below.

const char * toSend = "hello world in HTTP/2!";

The setup

As usual, we will start the setup function by initializing the serial interface, so we can output the results of our program. After that, we will connect the ESP32 to the WiFi network, using the credentials declared as global variables.

To complete the setup function, we will launch the FreeRTOS task that will be responsible for handling the HTTP/2 related functionalities. Note that we are keeping the same configuration parameters for the FreeRTOS task that are used in IDF’s original HTTP/2 example, which I encourage you to check.

The full setup function can be seen below.

void setup() {
  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);
 
}

The HTTP/2 FreeRTOS task

We will start our task function by declaring a struct of type sh2lib_handle, which will be used in the sh2lib function calls we will make.

After that, we will connect to the server by calling the sh2lib_connect function, which receives as first input the address of our sh2lib_handle variable and as second input the URL of the server.

struct sh2lib_handle hd;
 
if (sh2lib_connect(&hd, "https://nghttp2.org") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
}
 
Serial.println("Connected");

As mentioned in the introductory section, we will set the Content-Length header in our request, which means we need to obtain the length of the string that we are going to send to the server as the request body data.

We can obtain the length of the toSend string we have previously declared using the strlen function. This function receives as input the string and returns its length, as an integer.

Since, as we will see below, the headers are defined as name-value pairs in string format, we first print the content length to a string by using the sprintf function and the %d format specifier. You can read more about format specifiers here.

Naturally, we will need to declare a char buffer that will be used as destination buffer by the sprintf function, to print the string with the data length.

char len[3];
 
sprintf(len, "%d", strlen(toSend));

As already covered on this post, we will define each header as a name-value pair.

Nonetheless, the NGHTTP2 library uses a data structure called nghttp2_nv that is slightly more complex and has some additional fields other than the name of the header and the value of the header.

Fortunately, the sh2lib wrapper has a macro called SH2LIB_MAKE_NV which simply receives the name and the value of the header and builds the corresponding nghttp2_nv struct for us. So, we will leverage this macro to create our headers.

Since we want to define multiple headers, we will need an array of variables of the mentioned nghttp2_nv struct.

As already indicated, we will set the Content-Length header. Additionally, we will need to set the following HTTP/2 pseudo headers, which must be part of the request:

:method – The request HTTP method, which is “POST“;
:path – The request relative path, which is “/httpbin/post“;
:scheme – The request scheme, which is “https“;
:authority – The authority portion of the target URL, which corresponds to the host name. This value can be obtained from the hostname field of the sh2lib_handle struct.

const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "POST"),
                           SH2LIB_MAKE_NV(":scheme", "https"),
                           SH2LIB_MAKE_NV(":authority", hd.hostname),
                           SH2LIB_MAKE_NV(":path", "/httpbin/post"),
                           SH2LIB_MAKE_NV("Content-Length", len)
                          };

sh2lib_do_putpost_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), send_post_data, handle_post_response);

As mentioned, the previous function call only does the setup of the request. Now, we need to call the sh2lib_execute function periodically to execute the actual exchange of data with the server.

We will do it in an infinite loop that will break when the server response handling function callback sets the request_finished flag to true, like we have been doing in previous tutorials.

After the loop breaks, it means that the request is finished. So, we can disconnect from the server and delete the FreeRTOS task, which will no longer be needed.

The full FreeRTOS task function code can be seen below.

void http2_task(void *args)
{
  struct sh2lib_handle hd;
 
  if (sh2lib_connect(&hd, "https://nghttp2.org") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
  }
 
  Serial.println("Connected");
 
  char len[3];
  sprintf(len, "%d", strlen(toSend));
 
  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "POST"),
                             SH2LIB_MAKE_NV(":scheme", "https"),
                             SH2LIB_MAKE_NV(":authority", hd.hostname),
                             SH2LIB_MAKE_NV(":path", "/httpbin/post"),
                             SH2LIB_MAKE_NV("Content-Length", len)
                            };
 
  sh2lib_do_putpost_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), send_post_data, handle_post_response);
 
  while (1) {
 
    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in execute");
      break;
    }
 
    if (request_finished) {
      break;
    }
 
    vTaskDelay(10);
  }
 
  sh2lib_free(&hd);
  Serial.println("Disconnected");
 
  vTaskDelete(NULL);
}

The send callback function

The send callback function implementation will be similar to what we have already covered in this previous tutorial, where we have seen how to do a simple HTTP/2 POST request.

Basically, this callback function needs to follow a predefined signature where the second parameter corresponds to a data buffer to which we should copy the content to be sent to the server.

Nonetheless, there’s a maximum length of the data that we can copy to that buffer when the callback function is invoked. This maximum length is passed as third parameter of the callback function.

So, before we copy the content to the buffer, we need to perform a check to confirm that the length of the data we are copying is lesser than the maximum length allowed.

Note that, since we are only going to send a very small string, this callback function shouldn’t be called multiple times by NGHTTP2. So, after copying the content to the send buffer, we need to specify that there’s no more data to copy.

We signal this by using the fourth parameter of the callback function, which corresponds to a pointer to some data flags. When we want to indicate that there is no more data, we should set the NGHTTP2_DATA_FLAG_EOF flag to 1.

Finally, as output of this function, we need to return the actual number of bytes copied to the send buffer. The full function code can be seen below. For a more detailed explanation of all the steps, please consult the mentioned previous tutorial.

int send_post_data(struct sh2lib_handle *handle, char *buf, size_t length, uint32_t *data_flags)
{
    int copylen =  strlen(toSend);
    if (copylen < length) {
        memcpy(buf, toSend, copylen);
    } else {
        copylen = 0;
    }
 
    (*data_flags) |= NGHTTP2_DATA_FLAG_EOF;
    return copylen;
}

The response handling function

The response handling function will also be very similar to what we have been covering in previous tutorials.

Two of the parameters that this handling function receives are a pointer to the buffer with the data sent by the server and the length of that data. So, the first thing we will do is checking if the length of the data is greater than zero and, in case it is, print it.

Additionally, we will check when the HTTP/2 stream is closed, which indicates the server finished sending the response and the request is completed.

We can confirm that by looking into the fourth parameter of the callback function and, if it is equal to DATA_RECV_RST_STREAM, then we know the stream is closed. Consequently, we will set the request_finished flag to true, thus indicating to the FreeRTOS task loop that the request was finished.

The full callback function code can be seen below.

int handle_post_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
    if (len > 0) {
        Serial.printf("%.*s\n", len, data);
    }
 
    if (flags == DATA_RECV_RST_STREAM) {
        request_finished = true;
        Serial.println("STREAM CLOSED");
    }
    return 0;
}

The final code

The final complete code can be seen below.

#include "WiFi.h"
 
extern "C"{
#include "sh2lib.h"
}
 
const char* ssid = "yourNetworkName";
const char* password =  "yourPassword";
 
bool request_finished = false;
 
const char * toSend = "hello world in HTTP/2!";
 
int handle_post_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
    if (len > 0) {
        Serial.printf("%.*s\n", len, data);
    }
 
    if (flags == DATA_RECV_RST_STREAM) {
        request_finished = true;
        Serial.println("STREAM CLOSED");
    }
    return 0;
}
 
int send_post_data(struct sh2lib_handle *handle, char *buf, size_t length, uint32_t *data_flags)
{
    int copylen =  strlen(toSend);
    if (copylen < length) {
        memcpy(buf, toSend, copylen);
    } else {
        copylen = 0;
    }
 
    (*data_flags) |= NGHTTP2_DATA_FLAG_EOF;
    return copylen;
}
 
void http2_task(void *args)
{
  struct sh2lib_handle hd;
 
  if (sh2lib_connect(&hd, "https://nghttp2.org") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
  }
 
  Serial.println("Connected");
 
  char len[3];
  sprintf(len, "%d", strlen(toSend));
 
  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "POST"),
                             SH2LIB_MAKE_NV(":scheme", "https"),
                             SH2LIB_MAKE_NV(":authority", hd.hostname),
                             SH2LIB_MAKE_NV(":path", "/httpbin/post"),
                             SH2LIB_MAKE_NV("Content-Length", len)
                            };
 
  sh2lib_do_putpost_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), send_post_data, handle_post_response);
 
  while (1) {
 
    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in execute");
      break;
    }
 
    if (request_finished) {
      break;
    }
 
    vTaskDelay(10);
  }
 
  sh2lib_free(&hd);
  Serial.println("Disconnected");
 
  vTaskDelete(NULL);
}
 
void setup() {
  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);
 
}
 
void loop() {
  vTaskDelete(NULL);
}

Testing the code

To test the code, simply compile it and upload it to your ESP32. When the procedure finishes, open your Arduino IDE serial monitor.

You should see a result similar to figure 1. As can be seen, the data we have sent as body of the request is echoed back to us in the “data” field of the response JSON. Additionally, we can see the Content-Length header also being echoed back.

Note that there are some additional headers in the response that we did not setup that are probably being returned because of the existence of some proxy (you can check in more detail here the information about the Via header that we did not setup but is included in the response).