ESP32 Arduino Tutorial: 36. Decompressing Brotli encoded content

Introduction

In this tutorial we are going to check how to do a HTTP/2 GET request, accepting the content compressed with the Brotli algorithm. Thus, we will later decompress it back, after receiving the response.

Please check this previous tutorial for an explanation on how to install the HTTP/2 sh2lib wrapper we are going to need. This wrapper works on top of the NGHTTP2 library and offers some higher level abstractions to make it easier to get started using the HTTP/2 protocol.

For a tutorial on how to do a HTTP/2 GET request, please check here. For a tutorial on how to use the ESP32 to decompress content compressed with the Brotli algorithm, please check this previous tutorial.

As can be seen here, there’s a HTTP request header called Accept-Encoding that allows the client to indicate to the server which content encoding it is capable of decoding [1].

This content encoding usually corresponds to a compression algorithm and Brotli is one of the supported algorithms [1]. So, if the client can work with Brotli, then this header should take the value “br” [1].

So, when setting up our HTTP/2 GET request, we will include the Accept-Encoding header with the value “br“, so the server sends back the content encoded with the Brotli algorithm. This previous tutorial explains how to add headers to a HTTP/2 GET request, using the sh2lib library.

Naturally, not only the client but also the server must support the compression format. Otherwise, even if the Accept-Encoding header is specified by the client, the content won’t be compressed. And even if the server also supports this encoding, it may still choose to not compress the body of the response [1].

So, for testing, we will send the request to this endpoint, which supports the Brotli compression. As can be seen in figure 1, when we send a request using a web browser (in my case, Google Chrome), if the Accept-Encoding header has the “br” value included in the list of supported formats, then the server will use this compression algorithm.

Note that, as highlighted in figure 1, we can know if the server has used some encoding format by looking into the Content-Encoding response header. As shown, the server sent back that header with the value “br“, thus indicating that Brotli was the compression format used.

Figure 2 – Response to HTTP/2 GET request with Brotli compression.

You can also check from figure 1 that the expected answer, after decompression, is a JSON listing some libraries. This is what we should expect to obtain after running the code shown from the next sections on the ESP32.

The tests were performed using a DFRobot’s ESP32 module integrated in a ESP32 development board.

The code

Includes and global variables

We will start our code by the library includes. As usual, we will need the WiFi.h library, so we can connect the device to a WiFi network.

Besides that, we will need the sh2lib.h wrapper, so we can send the HTTP/2 request, and the decode.h module from the Brotli libraries, so we can decompress the content we will receive as response of our request.

Note that we need to enclose both these two last includes in an extern “C” block, for all the code to compile fine.

#include "WiFi.h"

extern "C" {
#include "sh2lib.h"
#include "decode.h"
}

We will also need to store the credentials of the WiFi network to which we are going to connect our ESP32. The needed credentials are the network name (SSID) and the password.

To finalize the global variables declaration, we will also need a Boolean flag that will be used to signalize when the request is finished. Naturally, we will initialize it to false and later, after the request is completed, we will set its value to true.

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPassword";

bool request_finished = false;

Setup function

Moving on to the Arduino setup function, the code here will be the same we have been using in previous tutorials where we were doing HTTP/2 requests.

So, we start by opening a serial connection, to later output the results of our program. After that, we will connect the ESP32 to the WiFi network, using the previously declared credentials.

Finally, we will launch a FreeRTOS task that will be responsible for all the HTTP/2 related function calls.

void setup() {
  Serial.begin(115200);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }

  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);

}

Moving on to the implementation of the FreeRTOS task function, we will start by declaring a sh2lib_handle struct, which will be used as input of the next sh2lib function calls we will execute.

After that, we can connect to the server by calling the sh2lib_connect function, passing as first input the address of our handler, and as second input the server URL.

We will also perform an error check on the value returned by this function, to ensure the connection procedure was successful before we try to make the request.

struct sh2lib_handle hd;

if (sh2lib_connect(&hd, "https://api.cdnjs.com") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
}

Serial.println("Connected");

Now, since we want to indicate to the server that we can accept the content compressed with Brotli, we will need to set the “accept-encoding” header to the value “br“.

As covered in this previous tutorial, we can specify the headers of our request by creating an array that will contain the name-value pairs representing those headers. Additionally to the “accept-encoding” header, we will also need to specify the HTTP/2 pseudo headers, which were also explained in detail on the mentioned tutorial.

const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),
                           SH2LIB_MAKE_NV(":scheme", "https"),
                           SH2LIB_MAKE_NV(":authority", hd.hostname),
                           SH2LIB_MAKE_NV(":path", "/libraries/adapterjs"),
                           SH2LIB_MAKE_NV("accept-encoding", "br")
                          };

Then we will use the sh2lib_do_get_with_nv function to setup the request. This function receives as first input the address of our handle, as second the name-value array with the headers, as third the length of the name-value array and as fourth and final argument a callback function that will be invoked to handle the request response.

sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

Since the previous function call only takes care of the request setup, we still need to call the sh2lib_execute periodically to execute the actual exchange of data with the server. This function simply receives as input the address of our handle.

As we did in previous tutorials, we will invoke this function periodically inside an infinite loop with a small delay between each iteration.

The loop will break when the request is finished, which will be signaled by the response handling function by setting the request_finished variable to true.

while (1) {

    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in send/receive");
      break;
    }

    if (request_finished) {
      break;
    }

    vTaskDelay(10);
}

When the loop breaks, it means that the request is finished. So, after that, we will simply close the connection to the server by calling the sh2lib_free function and then we will delete the FreeRTOS task, which is no longer needed.

sh2lib_free(&hd);
Serial.println("Disconnected");

vTaskDelete(NULL);

The full FreeRTOS function implementation can be checked below.

void http2_task(void *args)
{
  struct sh2lib_handle hd;

  if (sh2lib_connect(&hd, "https://api.cdnjs.com") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
  }

  Serial.println("Connected");

  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),
                             SH2LIB_MAKE_NV(":scheme", "https"),
                             SH2LIB_MAKE_NV(":authority", hd.hostname),
                             SH2LIB_MAKE_NV(":path", "/libraries/adapterjs"),
                             SH2LIB_MAKE_NV("accept-encoding", "br")
                           };

  sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

  while (1) {

    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in send/receive");
      break;
    }

    if (request_finished) {
      break;
    }

    vTaskDelay(10);
  }

  sh2lib_free(&hd);
  Serial.println("Disconnected");

  vTaskDelete(NULL);
}

The response handling function

To finalize our code, we need to write the response handling function. As we have seen in previous tutorials, the signature of this function has the following signature:

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags){
// handing function implementation
}

We can check if we have received data if the third argument (the length of the data) is greater than zero.

if (len > 0) {
// Handle received data
}

As seen before, if we receive data, we expect that it will be compressed with the Brotli algorithm. Note that, for simplicity, we are not checking the response headers to confirm this was the used encoding, but you should do it in a real application scenario.

So, the first thing we will do is declaring a byte buffer to store the decompressed data. We are going to declare an array that will be big enough to hold all the data that will be returned by the endpoint we will contact.

uint8_t buffer [4000];

As covered on this previous tutorial, the function responsible for decompressing the content also receives the address of variable of type size_t, which will be used as an in and out parameter.

So, that variable should initially hold the size of the output data buffer (which we can obtain by using the sizeof operator) and then the decompress function will set that variable with the length of the actual decoded data.

In our case, the length of the decompressed content will be lesser than the total length the output buffer can hold, since we have declared an array that is bigger than what is actually needed. In a real case scenario, you may not know the actual size of the decompressed content (it may vary), so it makes sense to declare an array with the maximum length it can have.

size_t output_length = sizeof(buffer);

Then we will call the BrotliDecoderDecompress function, which will be responsible for doing the actual decompression.

As first input, this function receives the length of the compressed data. In our case, it corresponds to the third parameter that is passed to the HTTP/2 response handling function.

The second input corresponds to the buffer that holds the compressed data. In our case, it is the second argument of the response handling function.

Note that this argument is of type const char * and the decompress function argument is of type const uint8_t *, so we need to perform a cast when passing the variable.

As third argument, the function receives the address of the size_t variable we have declared before and initialized with the length of the output buffer.

As fourth and final argument, the BrotliDecoderDecompress function receives the output buffer, where it will write the decompressed content.

BrotliDecoderDecompress(
   len,
   (const uint8_t *)data,
   &output_length,
   buffer);

Then, after the decompression procedure is applied, we will print the resulting plain text data. We will do it using the printf function of the serial object and the %.*s format specifier, so we can directly use the output buffer and print it as a string.

Serial.printf("%.*s\n", output_length, buffer);

The final callback function can be seen below. It also contains the handling of the stream closed event, which will set the request_finished variable to true, thus signaling the FreeRTOS function loop that the request is finished.

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
  if (len > 0) {

    uint8_t buffer [4000];

    size_t output_length = sizeof(buffer);

    BrotliDecoderDecompress(
      len,
      (const uint8_t *)data,
      &output_length,
      buffer);

    Serial.printf("%.*s\n", output_length, buffer);
  }

  if (flags == DATA_RECV_RST_STREAM) {
    request_finished = true;
    Serial.println("STREAM CLOSED");
  }
  return 0;
}

The final code

The final code can be seen below.

#include "WiFi.h"

extern "C" {
#include "sh2lib.h"
#include "decode.h"
}

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPassword";

bool request_finished = false;

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
  if (len > 0) {

    uint8_t buffer [4000];

    size_t output_length = sizeof(buffer);

    BrotliDecoderDecompress(
      len,
      (const uint8_t *)data,
      &output_length,
      buffer);

    Serial.printf("%.*s\n", output_length, buffer);
  }

  if (flags == DATA_RECV_RST_STREAM) {
    request_finished = true;
    Serial.println("STREAM CLOSED");
  }
  return 0;
}

void http2_task(void *args)
{
  struct sh2lib_handle hd;

  if (sh2lib_connect(&hd, "https://api.cdnjs.com") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
  }

  Serial.println("Connected");

  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),
                             SH2LIB_MAKE_NV(":scheme", "https"),
                             SH2LIB_MAKE_NV(":authority", hd.hostname),
                             SH2LIB_MAKE_NV(":path", "/libraries/adapterjs"),
                             SH2LIB_MAKE_NV("accept-encoding", "br")
                           };

  sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

  while (1) {

    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in send/receive");
      break;
    }

    if (request_finished) {
      break;
    }

    vTaskDelay(10);
  }

  sh2lib_free(&hd);
  Serial.println("Disconnected");

  vTaskDelete(NULL);
}

void setup() {
  Serial.begin(115200);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }

  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);

}

void loop() {
  vTaskDelete(NULL);
}

Testing the code

To test the code, simply compile it and upload it to your ESP32, using the Arduino IDE. After the procedure finishes, open the serial monitor. You should get an output similar to figure 2, which shows the decompressed content.

Note that this shows just part of the response since the API sends the JSON without any newlines, but the whole response was correctly decompressed.

Figure 2 – Printing the decompressed content.