Register to get access to free programming courses with interactive exercises

Transfer-Encoding HTTP Protocol

Sometimes, the data that is passed to the server can be quite large. And moreover, we might not know their final size. For example, if you need to download an archive or stream a video.

To solve this problem, you can load the data completely into the server's ram, calculate the Content-Length and pass it. Once the content has been fully accepted by the browser, it'll display it straight away.

Transferring data in chunks

There is another solution that allows us to reliably pass data when we don't know their final size. This link leads to an example of an image that is rendered gradually as data is transferred. To do this, we use a technique for passing in small parts, called chunks, and a special Transfer-Encoding header with the value chunked.

In the standard response, we get the whole body and then process it. We can't process it in parts because then we'll be introducing unique rules of our own inside the protocol. But when transmitting chunks, we can process the response until the body is fully received.

Let's make a request to the site httpwatch.com:

telnet httpwatch.com 80

GET https://www.httpwatch.com/httpgallery/chunked/chunkedimage.aspx HTTP/1.1
Host: httpwatch.com
Connection: close

HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Trailer: X-HttpWatch-Sample
# Instead of Content-Length here's the title Transfer-Encoding
Transfer-Encoding: chunked 
Content-Type: image/jpeg; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Arr-Disable-Session-Affinity: True
Date: Fri, 10 Jul 2020 09:18:05 GMT

400 # chunk length
Some data from the first chunk
400
Data from the second chunk
400
etc
0 # the last chunk has a length of zero

Note that the headers are separated from the request body by a line break as always. The chunk's size is indicated at the beginning of each chunk. The data comes after it, a line break is inserted at the end of the chunk, then we have the next chunk and so on. By doing this, you can pass as many chunks as you want, the only limit is how long it takes before the server times out.

To finish passing, you need to pass the last chunk, which must be of zero length. Then, two line breaks follow, and the request is considered fully passed.

Message format

To separate records of block sizes (parts) from their contents, a CRLF separator is used (as a string: «\r\n»; as bytes in HEX format: 0x0D, 0x0A). The block length is the size of the block content, the CRLF separators are not considered.

Schematic representation: <block length in HEX><CRLF><block content><CRLF>

The last block is built in the same way, so it looks like this due to the lack of content: 0<CRLF><CRLF>

The standard also allows you to use only CR or only LF as a separator.


Recommended materials

  1. Link to the image transmitted using chunks (example from the video)
  2. Chunked transfer encoding / Wikipedia
  3. RFC2616 Hypertext Transfer Protocol

Hexlet Experts

Are there any more questions? Ask them in the Discussion section.

The Hexlet support team or other students will answer you.

About Hexlet learning process

For full access to the course you need a professional subscription.

A professional subscription will give you full access to all Hexlet courses, projects and lifetime access to the theory of lessons learned. You can cancel your subscription at any time.

Get access
130
courses
1000
exercises
2000+
hours of theory
3200
tests

Sign up

Programming courses for beginners and experienced developers. Start training for free

  • 130 courses, 2000+ hours of theory
  • 1000 practical tasks in a browser
  • 360 000 students
By sending this form, you agree to our Personal Policy and Service Conditions

Our graduates work in companies:

<span class="translation_missing" title="translation missing: en.web.courses.lessons.registration.bookmate">Bookmate</span>
<span class="translation_missing" title="translation missing: en.web.courses.lessons.registration.healthsamurai">Healthsamurai</span>
<span class="translation_missing" title="translation missing: en.web.courses.lessons.registration.dualboot">Dualboot</span>
<span class="translation_missing" title="translation missing: en.web.courses.lessons.registration.abbyy">Abbyy</span>
Suggested learning programs

From a novice to a developer. Get a job or your money back!

Frontend Developer icon
Profession
beginner
Development of front-end components for web applications
start anytime 10 months

Use Hexlet to the fullest extent!

  • Ask questions about the lesson
  • Test your knowledge in quizzes
  • Practice in your browser
  • Track your progress

Sign up or sign in

By sending this form, you agree to our Personal Policy and Service Conditions
Toto Image

Ask questions if you want to discuss a theory or an exercise. Hexlet Support Team and experienced community members can help find answers and solve a problem.