We built a dedicated file storage service for logs, XML, JSON, other text files, and PDF documents.

Context

The service is a REST API. Its database keeps only the metadata of each file; the content itself is stored in folders on disk. The volume of files is large, so we decided to compress the content.

Challenge

Compression saved a lot of space, but large files became slow to retrieve: the client had to wait until the whole file was decompressed before the first byte arrived.

The question was whether to keep compressing at all.

Approach

We kept compression and changed how files are written and read.

  • Stream, do not unpack. The response is decompressed on the fly while it is sent, so the first bytes leave at once and memory use does not depend on the file size.
  • Send compressed bytes when the client accepts them. When a client accepts the encoding a file is stored in, the stored bytes go out as they are, with a Content-Encoding header. The server does no decompression at all, and less data travels over the network.
  • A faster codec. New files are written with Zstandard (zstd), which decompresses several times faster than gzip at a similar or better ratio. The codec is stored in the metadata of each file, so old and new files are served side by side.
  • Compress only what shrinks. PDF documents are already compressed inside. A file is stored compressed only when that makes it noticeably smaller; otherwise it is stored as it is, with no codec in its metadata.
HTTP
GET /files/7f3a…/content HTTP/1.1
Accept-Encoding: zstd, gzip

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Encoding: zstd

Outcome

Files stay compressed on disk. A large file starts arriving as soon as it is requested, instead of after it has been decompressed, and clients that accept compressed responses receive several times less data.