Dynamic Buffers

This page explains dynamic buffers—containers that manage resizable storage with a producer/consumer model designed for streaming I/O.

Code snippets assume using namespace boost::capy; is in effect.

The Producer/Consumer Model

Dynamic buffers serve as intermediate storage between a producer that writes data and a consumer that reads it. The classic example is network I/O: the network produces data that your application consumes.

    Network (Producer)
          |
          v
    +-----------------+
    | Dynamic Buffer  |
    | [readable data] |
    +-----------------+
          |
          v
    Application (Consumer)

The buffer API reflects this model:

  • Producer side: prepare(n) → write data → commit(n)

  • Consumer side: data() → read data → consume(n)

This separation prevents reading uncommitted data or overwriting unread data.

The DynamicBuffer Concept

template<class T>
concept DynamicBuffer = requires(T& buf, T const& cbuf, std::size_t n)
{
    // Capacity management
    { cbuf.size() } -> std::convertible_to<std::size_t>;      // Readable bytes
    { cbuf.max_size() } -> std::convertible_to<std::size_t>;  // Maximum capacity
    { cbuf.capacity() } -> std::convertible_to<std::size_t>;  // Current capacity

    // Consumer side
    { cbuf.data() } -> ConstBufferSequence;   // Get readable data
    buf.consume(n);                            // Discard n bytes

    // Producer side
    { buf.prepare(n) } -> MutableBufferSequence;  // Get writable space
    buf.commit(n);                                 // Make n bytes readable
};

Producer Operations

flat_dynamic_buffer buf(storage, sizeof(storage));

// 1. Get writable space
auto writable = buf.prepare(1024);

// 2. Write data (e.g., from network)
std::size_t n = receive_data(writable);

// 3. Make written bytes readable
buf.commit(n);

Consumer Operations

// 1. Get readable data
auto readable = buf.data();

// 2. Process the data
std::size_t processed = parse_message(readable);

// 3. Discard processed bytes
buf.consume(processed);

DynamicBufferParam for Coroutines

When passing dynamic buffers to coroutine functions, use DynamicBufferParam with a forwarding reference:

auto read(ReadSource auto& source, DynamicBufferParam auto&& buffers)
    -> task<io_result<std::size_t>>;

This concept enforces safe parameter passing:

  • Lvalues: Always allowed—the caller manages the buffer’s lifetime

  • Rvalues: Only allowed for adapter types that update external storage

Why the Distinction?

Some buffer types store bookkeeping internally:

// BAD: rvalue non-adapter loses bookkeeping
flat_dynamic_buffer buf(storage, sizeof(storage));
co_await read(source, std::move(buf));  // Compile error!

// GOOD: lvalue keeps bookkeeping accessible
flat_dynamic_buffer buf(storage, sizeof(storage));
co_await read(source, buf);  // OK

Adapters update external storage, so rvalues are safe:

// GOOD: adapter rvalue, string retains data
std::string body;
co_await read(source, string_dynamic_buffer(&body));  // OK

Marking Adapter Types

Types safe to pass as rvalues define a nested tag:

class string_dynamic_buffer {
public:
    using is_dynamic_buffer_adapter = void;
    // ...
};

Provided Implementations

flat_dynamic_buffer

Uses a single contiguous memory region:

#include <boost/capy/buffers/flat_dynamic_buffer.hpp>

char storage[4096];
flat_dynamic_buffer buf(storage, sizeof(storage));

// prepare() and data() always return single-element sequences
auto writable = buf.prepare(100);  // Single mutable_buffer
auto readable = buf.data();         // Single const_buffer

Characteristics:

  • Data is always contiguous—good for parsers requiring linear access

  • Buffer sequences have exactly one element

  • May waste space after consume (gap at front)

Best for: Protocols requiring contiguous parsing, simple request/response patterns.

circular_dynamic_buffer

Uses a ring buffer that wraps around:

#include <boost/capy/buffers/circular_dynamic_buffer.hpp>

char storage[4096];
circular_dynamic_buffer buf(storage, sizeof(storage));

// Data may wrap around
auto readable = buf.data();  // May return const_buffer_pair

Characteristics:

  • No space wasted after consume (wraps around)

  • Buffer sequences may have two elements when data spans the wrap point

  • Fixed maximum capacity

Best for: Continuous streaming, bidirectional protocols, high-throughput scenarios.

string_dynamic_buffer

Adapts a std::string as a dynamic buffer:

#include <boost/capy/buffers/string_dynamic_buffer.hpp>

std::string body;
string_dynamic_buffer buf(&body);

// Buffer operations modify the string
co_await read(source, buf);
// body now contains the data

Characteristics:

  • Does not own storage—wraps an existing string

  • String grows as needed (up to max_size)

  • Destructor resizes string to final readable size

  • Move-only (cannot be copied)

  • Marked as adapter—safe to pass as rvalue

Best for: Building string results, HTTP bodies, text protocols.

vector_dynamic_buffer

Adapts a std::vector<unsigned char> as a dynamic buffer:

#include <boost/capy/buffers/vector_dynamic_buffer.hpp>

std::vector<unsigned char> data;
vector_dynamic_buffer buf(&data);

co_await read(source, buf);
// data now contains the bytes

Characteristics:

  • Similar to string_dynamic_buffer but for binary data

  • Vector grows as needed

  • Marked as adapter—safe to pass as rvalue

Best for: Binary protocols, file I/O, binary message bodies.

Real-World Examples

Reading HTTP Body

task<std::string> read_body(ReadSource auto& source, std::size_t content_length)
{
    std::string body;
    string_dynamic_buffer buf(&body, content_length);

    // Read until we have the full body
    char temp[4096];
    std::size_t remaining = content_length;
    while (remaining > 0)
    {
        auto to_read = (std::min)(remaining, sizeof(temp));
        auto [ec, n] = co_await source.read(mutable_buffer(temp, to_read));
        if (ec.failed())
            co_return {};

        auto writable = buf.prepare(n);
        buffer_copy(writable, const_buffer(temp, n));
        buf.commit(n);
        remaining -= n;
    }

    co_return body;
}

Streaming Protocol Parser

task<void> parse_stream(ReadStream auto& stream)
{
    char storage[8192];
    circular_dynamic_buffer buf(storage, sizeof(storage));

    for (;;)
    {
        // Read more data
        auto [ec, n] = co_await stream.read_some(buf.prepare(1024));
        if (ec == cond::eof)
            break;
        if (ec.failed())
            co_return;
        buf.commit(n);

        // Parse complete messages
        while (auto msg = try_parse_message(buf.data()))
        {
            handle_message(*msg);
            buf.consume(msg->size());
        }
    }
}

Read Until EOF

task<std::vector<unsigned char>> read_all(ReadSource auto& source)
{
    std::vector<unsigned char> result;
    vector_dynamic_buffer buf(&result);

    auto [ec, total] = co_await read(source, buf);
    // read() loops until EOF, growing the buffer as needed

    co_return result;
}

Choosing a Buffer Type

Type Best For Trade-off

flat_dynamic_buffer

Protocols requiring contiguous data

May waste space after consume

circular_dynamic_buffer

High-throughput streaming

Two-element sequences add complexity

string_dynamic_buffer

Building string results

Requires external string ownership

vector_dynamic_buffer

Binary data accumulation

Requires external vector ownership

Summary

Component Purpose

DynamicBuffer

Concept for resizable buffers with prepare/commit semantics

DynamicBufferParam

Safe parameter passing constraint for coroutines

flat_dynamic_buffer

Contiguous storage, single-element sequences

circular_dynamic_buffer

Ring buffer, no space waste, may have two-element sequences

string_dynamic_buffer

Adapter for std::string

vector_dynamic_buffer

Adapter for std::vector<unsigned char>

Navigation

← Buffer Algorithms | Next: Stream Concepts →

Edit this Page