External Buffers and Strings

If an embedder wants to expose a 2 GB archive or dozens of 400 MB .flac files to a Luau script, it has to clone the entire file in memory. In Luwu, it can use an external buffer instead.

External buffers and strings are backed by embedder-defined memory and work with the buffer and string libraries. Like regular buffers, external buffers are mutable by default. Like regular strings, external strings are always immutable.

External buffers

An external buffer is a regular buffer whose bytes live in host memory.

The whole buffer library works on it, including the native code fast paths. Wrapping host data in a userdata instead would lose all of that.

seal uses them for bytes it builds in Rust, like archive contents and websocket frames. The Rust allocation becomes the buffer instead of being copied into one.

Frozen buffers

The host decides whether scripts may write to an external buffer. A read-only one is frozen and writing to it is an error.

That makes it safe to share memory the host must not see change, like a reference-counted file. Scripts check with buffer.isfrozen:

const function patch(bytes: buffer, offset: number, value: number)
    if buffer.isfrozen(bytes) then
        error("this buffer is read-only")
    end
    buffer.writeu8(bytes, offset, value)
end

const bytes = buffer.create(4)
patch(bytes, 0, 42)
print(buffer.readu8(bytes, 0))

The C API

#define LUA_BLUAU 0
#define LUA_BHOST_IMMUTABLE 1
#define LUA_BHOST_MUTABLE 2

typedef void (*lua_BufferFree)(lua_State* L, void* data, size_t sz, void* userdata);

LUA_API void* lua_newexternalbuffer(
    lua_State* L, size_t sz, void* data, void* userdata, lua_BufferFree free_cb, int mode);
LUA_API int lua_getbuffermode(lua_State* L, int idx);
LUA_API void* lua_getbufferuserdata(lua_State* L, int idx);

From Rust, mluau has create_external_buffer (frozen) and create_external_buffer_mut.

External strings

An external string is a string whose bytes live in host memory. Scripts can't tell it apart from any other string.

It works with the string library, concatenation and string patterns. It can be a table key and compares equal to the same text made any other way.

Use one when a script expects a string, not a buffer. Error tracebacks, file contents and strings inside JSON are all strings by convention. Lua 5.5 has external strings too.

The C API

typedef void (*lua_StringFree)(lua_State* L, const char* data, size_t sz, void* userdata);

LUA_API const char* lua_pushexternalstring(
    lua_State* L, const char* data, size_t len, void* userdata, lua_StringFree free_cb);
LUA_API int lua_isstringexternal(lua_State* L, int idx);
LUA_API void* lua_getstringexternaluserdata(lua_State* L, int idx);

Interning

Luwu interns strings: it keeps one copy of each distinct string and compares strings by pointer. External strings are interned like any other. That has two effects on when your memory gets freed:

Things to know