Skip to content

Data

All Codecs ingest and produce Data.

There are a few concepts in Data:

  • Data always has a Type
  • Data lifetime is managed by the Engine

Data Types

enum ZL_Type

Any Data object has necessary a Type. The least specific Type is ZL_Type_serial, which means it's just a blob of bytes. Codecs can only accept and produce specified data Types. In contrast, Selectors & Graphs may optionally accept multiple data Types, using bitmap masking (example: ZL_Type_struct | ZL_Type_numeric). 

ZL_Type_serial = 1

ZL_Type_struct = 2

ZL_Type_numeric = 4

ZL_Type_string = 8

Accessors

size_t ZL_Input_numElts(const ZL_Input* input);

Note

invoking ZL_Data_numElts() is only valid for committed Data. If the Data object was received as an input, it's necessarily valid. So the issue can only happen for outputs, between allocation and commit. Querying ZL_Data_numElts() is not expected to be useful for output Data.

Note

ZL_Type_serial doesn't really have a concept of "elt". In this case, it returns Data size in bytes. 

size_t ZL_Input_eltWidth(const ZL_Input* input);

Returns:

element width in nb of bytes This is only valid for fixed size elements, such as ZL_Type_struct or ZL_Type_numeric. If Type is ZL_Type_string, it returns 0 instead. 

size_t ZL_Input_contentSize(const ZL_Input* input);

Returns:

the nb of bytes committed into data's buffer (generally == data->eltWidth * data->nbElts).

For ZL_Type_string, result is equal to sum(data->stringLens). Returned value is provided in nb of bytes.

Note

invoking this symbol only makes sense if Data was previously committed.

Note

(@cyan): ZS2_Data_byteSize() is another name candidate. 

const void* ZL_Input_ptr(const ZL_Input* input);

These methods provide direct access to internal buffer. Warning : users must pay attention to buffer boundaries.

Returns:

pointer to the beginning of buffer.

Note

for ZL_Type_string, returns a pointer to the buffer containing the concatenated strings. 

void* ZL_Output_ptr(ZL_Output* output);

These methods provide direct access to internal buffer. Warning : users must pay attention to buffer boundaries.

Returns:

pointer to buffer position to resume writing.

Note

for ZL_Type_string, returns a pointer to the buffer containing the concatenated strings. 

const uint32_t* ZL_Input_stringLens(const ZL_Input* input);

This method is only valid for ZL_Type_string Data.

Returns:

a pointer to the array of string lengths. The size of this array is == ZL_Data_numElts(data).

Returns:

NULL if incorrect data type, or StringLens not allocated yet. 

uint32_t* ZL_Output_stringLens(ZL_Output* output);

This method is only valid for ZL_Type_string Data. It requests write access into StringLens array. Only valid if StringLens array has already been allocated.

Returns:

pointer to array position to resume writing. or NULL if any of above conditions is violated.

Array's capacity is supposed known from reservation request. After writing into the array, the nb of Strings, which is also the nb of String Lengths written, must be provided using ZL_Data_commit(). 

uint32_t* ZL_Output_reserveStringLens(ZL_Output* output, size_t numStrings);

This method is only valid for ZL_Type_string Data. It reserves memory space for StringLens array, and returns a pointer to it. The buffer is owned by data and has the same lifetime. The returned pointer can be used to write into the array. After writing into the array, the nb of String Lengths provided must be signaled using ZL_Output_commit(). This method will fail if StringLens is already allocated.

Returns:

NULL if incorrect data type, or allocation error. 

ZL_Report ZL_Output_commit(ZL_Output* output, size_t numElts);

Commit the number of elements written into data.

This method must be called exactly once for every output. The nbElts must be <= reserved capacity of data. Note that, for ZL_Type_string, this is the number of strings written into data. The operation will automatically determine the total size of all Strings within data.

Returns:

Success or an error. This function will fail if it is called more than once on the same data, or if nbElts is greater than data's capacity.

Terminating a Codec without committing anything to data (not even 0) is considered an error, that is caught by the Engine (classified as node processing error).

Note

nbElts, as "number of elements", is not the same as size in bytes written in the buffer. For Numerics and Structs, the translation is straighforward. For Strings, the field sizes array must be provided first, using ZL_Data_reserveStringLens() to create and access the array. The resulting useful content size will then be calculated from the sum of field sizes. It will be controlled, and there will be an error if sum(sizes) > bufferCapacity.