Skip to content

Compressor

The ZL_Compressor object is the core concept of the OpenZL compression engine.

A OpenZL compression graph is built out of graphs and nodes.

  • graph: Compresses one or more inputs. All graphs are built out of nodes, except for the unit graph ZL_GRAPH_STORE which takes one input and stores it in the compressed frame. OpenZL also provides many commonly used pre-built graphs in zl_public_nodes.h. Custom graphs can also be constructed using the ZL_Compressor object. Graphs are compression-time only, so custom graphs can be added any time without impacting the decompressor.

  • node: Transforms one or more inputs into one or more outputs. OpenZL provides many builtin standard nodes in zl_public_nodes.h. Users can also register custom nodes, however this requires registering a decoder with the [ZL_DCtx][] in order to decode the data.

Lifetime Management 

typedef struct ZL_Compressor_s ZL_Compressor;

ZL_Compressor* ZL_Compressor_create(void);

Create a new ZL_Compressor.

The ZL_Compressor must be freed with ZL_Compressor_free.

Returns:

The ZL_Compressor pointer or NULL on error. 

void ZL_Compressor_free(ZL_Compressor* compressor);

Frees a ZL_Compressor.

If compressor is NULL this function does nothing.

Parameters:

Errors and Warnings

During graph creation, it is common to accidentally build invalid graphs. For example, by hooking up nodes with incompatible types. Graph creation will ultimately fail during graph validation, at which point a ZL_Report will be returned. On an error ZL_Compressor_getErrorContextString will return detailed information about the error and where the error occurred. ZL_Compressor_getWarnings will return a list of warnings, or earlier errors for even more details. 

char const* ZL_Compressor_getErrorContextString(ZL_Compressor const* compressor,
                                                ZL_Report report);

Returns:

A verbose error string containing context about the error that occurred. This is useful for debugging, and for submitting bug reports to OpenZL developers.

Note

This string is stored within the compressor and is only valid for the lifetime of the compressor. 

char const* ZL_Compressor_getErrorContextString_fromError(
    ZL_Compressor const* compressor, ZL_Error error);

See ZL_Compressor_getErrorContextString()

The same as ZL_Compressor_getErrorContextString() except works on a ZL_Error. 

ZL_Error_Array ZL_Compressor_getWarnings(const ZL_Compressor* compressor);

Returns:

The array of warnings that were encountered during the creation of the compressor.

Note

The array's and the errors' lifetimes are valid until the next non- const call on the compressor.

Parameterization

Compression parameters can be attached to compressors. These parameters are used to configure the behavior of compression, e.g. by setting the compression level or format version. These parameters can be overridden by parameters set on a ZL_CCtx. 

ZL_Report ZL_Compressor_setParameter(ZL_Compressor* compressor,
                                     ZL_CParam gcparam, int value);

Set global parameters via compressor. In this construction, global parameters are attached to a Compressor object. Global Parameters set at Compressor level can be overridden later at CCtx level.

Returns:

Success or an error which can be checked with ZL_isError().

Parameters:

  • gcparam –

    The global parameter to set.

  • value –

    The value to set for the global parameter. 

int ZL_Compressor_getParameter(const ZL_Compressor* compressor,
                               ZL_CParam gcparam);

Read a parameter's configured value in the Compressor and returns it.

Returns:

Returns the value of the parameter if it is set, or 0 if unset.

Parameters:

  • gcparam –

    The global parameter to read.

Static Graph Creation

There are two types of graphs in OpenZL: static graphs and dynamic graphs. Static graphs take a single input, pass that input to a codec, and the outputs of that codec are sent to the successor graphs. Dynamic graphs are graphs that inspect the input at runtime to make different decisions. These are either function graphs or selectors.

This API allows the construction of static graphs. The head node and the successor graphs must be specified. Additionally, a name can be provided for the graph, which can aid in debugging. Finally, the graph can be parameterized, which sends the local parameters to the head node.

The main function is ZL_Compressor_buildStaticGraph. The other functions are older variants that will eventually be removed. 

struct ZL_StaticGraphParameters

const char* name;

Optionally a name for the graph for debugging. If NULL, then the static graph will not have a name. 

const ZL_LocalParams* localParams;

Optionally local parameters to pass to the head node. If NULL, then the head node's local parameters will not be overridden. 

struct ZL_StaticGraphDesc

const char* name;

ZL_NodeID headNodeid;

const ZL_GraphID* successor_gids;

size_t nbGids;

const ZL_LocalParams* localParams;

ZL_Result_ZL_GraphID ZL_Compressor_buildStaticGraph(
    ZL_Compressor* compressor, ZL_NodeID headNode,
    const ZL_GraphID* successorGraphs, size_t numSuccessorGraphs,
    const ZL_StaticGraphParameters* params);

Build a new graph out of pre-existing components. The new graph passes its data to headNode, and then each output of headNode is set to the corresponding successorGraph.

Parameters:

  • headNode –

    Pass the input data to this node

  • successorGraphs –

    Pass the outputs of headNode to these graphs

  • numSuccessorGraphs –

    Number of successor graphs

  • params –

    Optionally extra parameters for the static graph, or NULL.

Returns:

Thew new graph ID, or an error. 

ZL_GraphID ZL_Compressor_registerStaticGraph_fromNode1o(
    ZL_Compressor* compressor, ZL_NodeID headNode, ZL_GraphID dstGraph);

Create a graph from a single input & output node.

Simplified variant of ZL_Compressor_registerStaticGraph_fromNode that only works for nodes that have one input and one output. Creates a new graph headed by headNode, whose output gets sent to dstGraph.

Returns:

The newly created graph or ZL_GRAPH_ILLEGAL on error. The user may check for errors using ZL_GraphID_isValid().

Parameters:

  • headNode –

    The node executed first in the newly created graph.

  • dstGraph –

    The graph that will receive the output of headNode. 

ZL_GraphID ZL_Compressor_registerStaticGraph_fromPipelineNodes1o(
    ZL_Compressor* compressor, const ZL_NodeID* nodes, size_t nbNodes,
    ZL_GraphID dstGraph);

Creates a graph consisting of a series of nodes executed in succession in the order provided and then sent to dstGraph.

Returns:

The newly created graph or ZL_GRAPH_ILLEGAL on error. The user may check for errors using ZL_GraphID_isValid().

Parameters:

  • nodes –

    The nodes to execute in the newly created graph.

  • nbNodes –

    The number of nodes in nodes.

  • dstGraph –

    The graph that will receive the output of the last node in nodes. 

ZL_GraphID ZL_Compressor_registerStaticGraph_fromNode(
    ZL_Compressor* compressor, ZL_NodeID headNode, const ZL_GraphID* dstGraphs,
    size_t nbDstGraphs);

Create a graph from a head node.

Creates a new graph headed by headNode, which produces nbDstGraphs outcomes. Each outcome of headNode gets sent to the corresponding graph in dstGraphs.

Parameters:

  • headNode –

    The head node in the newly created graph.

  • dstGraphs –

    Array of graphs of size nbDstGraphs.

  • nbDstGraphs –

    Must be equal to the number of outputs of headNode.

Returns:

The newly created graph or ZL_GRAPH_ILLEGAL on error. The user may check for errors using ZL_GraphID_isValid().

Note

Successor dstGraphs can only be employed in single-input mode. Multi-input Graphs can only be invoked from a function graph. 

ZL_GraphID ZL_Compressor_registerStaticGraph(ZL_Compressor* compressor,
                                             const ZL_StaticGraphDesc* sgDesc);

This is the more complete declaration variant, offering more control and capabilities. In order to be valid, a Static Graph Description must :

  • provide exactly as many successors as nb of outcomes defined by head Node

  • only employ single-input Graphs as successors

  • match each outcome type with a successor using a compatible input type

  • optionally, can specify @localParams for a Static Graph. In this case, these parameters are forwarded to the Head Node, replacing any previous local parameter that may have been already set on the Head Node.

If a declaration is invalid, it results in an invalid GraphID, which can be tested using ZL_GraphID_isValid() on the return value. Note: ZL_GraphID_isValid() is currently defined in zs2_graph_api.h.

#define ZL_NODELIST(...) ZL_GENERIC_LIST(ZL_NodeID, __VA_ARGS__)

C11 Helper, to define a list of Nodes as compound literals. Example: ZL_Compressor_registerStaticGraph_fromPipelineNodes1o( compressor, ZL_NODELIST(node1, node2, node3), finalGraph)

#define ZL_GRAPHLIST(...) ZL_GENERIC_LIST(ZL_GraphID, __VA_ARGS__)

C11 Helper, to define a list of Graphs as compound literals. Example : ZL_Compressor_registerStaticGraph_fromNode(compressor, startingNodeID, ZL_GRAPHLIST(graph1, graph2, graph3) )

Node Customization

Nodes can be customized to override their name and local parameters. This is an advanced use case, and mainly an implementation detail of nodes. Most nodes that accept parameters provide helper functions to correctly parameterize the node. 

struct ZL_NodeParameters

const char* name;

Optionally a new name, if NULL it is derived from the node's name. 

const ZL_LocalParams* localParams;

Optionally the new local params, if NULL then the parameters are not updated. 

struct ZL_ParameterizedNodeDesc

const char* name;

Optionally a new name, if NULL it is derived from the node's name. 

ZL_NodeID node;

Node to parameterize. 

const ZL_LocalParams* localParams;

Optionally the new local params, if NULL then the parameters are not updated. 

ZL_Result_ZL_NodeID ZL_Compressor_parameterizeNode(
    ZL_Compressor* compressor, ZL_NodeID node, const ZL_NodeParameters* params);

Parameterize an existing node by overriding its name and/or local parameters.

Parameters:

  • node –

    The node to parameterize.

  • params –

    The new parameters, which must be non-null.

Returns:

The new node ID on success, or an error. 

ZL_NodeID ZL_Compressor_registerParameterizedNode(
    ZL_Compressor* compressor, const ZL_ParameterizedNodeDesc* desc);

Clone an existing [ZL_NodeID][] from an existing node, but optionally with a new name & new parameters.

Parameters:

  • desc –

    The parameterization options.

Returns:

The new node id of the cloned node. 

ZL_NodeID ZL_Compressor_cloneNode(ZL_Compressor* compressor, ZL_NodeID nodeid,
                                  const ZL_LocalParams* localParams);

Simplified variant of ZL_Compressor_registerParameterizedNode(). Clone an existing [ZL_NodeID][] from an already registered nodeid but employs new parameters, set via localParams.

Returns:

The new node id of the cloned node.

Parameters:

  • nodeid –

    The node to clone.

  • localParams –

    The local parameters to use for the node.

Graph Customization

Graphs can be customized to override their name, local parameters, custom nodes and custom graphs. This is an advanced use case, and mainly an implementation detail of graphs. Most graphs which accept parameters provide helper functions to correctly parameterize the graph. 

struct ZL_GraphParameters_s

const char* name;

Optional, for debug traces, otherwise it is derived from graph's name. 

const ZL_GraphID* customGraphs;

Empty means don't override. 

size_t nbCustomGraphs;

const ZL_NodeID* customNodes;

Empty means don't override. 

size_t nbCustomNodes;

const ZL_LocalParams* localParams;

NULL means don't override. 

struct ZL_ParameterizedGraphDesc

This creates a new Graphs, based on an existing Graph, but modifying all or parts of its exposed parameters. 

const char* name;

Optionally a new name, otherwise it is derived from graph's name. 

ZL_GraphID graph;

const ZL_GraphID* customGraphs;

Empty means don't override. 

size_t nbCustomGraphs;

const ZL_NodeID* customNodes;

Empty means don't override. 

size_t nbCustomNodes;

const ZL_LocalParams* localParams;

NULL means don't override. 

typedef struct ZL_GraphParameters_s ZL_GraphParameters;

ZL_Result_ZL_GraphID ZL_Compressor_parameterizeGraph(
    ZL_Compressor* compressor, ZL_GraphID graph,
    const ZL_GraphParameters* params);

Parameterizes an existing graph by overriding its name, customGraphs, customNodes, and/or localParams.

Parameters:

  • graph –

    The graph to parameterize.

  • params –

    The new parameters, which must be non-null.

Returns:

The new graph ID on success, or an error. 

ZL_GraphID ZL_Compressor_registerParameterizedGraph(
    ZL_Compressor* compressor, const ZL_ParameterizedGraphDesc* desc);

Create a new GraphID by the one from gid, just replacing the localParams by the provided ones. Used to create custom variants of Standard Graphs for example.

Note

the original @gid still exists and remains accessible.

Note

@localParams==NULL means "do not change the parameters", in which case, this function simply returns @gid.

Returns:

The GraphID of the newly created graph, or ZL_GRAPH_ILLEGAL on error.

Parameters:

  • gid –

    The GraphID to clone.

  • localParams –

    The local parameters to use inside the graph.

Graph Component Lookup

Any registered node or graph that has an explicit name can be queried using these lookup functions. See names.md for details. 

ZL_NodeID ZL_Compressor_getNode(const ZL_Compressor* compressor,
                                const char* name);

Lookup a node by name.

Looks up a node with the given name and returns it. Anchor nodes (nodes whose name starts with '!') can be looked up by name, excluding the leading '!'. Standard nodes can also be looked up by name. Non-anchor nodes are assigned a unique name by suffixing them with #${unique}. They can be looked up if you know the unique name.

Returns:

The node if it exists, or ZL_NODE_ILLEGAL. 

ZL_GraphID ZL_Compressor_getGraph(const ZL_Compressor* compressor,
                                  const char* graph);

Lookup a graph by name.

Looks up a graph with the given name and returns it. Anchor graphs (graphs whose name starts with '!') can be looked up by name, excluding the leading '!'. Standard graphs can also be looked up by name. Non-anchor graphs are assigned a unique name by suffixing them with #${unique}. They can be looked up if you know the unique name.

Returns:

The graph if it exists, or ZL_GRAPH_ILLEGAL.

Graph Finalization

After creation, the graph is finalized with ZL_Compressor_selectStartingGraphID. At this point any errors during graph creation that haven't already been caught will be surfaced. 

ZL_Report ZL_Compressor_selectStartingGraphID(ZL_Compressor* compressor,
                                              ZL_GraphID graph);

Selects a graph as the default entry point for the compressor.

By default, a compressor's entry point is its most recently registered graph. This function allows explicit selection of a different graph as the default entry point for subsequent compression operations.

Parameters:

  • compressor –

    The compressor instance to configure. Must not be NULL.

  • graph –

    The graph ID to set as the default entry point. Must be a valid graph ID that has been registered with this compressor.

Returns:

ZL_Report indicating success or failure. Use ZL_isError() to check for errors.

Note

The compressor can still be used as a collection of multiple entry points. Alternative entry points can be selected at runtime using ZL_CCtx_selectStartingGraphID().

Note

This operation automatically validates the compressor by calling ZL_Compressor_validate() internally.

See ZL_CCtx_selectStartingGraphID() for runtime entry point selection 

ZL_Report ZL_Compressor_validate(ZL_Compressor* compressor,
                                 ZL_GraphID starting_graph);

Validates a graph maintains basic invariants to reduce the chance of errors being triggered when compressing.

Note

this operation is also integrated as part of ZL_Compressor_selectStartingGraphID(). This function is kept for backward compatibility.

Returns:

Success if graph is valid, error otherwise.

Parameters:

  • starting_graph –

    The starting graph to validate.

Reference Compressor

To use a compressor to compress an input, call ZL_CCtx_refCompressor. 

ZL_Report ZL_CCtx_refCompressor(ZL_CCtx* cctx, const ZL_Compressor* compressor);

Pass compressor as a ZL_Compressor* object to the compression state. Compression will start with the default Starting GraphID of compressor, using its default parameters provided at registration time.

Note

Only one compressor can be referenced at a time. Referencing a new compressor deletes previous reference.

Note

If a custom GraphID and parameters were previously set, invoking this method will reset them to default.

Preconditions:

  • compressor must remain valid for the duration of its usage.

  • compressor must be already validated. 

ZL_Report ZL_CCtx_selectStartingGraphID(ZL_CCtx* cctx,
                                        const ZL_Compressor* compressor,
                                        ZL_GraphID graphID,
                                        const ZL_GraphParameters* rgp);

Set the starting Graph of next compression, as graphID referenced in the provided compressor, optionally providing it with some runtime parameters.

Preconditions:

  • compressor must remain valid for the duration of its usage.

  • compressor must be already validated.

Parameters:

  • cctx –

    The active compression state

  • compressor –

    The reference compressor containing graph definitions. If NULL, it uses the currently registered compressor.

  • graphID –

    The ID of the starting graph.

  • rgp –

    Optional parameters to apply to the starting graph. NULL means don't override.

Note: like all global parameters, these parameters are reset at end of compression.