More tools and compilation modes
Build & Install the Python Extension
Optionally, you can build and install the Python extension:
# Optionally create a virtualenv
python -m venv openzl-virtualenv
source ./openzl-virtualenv/bin/activate
# Install the Python extension
cd py
pip install .
See the Python API Reference to start using it.
Building OpenZL
Build OpenZL and its dependencies with CMake with the following commands. These commands will:
- Download
googletest
andzstd
from GitHub & build them. - Build OpenZL.
- Run tests.
- Installs OpenZL and Zstd to
install/
. You can change theCMAKE_INSTALL_PREFIX
to set the install directory, or leave it unset to install system-wide.
Note
This installs zstd
as well as openzl
, because we currently don't use the system zstd
.
We have the GitHub Issue#1728 to fix this.
mkdir build-cmake
cd build-cmake
cmake .. -DCMAKE_BUILD_TYPE=Release -DOPENZL_BUILD_TESTS=ON -DCMAKE_INSTALL_PREFIX=install
make -j
ctest . -j 10
make install
Running an example
Find your favorite numeric data, or use this generated data:
Run help to see the options:
$ ./examples/numeric_array --help
Usage: ./examples/numeric_array [standard|sorted|int|bfloat16|float16|float32|brute_force] <width> <input>
Compresses native-endian numeric data of the given width using the specified compressor
The first option is the compression profile, which is the compressor that should be used to compress the data.
When in doubt, you can try brute_force
, which attempts using every compressor and selects the best one.
The next argument is the integer width, in this example we'll use 4
because the pattern is "yes\n"
, which is
4 bytes.
The last argument is the input file to compress.
This particular data file is not very interesting, so feel free to bring your own numeric data & try it out.
Building against an installed OpenZL
#include <openzl/zl_compress.h>
#include <openzl/openzl.hpp>
int main(int argc, const char** argv)
{
ZL_CCtx* cctx = ZL_CCtx_create();
ZL_CCtx_free(cctx);
openzl::DCtx dctx; // C++ is also supported
}
Once you install OpenZL, you can build arbitrary code and link against it.
export OPENZL_INSTALL_PREFIX=install/
g++ main.cpp -I"$OPENZL_INSTALL_PREFIX/include" -L"$OPENZL_INSTALL_PREFIX/lib" -L"$OPENZL_INSTALL_PREFIX/lib64" -lopenzl_cpp -lopenzl -lzstd -o main
Using find_package to build OpenZL
You can also have CMake find & build OpenZL in your own project:
#include <openzl/zl_compress.h>
#include <openzl/openzl.hpp>
int main(int argc, const char** argv)
{
ZL_CCtx* cctx = ZL_CCtx_create();
ZL_CCtx_free(cctx);
openzl::DCtx dctx; // C++ is also supported
}
cmake_minimum_required(VERSION 3.20.2 FATAL_ERROR)
project(tester)
set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
include(FetchContent)
find_package(openzl)
if(NOT openzl_FOUND)
FetchContent_Declare(
openzl
GIT_REPOSITORY https://github.com/facebook/openzl.git
GIT_TAG dev
OVERRIDE_FIND_PACKAGE
)
find_package(openzl REQUIRED)
message(STATUS "Downloaded openzl")
else()
message(STATUS "Found openzl")
endif()
add_executable(main main.cpp)
target_link_libraries(main openzl openzl_cpp)
Next steps
- Read the Introduction and Concepts pages to learn more about OpenZL
- Check out the numeric_array example
- Check out the how to use OpenZL for getting started with custom compression
- Check out our API Reference
- See our PyTorch Model Compressor for a complete production compressor.