StarMirror/sandboxed-api
1
0
Fork 0
mirror of https://github.com/google/sandboxed-api.git synced 2024-03-22 13:11:30 +08:00
Code Issues Projects Releases Wiki Activity

Merge pull request #49 from bohdanty:master

Browse Source 
PiperOrigin-RevId: 332240434
Change-Id: I9e71cf3d3ab50e3a379e3c651b931ca95692666f
This commit is contained in:
Copybara-Service 2020-09-17 08:36:07 -07:00
commit c663427cf9
21 changed files with 1375 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
oss-internship-2020/guetzli/.bazelrc Normal file
View File

@ -0,0 +1,2 @@
# Build in C++17 mode without a custom CROSSTOOL
build --cxxopt=-std=c++17

83
oss-internship-2020/guetzli/BUILD.bazel Normal file
View File

@ -0,0 +1,83 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
load(
"@com_google_sandboxed_api//sandboxed_api/bazel:sapi.bzl",
"sapi_library",
)
licenses(["notice"])
cc_library(
name = "guetzli_wrapper",
srcs = ["guetzli_entry_points.cc"],
hdrs = ["guetzli_entry_points.h"],
deps = [
"@com_google_sandboxed_api//sandboxed_api:lenval_core",
"@com_google_sandboxed_api//sandboxed_api:vars",
"@guetzli//:guetzli_lib",
"@png_archive//:png",
],
)
sapi_library(
name = "guetzli_sapi",
srcs = ["guetzli_transaction.cc"],
hdrs = [
"guetzli_sandbox.h",
"guetzli_transaction.h",
],
functions = [
"ProcessJpeg",
"ProcessRgb",
"WriteDataToFd",
],
input_files = ["guetzli_entry_points.h"],
lib = ":guetzli_wrapper",
lib_name = "Guetzli",
namespace = "guetzli::sandbox",
visibility = ["//visibility:public"],
)
cc_binary(
name = "guetzli_sandboxed",
srcs = ["guetzli_sandboxed.cc"],
deps = [
":guetzli_sapi",
],
)
cc_test(
name = "transaction_tests",
size = "large",
srcs = ["guetzli_transaction_test.cc"],
data = glob(["testdata/*"]),
visibility = ["//visibility:public"],
deps = [
"//:guetzli_sapi",
"@com_google_googletest//:gtest_main",
],
)
cc_test(
name = "sapi_lib_tests",
size = "large",
srcs = ["guetzli_sapi_test.cc"],
data = glob(["testdata/*"]),
visibility = ["//visibility:public"],
deps = [
"//:guetzli_sapi",
"@com_google_googletest//:gtest_main",
],
)

32
oss-internship-2020/guetzli/README.md Normal file
View File

@ -0,0 +1,32 @@
# Guetzli Sandbox
This is an example implementation of a sandbox for the [Guetzli](https://github.com/google/guetzli) library using [Sandboxed API](https://github.com/google/sandboxed-api).
Please read Guetzli's [documentation](https://github.com/google/guetzli#introduction) to learn more about it.
## Implementation details
Because Guetzli provides a C++ API and SAPI requires functions to be `extern "C"`, a wrapper library has been written for the compatibility. SAPI provides a Transaction class, which is a convenient way to create a wrapper for your sandboxed API that handles internal errors. The original Guetzli has a command-line utility to encode images, so a fully compatible utility that uses sandboxed Guetzli is provided.
The wrapper around Guetzli uses file descriptors to pass data to the sandbox. This approach restricts the sandbox from using the `open()` syscall and also helps to prevent making copies of data, because you need to synchronize it between processes.
## Build Guetzli Sandboxed
Right now Sandboxed API support only Linux systems, so you need one to build it. Guetzli sandboxed uses [Bazel](https://bazel.build/) as a build system so you need to [install it](https://docs.bazel.build/versions/3.4.0/install.html) before building.
To build Guetzli sandboxed encoding utility you can use this command:
`bazel build //:guetzli_sandboxed`
Then you can use it in this way:
```
guetzli_sandboxed [--quality Q] [--verbose] original.png output.jpg
guetzli_sandboxed [--quality Q] [--verbose] original.jpg output.jpg
```
Refer to Guetzli's [documentation](https://github.com/google/guetzli#using) to read more about usage.
## Examples
There are two different sets of unit tests which demonstrate how to use different parts of Guetzli sandboxed:
* `tests/guetzli_sapi_test.cc` - example usage of Guetzli sandboxed API.
* `tests/guetzli_transaction_test.cc` - example usage of Guetzli transaction.
To run tests use the following command:
`bazel test ...`
Also, there is an example of custom security policy for your sandbox in
`guetzli_sandbox.h`

100
oss-internship-2020/guetzli/WORKSPACE Normal file
View File

@ -0,0 +1,100 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
workspace(name = "guetzli_sandboxed")
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
# Include the Sandboxed API dependency if it does not already exist in this
# project. This ensures that this workspace plays well with other external
# dependencies that might use Sandboxed API.
maybe(
git_repository,
name = "com_google_sandboxed_api",
# This example depends on the latest master. In an embedding project, it
# is advisable to pin Sandboxed API to a specific revision instead.
# commit = "ba47adc21d4c9bc316f3c7c32b0faaef952c111e", # 2020-05-15
branch = "master",
remote = "https://github.com/google/sandboxed-api.git",
)
# From here on, Sandboxed API files are available. The statements below setup
# transitive dependencies such as Abseil. Like above, those will only be
# included if they don't already exist in the project.
load(
"@com_google_sandboxed_api//sandboxed_api/bazel:sapi_deps.bzl",
"sapi_deps",
)
sapi_deps()
# Need to separately setup Protobuf dependencies in order for the build rules
# to work.
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
http_archive(
name = "guetzli",
build_file = "guetzli.BUILD",
sha256 = "39632357e49db83d9560bf0de560ad833352f36d23b109b0e995b01a37bddb57",
strip_prefix = "guetzli-master",
url = "https://github.com/google/guetzli/archive/master.zip",
)
http_archive(
name = "butteraugli",
build_file = "butteraugli.BUILD",
sha256 = "39632357e49db83d9560bf0de560ad833352f36d23b109b0e995b01a37bddb57",
strip_prefix = "guetzli-master/third_party/butteraugli",
url = "https://github.com/google/guetzli/archive/master.zip",
)
http_archive(
name = "png_archive",
build_file = "png.BUILD",
sha256 = "a941dc09ca00148fe7aaf4ecdd6a67579c293678ed1e1cf633b5ffc02f4f8cf7",
strip_prefix = "libpng-1.2.57",
url = "http://github.com/glennrp/libpng/archive/v1.2.57.zip",
)
http_archive(
name = "jpeg_archive",
build_file = "jpeg.BUILD",
sha256 = "240fd398da741669bf3c90366f58452ea59041cacc741a489b99f2f6a0bad052",
strip_prefix = "jpeg-9b",
url = "http://www.ijg.org/files/jpegsrc.v9b.tar.gz",
)
http_archive(
name = "net_zlib",
build_file = "zlib.BUILD",
sha256 = "c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1", # 2020-04-23
strip_prefix = "zlib-1.2.11",
urls = [
"https://mirror.bazel.build/zlib.net/zlib-1.2.11.tar.gz",
"https://www.zlib.net/zlib-1.2.11.tar.gz",
],
)
# GoogleTest/GoogleMock
maybe(
http_archive,
name = "com_google_googletest",
sha256 = "a6ab7c7d6fd4dd727f6012b5d85d71a73d3aa1274f529ecd4ad84eb9ec4ff767", # 2020-04-16
strip_prefix = "googletest-dcc92d0ab6c4ce022162a23566d44f673251eee4",
urls = ["https://github.com/google/googletest/archive/dcc92d0ab6c4ce022162a23566d44f673251eee4.zip"],
)

28
oss-internship-2020/guetzli/external/butteraugli.BUILD vendored Normal file
View File

@ -0,0 +1,28 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"])
cc_library(
name = "butteraugli",
srcs = [
"butteraugli/butteraugli.cc",
"butteraugli/butteraugli.h",
],
hdrs = [
"butteraugli/butteraugli.h",
],
copts = ["-Wno-sign-compare"],
visibility = ["//visibility:public"],
)

32
oss-internship-2020/guetzli/external/guetzli.BUILD vendored Normal file
View File

@ -0,0 +1,32 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"])
cc_library(
name = "guetzli_lib",
srcs = glob(
[
"guetzli/*.h",
"guetzli/*.cc",
"guetzli/*.inc",
],
exclude = ["guetzli/guetzli.cc"],
),
copts = [ "-Wno-sign-compare" ],
visibility= [ "//visibility:public" ],
deps = [
"@butteraugli//:butteraugli",
],
)

89
oss-internship-2020/guetzli/external/jpeg.BUILD vendored Normal file
View File

@ -0,0 +1,89 @@
# Description:
# The Independent JPEG Group's JPEG runtime library.
licenses(["notice"]) # custom notice-style license, see LICENSE
cc_library(
name = "jpeg",
srcs = [
"cderror.h",
"cdjpeg.h",
"jaricom.c",
"jcapimin.c",
"jcapistd.c",
"jcarith.c",
"jccoefct.c",
"jccolor.c",
"jcdctmgr.c",
"jchuff.c",
"jcinit.c",
"jcmainct.c",
"jcmarker.c",
"jcmaster.c",
"jcomapi.c",
"jconfig.h",
"jcparam.c",
"jcprepct.c",
"jcsample.c",
"jctrans.c",
"jdapimin.c",
"jdapistd.c",
"jdarith.c",
"jdatadst.c",
"jdatasrc.c",
"jdcoefct.c",
"jdcolor.c",
"jdct.h",
"jddctmgr.c",
"jdhuff.c",
"jdinput.c",
"jdmainct.c",
"jdmarker.c",
"jdmaster.c",
"jdmerge.c",
"jdpostct.c",
"jdsample.c",
"jdtrans.c",
"jerror.c",
"jfdctflt.c",
"jfdctfst.c",
"jfdctint.c",
"jidctflt.c",
"jidctfst.c",
"jidctint.c",
"jinclude.h",
"jmemmgr.c",
"jmemnobs.c",
"jmemsys.h",
"jmorecfg.h",
"jquant1.c",
"jquant2.c",
"jutils.c",
"jversion.h",
"transupp.h",
],
hdrs = [
"jerror.h",
"jpegint.h",
"jpeglib.h",
],
includes = ["."],
visibility = ["//visibility:public"],
)
genrule(
name = "configure",
outs = ["jconfig.h"],
cmd = "cat <<EOF >$@\n" +
"#define HAVE_PROTOTYPES 1\n" +
"#define HAVE_UNSIGNED_CHAR 1\n" +
"#define HAVE_UNSIGNED_SHORT 1\n" +
"#define HAVE_STDDEF_H 1\n" +
"#define HAVE_STDLIB_H 1\n" +
"#ifdef WIN32\n" +
"#define INLINE __inline\n" +
"#else\n" +
"#define INLINE __inline__\n" +
"#endif\n" +
"EOF\n",
)

33
oss-internship-2020/guetzli/external/png.BUILD vendored Normal file
View File

@ -0,0 +1,33 @@
# Description:
# libpng is the official PNG reference library.
licenses(["notice"]) # BSD/MIT-like license
cc_library(
name = "png",
srcs = [
"png.c",
"pngerror.c",
"pngget.c",
"pngmem.c",
"pngpread.c",
"pngread.c",
"pngrio.c",
"pngrtran.c",
"pngrutil.c",
"pngset.c",
"pngtrans.c",
"pngwio.c",
"pngwrite.c",
"pngwtran.c",
"pngwutil.c",
],
hdrs = [
"png.h",
"pngconf.h",
],
includes = ["."],
linkopts = ["-lm"],
visibility = ["//visibility:public"],
deps = ["@net_zlib//:zlib"],
)

56
oss-internship-2020/guetzli/external/zlib.BUILD vendored Normal file
View File

@ -0,0 +1,56 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"])
cc_library(
name = "zlib",
srcs = [
"adler32.c",
"compress.c",
"crc32.c",
"deflate.c",
"gzclose.c",
"gzlib.c",
"gzread.c",
"gzwrite.c",
"infback.c",
"inffast.c",
"inflate.c",
"inflate.h",
"inftrees.c",
"trees.c",
"trees.h",
"uncompr.c",
"zutil.c",
],
hdrs = ["zlib.h"],
# Use -Dverbose=-1 to turn off zlib's trace logging. (#3280)
copts = [
"-w",
"-Dverbose=-1",
],
includes = ["."],
textual_hdrs = [
"crc32.h",
"deflate.h",
"gzguts.h",
"inffast.h",
"inffixed.h",
"inftrees.h",
"zconf.h",
"zutil.h",
],
visibility = ["//visibility:public"],
)

293
oss-internship-2020/guetzli/guetzli_entry_points.cc Normal file
View File

@ -0,0 +1,293 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "guetzli_entry_points.h" // NOLINT(build/include)
#include <sys/stat.h>
#include <algorithm>
#include <cstdio>
#include <iostream>
#include <string>
#include <vector>
#include "guetzli/jpeg_data_reader.h"
#include "guetzli/quality.h"
#include "png.h" // NOLINT(build/include)
#include "sandboxed_api/sandbox2/util/fileops.h"
#include "sandboxed_api/util/statusor.h"
namespace {
constexpr int kBytesPerPixel = 350;
constexpr int kLowestMemusageMB = 100;
struct GuetzliInitData {
std::string in_data;
guetzli::Params params;
guetzli::ProcessStats stats;
};
struct ImageData {
int xsize;
int ysize;
std::vector<uint8_t> rgb;
};
sapi::LenValStruct CreateLenValFromData(const void* data, size_t size) {
void* new_data = malloc(size);
memcpy(new_data, data, size);
return {size, new_data};
}
sapi::StatusOr<std::string> ReadFromFd(int fd) {
struct stat file_data;
int status = fstat(fd, &file_data);
if (status < 0) {
return absl::FailedPreconditionError("Error reading input from fd");
}
std::string result;
result.resize(file_data.st_size);
status = read(fd, result.data(), result.size());
if (status < 0) {
return absl::FailedPreconditionError("Error reading input from fd");
}
return result;
}
sapi::StatusOr<GuetzliInitData> PrepareDataForProcessing(
const ProcessingParams& processing_params) {
sapi::StatusOr<std::string> input = ReadFromFd(processing_params.remote_fd);
if (!input.ok()) {
return input.status();
}
guetzli::Params guetzli_params;
guetzli_params.butteraugli_target = static_cast<float>(
guetzli::ButteraugliScoreForQuality(processing_params.quality));
guetzli::ProcessStats stats;
if (processing_params.verbose) {
stats.debug_output_file = stderr;
}
return GuetzliInitData{std::move(input.value()), guetzli_params, stats};
}
inline uint8_t BlendOnBlack(const uint8_t val, const uint8_t alpha) {
return (static_cast<int>(val) * static_cast<int>(alpha) + 128) / 255;
}
// Modified version of ReadPNG from original guetzli.cc
sapi::StatusOr<ImageData> ReadPNG(const std::string& data) {
std::vector<uint8_t> rgb;
int xsize, ysize;
png_structp png_ptr =
png_create_read_struct(PNG_LIBPNG_VER_STRING, nullptr, nullptr, nullptr);
if (!png_ptr) {
return absl::FailedPreconditionError(
"Error reading PNG data from input file");
}
png_infop info_ptr = png_create_info_struct(png_ptr);
if (!info_ptr) {
png_destroy_read_struct(&png_ptr, nullptr, nullptr);
return absl::FailedPreconditionError(
"Error reading PNG data from input file");
}
if (setjmp(png_jmpbuf(png_ptr)) != 0) {
// Ok we are here because of the setjmp.
png_destroy_read_struct(&png_ptr, &info_ptr, nullptr);
return absl::FailedPreconditionError(
"Error reading PNG data from input file");
}
std::istringstream memstream(data, std::ios::in | std::ios::binary);
png_set_read_fn(
png_ptr, static_cast<void*>(&memstream),
[](png_structp png_ptr, png_bytep outBytes, png_size_t byteCountToRead) {
std::istringstream& memstream =
*static_cast<std::istringstream*>(png_get_io_ptr(png_ptr));
memstream.read(reinterpret_cast<char*>(outBytes), byteCountToRead);
if (memstream.eof()) png_error(png_ptr, "unexpected end of data");
if (memstream.fail()) png_error(png_ptr, "read from memory error");
});
// The png_transforms flags are as follows:
// packing == convert 1,2,4 bit images,
// strip == 16 -> 8 bits / channel,
// shift == use sBIT dynamics, and
// expand == palettes -> rgb, grayscale -> 8 bit images, tRNS -> alpha.
const unsigned int png_transforms =
PNG_TRANSFORM_PACKING | PNG_TRANSFORM_EXPAND | PNG_TRANSFORM_STRIP_16;
png_read_png(png_ptr, info_ptr, png_transforms, nullptr);
png_bytep* row_pointers = png_get_rows(png_ptr, info_ptr);
xsize = png_get_image_width(png_ptr, info_ptr);
ysize = png_get_image_height(png_ptr, info_ptr);
rgb.resize(3 * xsize * ysize);
const int components = png_get_channels(png_ptr, info_ptr);
switch (components) {
case 1: {
// GRAYSCALE
for (int y = 0; y < ysize; ++y) {
const uint8_t* row_in = row_pointers[y];
uint8_t* row_out = &rgb[3 * y * xsize];
for (int x = 0; x < xsize; ++x) {
const uint8_t gray = row_in[x];
row_out[3 * x + 0] = gray;
row_out[3 * x + 1] = gray;
row_out[3 * x + 2] = gray;
}
}
break;
}
case 2: {
// GRAYSCALE + ALPHA
for (int y = 0; y < ysize; ++y) {
const uint8_t* row_in = row_pointers[y];
uint8_t* row_out = &rgb[3 * y * xsize];
for (int x = 0; x < xsize; ++x) {
const uint8_t gray = BlendOnBlack(row_in[2 * x], row_in[2 * x + 1]);
row_out[3 * x + 0] = gray;
row_out[3 * x + 1] = gray;
row_out[3 * x + 2] = gray;
}
}
break;
}
case 3: {
// RGB
for (int y = 0; y < ysize; ++y) {
const uint8_t* row_in = row_pointers[y];
uint8_t* row_out = &rgb[3 * y * xsize];
memcpy(row_out, row_in, 3 * xsize);
}
break;
}
case 4: {
// RGBA
for (int y = 0; y < ysize; ++y) {
const uint8_t* row_in = row_pointers[y];
uint8_t* row_out = &rgb[3 * y * xsize];
for (int x = 0; x < xsize; ++x) {
const uint8_t alpha = row_in[4 * x + 3];
row_out[3 * x + 0] = BlendOnBlack(row_in[4 * x + 0], alpha);
row_out[3 * x + 1] = BlendOnBlack(row_in[4 * x + 1], alpha);
row_out[3 * x + 2] = BlendOnBlack(row_in[4 * x + 2], alpha);
}
}
break;
}
default:
png_destroy_read_struct(&png_ptr, &info_ptr, nullptr);
return absl::FailedPreconditionError(
"Error reading PNG data from input file");
}
png_destroy_read_struct(&png_ptr, &info_ptr, nullptr);
return ImageData{xsize, ysize, std::move(rgb)};
}
bool CheckMemoryLimitExceeded(int memlimit_mb, int xsize, int ysize) {
double pixels = static_cast<double>(xsize) * ysize;
return memlimit_mb != -1 &&
(pixels * kBytesPerPixel / (1 << 20) > memlimit_mb ||
memlimit_mb < kLowestMemusageMB);
}
} // namespace
extern "C" bool ProcessJpeg(const ProcessingParams* processing_params,
sapi::LenValStruct* output) {
auto processing_data = PrepareDataForProcessing(*processing_params);
if (!processing_data.ok()) {
std::cerr << processing_data.status().ToString() << std::endl;
return false;
}
guetzli::JPEGData jpg_header;
if (!guetzli::ReadJpeg(processing_data->in_data, guetzli::JPEG_READ_HEADER,
&jpg_header)) {
std::cerr << "Error reading JPG data from input file" << std::endl;
return false;
}
if (CheckMemoryLimitExceeded(processing_params->memlimit_mb, jpg_header.width,
jpg_header.height)) {
std::cerr << "Memory limit would be exceeded" << std::endl;
return false;
}
std::string out_data;
if (!guetzli::Process(processing_data->params, &processing_data->stats,
processing_data->in_data, &out_data)) {
std::cerr << "Guezli processing failed" << std::endl;
return false;
}
*output = CreateLenValFromData(out_data.data(), out_data.size());
return true;
}
extern "C" bool ProcessRgb(const ProcessingParams* processing_params,
sapi::LenValStruct* output) {
auto processing_data = PrepareDataForProcessing(*processing_params);
if (!processing_data.ok()) {
std::cerr << processing_data.status().ToString() << std::endl;
return false;
}
auto png_data = ReadPNG(processing_data->in_data);
if (!png_data.ok()) {
std::cerr << "Error reading PNG data from input file" << std::endl;
return false;
}
if (CheckMemoryLimitExceeded(processing_params->memlimit_mb, png_data->xsize,
png_data->ysize)) {
std::cerr << "Memory limit would be exceeded" << std::endl;
return false;
}
std::string out_data;
if (!guetzli::Process(processing_data->params, &processing_data->stats,
png_data->rgb, png_data->xsize, png_data->ysize,
&out_data)) {
std::cerr << "Guetzli processing failed" << std::endl;
return false;
}
*output = CreateLenValFromData(out_data.data(), out_data.size());
return true;
}
extern "C" bool WriteDataToFd(int fd, sapi::LenValStruct* data) {
return sandbox2::file_util::fileops::WriteToFD(
fd, static_cast<const char*>(data->data), data->size);
}

35
oss-internship-2020/guetzli/guetzli_entry_points.h Normal file
View File

@ -0,0 +1,35 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef GUETZLI_SANDBOXED_GUETZLI_ENTRY_POINTS_H_
#define GUETZLI_SANDBOXED_GUETZLI_ENTRY_POINTS_H_
#include "guetzli/processor.h"
#include "sandboxed_api/lenval_core.h"
#include "sandboxed_api/vars.h"
struct ProcessingParams {
int remote_fd = -1;
int verbose = 0;
int quality = 0;
int memlimit_mb = 0;
};
extern "C" bool ProcessJpeg(const ProcessingParams* processing_params,
sapi::LenValStruct* output);
extern "C" bool ProcessRgb(const ProcessingParams* processing_params,
sapi::LenValStruct* output);
extern "C" bool WriteDataToFd(int fd, sapi::LenValStruct* data);
#endif // GUETZLI_SANDBOXED_GUETZLI_ENTRY_POINTS_H_

45
oss-internship-2020/guetzli/guetzli_sandbox.h Normal file
View File

@ -0,0 +1,45 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef GUETZLI_SANDBOXED_GUETZLI_SANDBOX_H_
#define GUETZLI_SANDBOXED_GUETZLI_SANDBOX_H_
#include <syscall.h>
#include "guetzli_sapi.sapi.h" // NOLINT(build/include)
namespace guetzli::sandbox {
class GuetzliSapiSandbox : public GuetzliSandbox {
public:
std::unique_ptr<sandbox2::Policy> ModifyPolicy(
sandbox2::PolicyBuilder*) override {
return sandbox2::PolicyBuilder()
.AllowStaticStartup()
.AllowRead()
.AllowSystemMalloc()
.AllowWrite()
.AllowExit()
.AllowStat()
.AllowSyscalls({
__NR_futex, __NR_close,
__NR_recvmsg // To work with remote fd
})
.BuildOrDie();
}
};
} // namespace guetzli::sandbox
#endif // GUETZLI_SANDBOXED_GUETZLI_SANDBOX_H_

96
oss-internship-2020/guetzli/guetzli_sandboxed.cc Normal file
View File

@ -0,0 +1,96 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <cstdio>
#include <iostream>
#include "guetzli_transaction.h" // NOLINT(build/include)
#include "sandboxed_api/sandbox2/util/fileops.h"
#include "sandboxed_api/util/statusor.h"
namespace {
constexpr int kDefaultJPEGQuality = 95;
constexpr int kDefaultMemlimitMB = 6000;
void Usage() {
fprintf(stderr,
"Guetzli JPEG compressor. Usage: \n"
"guetzli [flags] input_filename output_filename\n"
"\n"
"Flags:\n"
" --verbose - Print a verbose trace of all attempts to standard "
"output.\n"
" --quality Q - Visual quality to aim for, expressed as a JPEG "
"quality value.\n"
" Default value is %d.\n"
" --memlimit M - Memory limit in MB. Guetzli will fail if unable to "
"stay under\n"
" the limit. Default limit is %d MB.\n"
" --nomemlimit - Do not limit memory usage.\n",
kDefaultJPEGQuality, kDefaultMemlimitMB);
exit(1);
}
} // namespace
int main(int argc, const char** argv) {
int verbose = 0;
int quality = kDefaultJPEGQuality;
int memlimit_mb = kDefaultMemlimitMB;
int opt_idx = 1;
for (; opt_idx < argc; opt_idx++) {
if (strnlen(argv[opt_idx], 2) < 2 || argv[opt_idx][0] != '-' ||
argv[opt_idx][1] != '-')
break;
if (!strcmp(argv[opt_idx], "--verbose")) {
verbose = 1;
} else if (!strcmp(argv[opt_idx], "--quality")) {
opt_idx++;
if (opt_idx >= argc) Usage();
quality = atoi(argv[opt_idx]); // NOLINT(runtime/deprecated_fn)
} else if (!strcmp(argv[opt_idx], "--memlimit")) {
opt_idx++;
if (opt_idx >= argc) Usage();
memlimit_mb = atoi(argv[opt_idx]); // NOLINT(runtime/deprecated_fn)
} else if (!strcmp(argv[opt_idx], "--nomemlimit")) {
memlimit_mb = -1;
} else if (!strcmp(argv[opt_idx], "--")) {
opt_idx++;
break;
} else {
fprintf(stderr, "Unknown commandline flag: %s\n", argv[opt_idx]);
Usage();
}
}
if (argc - opt_idx != 2) {
Usage();
}
guetzli::sandbox::TransactionParams params = {
argv[opt_idx], argv[opt_idx + 1], verbose, quality, memlimit_mb};
guetzli::sandbox::GuetzliTransaction transaction(std::move(params));
auto result = transaction.Run();
if (!result.ok()) {
std::cerr << result.ToString() << std::endl;
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}

124
oss-internship-2020/guetzli/guetzli_sapi_test.cc Normal file
View File

@ -0,0 +1,124 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <syscall.h>
#include <algorithm>
#include <fstream>
#include <memory>
#include <sstream>
#include "guetzli_sandbox.h" // NOLINT(build/include)
#include "gtest/gtest.h"
#include "sandboxed_api/sandbox2/util/fileops.h"
#include "sandboxed_api/vars.h"
namespace guetzli::sandbox::tests {
namespace {
constexpr absl::string_view kInPngFilename = "bees.png";
constexpr absl::string_view kInJpegFilename = "nature.jpg";
constexpr absl::string_view kPngReferenceFilename = "bees_reference.jpg";
constexpr absl::string_view kJpegReferenceFIlename = "nature_reference.jpg";
constexpr int kDefaultQualityTarget = 95;
constexpr int kDefaultMemlimitMb = 6000;
constexpr absl::string_view kRelativePathToTestdata =
"/guetzli_sandboxed/testdata/";
std::string GetPathToInputFile(absl::string_view filename) {
return absl::StrCat(getenv("TEST_SRCDIR"), kRelativePathToTestdata, filename);
}
std::string ReadFromFile(const std::string& filename) {
std::ifstream stream(filename, std::ios::binary);
if (!stream.is_open()) {
return "";
}
std::stringstream result;
result << stream.rdbuf();
return result.str();
}
} // namespace
class GuetzliSapiTest : public ::testing::Test {
protected:
void SetUp() override {
sandbox_ = std::make_unique<GuetzliSapiSandbox>();
ASSERT_EQ(sandbox_->Init(), absl::OkStatus());
api_ = std::make_unique<GuetzliApi>(sandbox_.get());
}
std::unique_ptr<GuetzliSapiSandbox> sandbox_;
std::unique_ptr<GuetzliApi> api_;
};
// This test can take up to few minutes depending on your hardware
TEST_F(GuetzliSapiTest, ProcessRGB) {
sapi::v::Fd in_fd(open(GetPathToInputFile(kInPngFilename).c_str(), O_RDONLY));
ASSERT_TRUE(in_fd.GetValue() != -1) << "Error opening input file";
ASSERT_EQ(api_->sandbox()->TransferToSandboxee(&in_fd), absl::OkStatus())
<< "Error transfering fd to sandbox";
ASSERT_TRUE(in_fd.GetRemoteFd() != -1) << "Error opening remote fd";
sapi::v::Struct<ProcessingParams> processing_params;
*processing_params.mutable_data() = {
in_fd.GetRemoteFd(), 0, kDefaultQualityTarget, kDefaultMemlimitMb};
sapi::v::LenVal output(0);
sapi::StatusOr<bool> processing_result =
api_->ProcessRgb(processing_params.PtrBefore(), output.PtrBoth());
ASSERT_TRUE(processing_result.value_or(false)) << "Error processing rgb data";
std::string reference_data =
ReadFromFile(GetPathToInputFile(kPngReferenceFilename));
ASSERT_EQ(output.GetDataSize(), reference_data.size())
<< "Incorrect result data size";
ASSERT_EQ(
std::string(output.GetData(), output.GetData() + output.GetDataSize()),
reference_data)
<< "Processed data doesn't match reference output";
}
// This test can take up to few minutes depending on your hardware
TEST_F(GuetzliSapiTest, ProcessJpeg) {
sapi::v::Fd in_fd(
open(GetPathToInputFile(kInJpegFilename).c_str(), O_RDONLY));
ASSERT_TRUE(in_fd.GetValue() != -1) << "Error opening input file";
ASSERT_EQ(api_->sandbox()->TransferToSandboxee(&in_fd), absl::OkStatus())
<< "Error transfering fd to sandbox";
ASSERT_TRUE(in_fd.GetRemoteFd() != -1) << "Error opening remote fd";
sapi::v::Struct<ProcessingParams> processing_params;
*processing_params.mutable_data() = {
in_fd.GetRemoteFd(), 0, kDefaultQualityTarget, kDefaultMemlimitMb};
sapi::v::LenVal output(0);
sapi::StatusOr<bool> processing_result =
api_->ProcessJpeg(processing_params.PtrBefore(), output.PtrBoth());
ASSERT_TRUE(processing_result.value_or(false)) << "Error processing jpg data";
std::string reference_data =
ReadFromFile(GetPathToInputFile(kJpegReferenceFIlename));
ASSERT_EQ(output.GetDataSize(), reference_data.size())
<< "Incorrect result data size";
ASSERT_EQ(
std::string(output.GetData(), output.GetData() + output.GetDataSize()),
reference_data)
<< "Processed data doesn't match reference output";
}
} // namespace guetzli::sandbox::tests

123
oss-internship-2020/guetzli/guetzli_transaction.cc Normal file
View File

@ -0,0 +1,123 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "guetzli_transaction.h" // NOLINT(build/include)
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <iostream>
#include <memory>
namespace guetzli::sandbox {
absl::Status GuetzliTransaction::Main() {
sapi::v::Fd in_fd(open(params_.in_file, O_RDONLY));
if (in_fd.GetValue() < 0) {
return absl::FailedPreconditionError("Error opening input file");
}
SAPI_ASSIGN_OR_RETURN(image_type_, GetImageTypeFromFd(in_fd.GetValue()));
SAPI_RETURN_IF_ERROR(sandbox()->TransferToSandboxee(&in_fd));
if (in_fd.GetRemoteFd() < 0) {
return absl::FailedPreconditionError(
"Error receiving remote FD: remote input fd is set to -1");
}
GuetzliApi api(sandbox());
sapi::v::LenVal output(0);
sapi::v::Struct<ProcessingParams> processing_params;
*processing_params.mutable_data() = {in_fd.GetRemoteFd(), params_.verbose,
params_.quality, params_.memlimit_mb};
auto result =
image_type_ == ImageType::kJpeg
? api.ProcessJpeg(processing_params.PtrBefore(), output.PtrBefore())
: api.ProcessRgb(processing_params.PtrBefore(), output.PtrBefore());
if (!result.value_or(false)) {
return absl::FailedPreconditionError(absl::StrCat(
"Error processing ", (image_type_ == ImageType::kJpeg ? "jpeg" : "rgb"),
" data"));
}
sapi::v::Fd out_fd(open(".", O_TMPFILE | O_RDWR, S_IRUSR | S_IWUSR));
if (out_fd.GetValue() < 0) {
return absl::FailedPreconditionError("Error creating temp output file");
}
SAPI_RETURN_IF_ERROR(sandbox()->TransferToSandboxee(&out_fd));
if (out_fd.GetRemoteFd() < 0) {
return absl::FailedPreconditionError(
"Error receiving remote FD: remote output fd is set to -1");
}
auto write_result = api.WriteDataToFd(out_fd.GetRemoteFd(), output.PtrNone());
if (!write_result.value_or(false)) {
return absl::FailedPreconditionError("Error writing file inside sandbox");
}
SAPI_RETURN_IF_ERROR(LinkOutFile(out_fd.GetValue()));
return absl::OkStatus();
}
absl::Status GuetzliTransaction::LinkOutFile(int out_fd) const {
if (access(params_.out_file, F_OK) != -1) {
if (remove(params_.out_file) < 0) {
return absl::FailedPreconditionError(absl::StrCat(
"Error deleting existing output file: ", params_.out_file));
}
}
std::string path = absl::StrCat("/proc/self/fd/", out_fd);
if (linkat(AT_FDCWD, path.c_str(), AT_FDCWD, params_.out_file,
AT_SYMLINK_FOLLOW) < 0) {
return absl::FailedPreconditionError(
absl::StrCat("Error linking: ", params_.out_file));
}
return absl::OkStatus();
}
sapi::StatusOr<ImageType> GuetzliTransaction::GetImageTypeFromFd(int fd) const {
static const unsigned char kPNGMagicBytes[] = {
0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n',
};
char read_buf[sizeof(kPNGMagicBytes)];
if (read(fd, read_buf, sizeof(kPNGMagicBytes)) != sizeof(kPNGMagicBytes)) {
return absl::FailedPreconditionError(
"Error determining type of the input file");
}
if (lseek(fd, 0, SEEK_SET) != 0) {
return absl::FailedPreconditionError(
"Error returnig cursor to the beginning");
}
return memcmp(read_buf, kPNGMagicBytes, sizeof(kPNGMagicBytes)) == 0
? ImageType::kPng
: ImageType::kJpeg;
}
} // namespace guetzli::sandbox

59
oss-internship-2020/guetzli/guetzli_transaction.h Normal file
View File

@ -0,0 +1,59 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef GUETZLI_SANDBOXED_GUETZLI_TRANSACTION_H_
#define GUETZLI_SANDBOXED_GUETZLI_TRANSACTION_H_
#include <syscall.h>
#include "guetzli_sandbox.h" // NOLINT(build/include)
#include "sandboxed_api/transaction.h"
#include "sandboxed_api/vars.h"
namespace guetzli::sandbox {
enum class ImageType { kJpeg, kPng };
struct TransactionParams {
const char* in_file = nullptr;
const char* out_file = nullptr;
int verbose = 0;
int quality = 0;
int memlimit_mb = 0;
};
// Instance of this transaction shouldn't be reused
// Create a new one for each processing operation
class GuetzliTransaction : public sapi::Transaction {
public:
explicit GuetzliTransaction(TransactionParams params, int retry_count = 0)
: sapi::Transaction(std::make_unique<GuetzliSapiSandbox>()),
params_(std::move(params)) {
set_retry_count(retry_count);
SetTimeLimit(absl::InfiniteDuration());
}
private:
absl::Status Main() final;
absl::Status LinkOutFile(int out_fd) const;
sapi::StatusOr<ImageType> GetImageTypeFromFd(int fd) const;
const TransactionParams params_;
ImageType image_type_ = ImageType::kJpeg;
};
} // namespace guetzli::sandbox
#endif // GUETZLI_SANDBOXED_GUETZLI_TRANSACTION_H_

145
oss-internship-2020/guetzli/guetzli_transaction_test.cc Normal file
View File

@ -0,0 +1,145 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "guetzli_transaction.h" // NOLINT(build/include)
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <fstream>
#include <memory>
#include <sstream>
#include "gtest/gtest.h"
#include "sandboxed_api/sandbox2/util/fileops.h"
namespace guetzli::sandbox::tests {
namespace {
constexpr absl::string_view kInPngFilename = "bees.png";
constexpr absl::string_view kInJpegFilename = "nature.jpg";
constexpr absl::string_view kOutJpegFilename = "out_jpeg.jpg";
constexpr absl::string_view kOutPngFilename = "out_png.png";
constexpr absl::string_view kPngReferenceFilename = "bees_reference.jpg";
constexpr absl::string_view kJpegReferenceFIlename = "nature_reference.jpg";
constexpr int kPngExpectedSize = 38'625;
constexpr int kJpegExpectedSize = 10'816;
constexpr int kDefaultQualityTarget = 95;
constexpr int kDefaultMemlimitMb = 6000;
constexpr absl::string_view kRelativePathToTestdata =
"/guetzli_sandboxed/testdata/";
std::string GetPathToFile(absl::string_view filename) {
return absl::StrCat(getenv("TEST_SRCDIR"), kRelativePathToTestdata, filename);
}
std::string ReadFromFile(const std::string& filename) {
std::ifstream stream(filename, std::ios::binary);
if (!stream.is_open()) {
return "";
}
std::stringstream result;
result << stream.rdbuf();
return result.str();
}
// Helper class to delete file after opening
class FileRemover {
public:
explicit FileRemover(const char* path)
: path_(path), fd_(open(path, O_RDONLY)) {}
~FileRemover() {
close(fd_);
remove(path_);
}
int get() const { return fd_; }
private:
const char* path_;
int fd_;
};
} // namespace
TEST(GuetzliTransactionTest, TestTransactionJpg) {
std::string in_path = GetPathToFile(kInJpegFilename);
std::string out_path = GetPathToFile(kOutJpegFilename);
TransactionParams params = {in_path.c_str(), out_path.c_str(), 0,
kDefaultQualityTarget, kDefaultMemlimitMb};
{
GuetzliTransaction transaction(std::move(params));
absl::Status result = transaction.Run();
ASSERT_TRUE(result.ok()) << result.ToString();
}
std::string reference_data =
ReadFromFile(GetPathToFile(kJpegReferenceFIlename));
FileRemover file_remover(out_path.c_str());
ASSERT_TRUE(file_remover.get() != -1) << "Error opening output file";
off_t output_size = lseek(file_remover.get(), 0, SEEK_END);
ASSERT_EQ(reference_data.size(), output_size)
<< "Different sizes of reference and returned data";
ASSERT_EQ(lseek(file_remover.get(), 0, SEEK_SET), 0)
<< "Error repositioning out file";
std::string output;
output.resize(output_size);
ssize_t status = read(file_remover.get(), output.data(), output_size);
ASSERT_EQ(status, output_size) << "Error reading data from temp output file";
ASSERT_EQ(output, reference_data) << "Returned data doesn't match reference";
}
TEST(GuetzliTransactionTest, TestTransactionPng) {
std::string in_path = GetPathToFile(kInPngFilename);
std::string out_path = GetPathToFile(kOutPngFilename);
TransactionParams params = {in_path.c_str(), out_path.c_str(), 0,
kDefaultQualityTarget, kDefaultMemlimitMb};
{
GuetzliTransaction transaction(std::move(params));
absl::Status result = transaction.Run();
ASSERT_TRUE(result.ok()) << result.ToString();
}
std::string reference_data =
ReadFromFile(GetPathToFile(kPngReferenceFilename));
FileRemover file_remover(out_path.c_str());
ASSERT_TRUE(file_remover.get() != -1) << "Error opening output file";
off_t output_size = lseek(file_remover.get(), 0, SEEK_END);
ASSERT_EQ(reference_data.size(), output_size)
<< "Different sizes of reference and returned data";
ASSERT_EQ(lseek(file_remover.get(), 0, SEEK_SET), 0)
<< "Error repositioning out file";
std::string output;
output.resize(output_size);
ssize_t status = read(file_remover.get(), output.data(), output_size);
ASSERT_EQ(status, output_size) << "Error reading data from temp output file";
ASSERT_EQ(output, reference_data) << "Returned data doesn't match refernce";
}
} // namespace guetzli::sandbox::tests

BIN
oss-internship-2020/guetzli/testdata/bees.png vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 173 KiB

BIN
oss-internship-2020/guetzli/testdata/bees_reference.jpg vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
oss-internship-2020/guetzli/testdata/nature.jpg vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

BIN
oss-internship-2020/guetzli/testdata/nature_reference.jpg vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB