Commit Graph

117 Commits (main)

Author SHA1 Message Date
Dmitri Gribenko 4d34bdb145 Integrate LLVM at llvm/llvm-project@3287ae8f65
Updates LLVM usage to match
[3287ae8f6520](https://github.com/llvm/llvm-project/commit/3287ae8f6520)

PiperOrigin-RevId: 586379104
Change-Id: Id69e27932def712bbb0f4a58112dfeb5b4f76275
2023-11-29 10:14:05 -08:00
Wiktor Garbacz 4630346fd1 Fix unaligned load
PiperOrigin-RevId: 577106224
Change-Id: Icd0e113031d367fa30557d4e77571abc78e8e378
2023-10-27 00:35:10 -07:00
Wiktor Garbacz 127176d72f Bulk IWYU and build_cleaner fixes
PiperOrigin-RevId: 559733768
Change-Id: Ia38f4c176e9f0abbfdb3a8f1109f482d8870eb0f
2023-08-24 06:23:36 -07:00
Christian Blichmann f378d22405 Clang tool: Skip protobuf namespaces when emitting headers
The protobuf internals are not needed in the API header and will be made
available as soon as user code includes any generated proto header.

PiperOrigin-RevId: 557749772
Change-Id: Idc48a652ab1892dae559192afbde20ae34e4c7ce
2023-08-17 01:58:33 -07:00
Christian Blichmann c501379056 Clang tool: Prevent extra nesting of namespaces
When specifying the `namespace` argument in Bazel (`NAMESPACE` in CMake), the
Clang tool used to put _all_ dependent types in that namespace.

For a declaration of `namespace a::b { struct S {...};` and a `namespace`
argument of `a::b`, this means that the header output was similar to
```
namespace a::b {
namespace a::b {
struct S { ...
```

This was never intended and also does not match the Python based header
generator. The Clang tool now "merges" those same namespaces. This is
correct, as it processes `namespace`d spellings with their full namespace
path.

PiperOrigin-RevId: 557393076
Change-Id: I1474dd30b6c4150d0ae3c1c48579f88060974980
2023-08-16 01:17:54 -07:00
Christian Blichmann ae3d334cc2 generator2: Skip anonymous structs/unions
The Python code has been relying on `spelling` to return `None` for skipping
anonymous structs/unions.

libclang has been returning a "spelling" for those for a while now (LLVM 16
introduced this in its branch in 2022), though, so this check no longer works.
Use the correct method `clang.CIndex.is_anonymous()` instead.

PiperOrigin-RevId: 557099905
Change-Id: I13707509dbae03481c5edce7fa92554cefdd57e7
2023-08-15 05:20:47 -07:00
Christian Blichmann 352d1f8fb2 Clang tool: Emit aggregates with default initialized members
PiperOrigin-RevId: 556765694
Change-Id: I2547919cdc1fcb048c99de325a8b2c24800b0e06
2023-08-14 06:14:43 -07:00
Sandboxed API Team a3fa7d27d5 Internal Code Change
PiperOrigin-RevId: 547689091
Change-Id: I76ddcaefcc50f8ce706d59dae99877ca6f28544d
2023-07-12 22:13:27 -07:00
Christian Blichmann 04cb14791e Clang tool: Enable incremental pre-processing
This avoids doing extra work when processing multiple input files.

PiperOrigin-RevId: 539884025
Change-Id: I8e48495f33c09bc53e70f4d5c1d730fe7c1202b2
2023-06-13 01:04:38 -07:00
Christian Blichmann f2048d028f Clang tool: Force-undefine feature preprocessor defines
To avoid code that is being parsed to include the intrinsics headers, undefine
a few key preprocessor defines.

PiperOrigin-RevId: 539878995
Change-Id: I8afb7cbdadcab3214c943c0acb9006e8bcc30611
2023-06-13 00:38:05 -07:00
Christian Blichmann 045ace8dcb Update Google dependencies
- Abseil
- Protobuf
- Benchmark
- Googletest

In turn, some code changes were necessary:
- Use absolute imports in `sapi_generator.py` when invoked by Bazel
- Add Abseil's source dir as include dir in generated proto `.cc` files
- Bazel: Use `@rules_proto` for `proto_library` and use native `cc_proto_library`

Drive-by:
- Update year in `README.md`
- Look for clang versions 16, 15, 14, and 13 as well in `code.py`
PiperOrigin-RevId: 539032012
Change-Id: Ib9cd1d7fb38409d884eb45e1fa08927f6af83a21
2023-06-09 03:22:00 -07:00
Christian Blichmann 72452e1582 Mostly internal change: Optimize OSS transforms
This should only affect the Bazel `BUILD.bazel` files and their formatting.

PiperOrigin-RevId: 538426054
Change-Id: I0162726d3fb4bcb4d7938cddc6f39e0d9f2b4a3d
2023-06-07 02:23:18 -07:00
Christian Blichmann 1bf9437f95 Add GitHub workflow to build Clang tool based header generator
Drive-by:
- Add flags to link libgcc and libstdc++ statically into the binary, making it
  "mostly static"
PiperOrigin-RevId: 532349354
Change-Id: I0a86eb29b6a40aec4cec3cffeaf9511726ee4dc8
2023-05-15 23:45:57 -07:00
Christian Blichmann 4925df5419 CMake: Add option to link the Clang libraries statically into the header generator
Tested on Debian 10.13 with `LLVM-{11,12,13,14,15,16,17}` packages from https://apt.llvm.org/.

PiperOrigin-RevId: 531211601
Change-Id: I91babb5d85be2a22a4b17d757a5f626de6c03881
2023-05-11 08:36:11 -07:00
Christian Blichmann a078043f8e CMake: Increase minimum required LLVM version to 11
LLVM 11 is now preseent in all major stable Linux distributions.

PiperOrigin-RevId: 531204137
Change-Id: I6f20aea425915023ea6113c17ff5a038a74aa919
2023-05-11 08:04:59 -07:00
Wiktor Garbacz a5bad44fac Fix wrong pytype annotation
PiperOrigin-RevId: 520972266
Change-Id: Ib5775e01bf3389e7d123480b3bb3b7a4f33a07b0
2023-03-31 11:30:33 -07:00
Dmitri Gribenko 858c407521 Replace llvm::makeArrayRef with ArrayRef constructors.
LLVM upstream is about to deprecate and remove llvm::makeArrayRef.

PiperOrigin-RevId: 501106516
Change-Id: Ice610c7e0190dc8608339de1b88b7a05e7771871
2023-01-10 15:14:03 -08:00
Christian Blichmann d31e691705 clang_generator: Implement `limit_scan_depth` feature
PiperOrigin-RevId: 490517864
Change-Id: I93c5df370501f9af745b53791ae8ac2a18220ff9
2022-11-23 09:09:50 -08:00
Christian Blichmann e223ef4790 clang_generator: Skip Abseil internal functions
PiperOrigin-RevId: 490513265
Change-Id: I7ddcdf612772167756039cb8f59b5b3aa60ed2ef
2022-11-23 08:47:47 -08:00
Christian Blichmann cded8655d3 clang_generator: Skip dependent types
PiperOrigin-RevId: 490508865
Change-Id: Ic1d49a4abfde4cd02f08dd24c3f7a20058ef4965
2022-11-23 08:32:19 -08:00
Christian Blichmann d7fe6cd334 clang_generator: Use fully qualified names
Use locally unqualified types to filter ordered type declarations in
`TypeCollector::GetTypeDeclarations()`. This is necessary, as
`clang::TypeName::getFullyQualifiedName()` and
`TypeDecl::getQualifiedNameAsString()` have different ideas which
qualifiers belong to the name. The former works on `QualType`s, while
the latter deals with the declaration directly. This change decays a
`TypeDecl` into its locally unqualified `QualType`.

PiperOrigin-RevId: 490500091
Change-Id: Ie2f4eece4e673f8b06ab6661d7b6611daf34fba9
2022-11-23 07:54:48 -08:00
Christian Blichmann 37ca6d0fc6 clang_generator: Access type collector/function via accessors
PiperOrigin-RevId: 490476261
Change-Id: Icbd51d3792c33dcfb185ec0514118a095135b3f6
2022-11-23 05:21:25 -08:00
Christian Blichmann 13c5b564b6 clang_generator: Collect function types directly
Do not try to peel off all layers of pointers/references when collecting
types, recursion works well here. Also collect the function type itself,
even if it is a pointer or if the function was the underlying type of a
typedef.

PiperOrigin-RevId: 490475937
Change-Id: I13cb3d9d3de7d25b9e627f43112c46758c7e6a22
2022-11-23 05:19:10 -08:00
Sandboxed API Team 3155cb0a67 Fix format string and/or arguments to match.
PiperOrigin-RevId: 488378036
Change-Id: I1e3dfa9a734f4c682f1b388f6ba54bf89bd98920
2022-11-14 08:51:06 -08:00
Christian Blichmann 67bc67bbef clang_generator: Handle inter-type dependencies better
This change changes the emitter to work on `clang::TypeDecl`s instead of
`clang::QualType`s, as the latter only describe complete definitions. This
allows us to better deal with situations where we would otherwise have a kind
of circular type dependencies: For example, a `typedef` that depends on a
`struct` which in turn has a member that references the typedef.
For this to work, we now also record the actual source order of declarations,
similar to what the libclang based header generator does.

Also add some more tests for the newly covered cases.

PiperOrigin-RevId: 488370914
Change-Id: I2d1baa863fb12d1801bf11a20f3f88df7fde3129
2022-11-14 08:19:13 -08:00
Christian Blichmann 1ae04ac332 clang_generator: Use fully-qualified names, emit in source order
When generating headers from libraries that export functions as `extern "C"`
but still use namespaces (C-compatible C++ libraries), we want to generate
a Sandboxed API that includes fully-qualified namespace names as well.

In addition, we want the generated API to have the same source order as the
original library. Not only is this less surprising when reading the generated
code, it's also more accurate. Previously, we'd bundle all definitions in a
namespace and sort those alphabetically, but for code that relies on symbols
from another namespace to be available, generation will fail:

```c++
namespace zzz {
using entity_count_t = uint64_t;
}  // namespace zzz
namespace sheep_counter {
using sheep_count_t = :💤:entity_count_t;
extern "C" void IncreaseSheepCounter(sheep_count_t increment);
}  // namespace sheep_counter
```

PiperOrigin-RevId: 486586024
Change-Id: I419c9db8e9cb5b904364b353e2dc3d7f1030fab3
2022-11-07 00:37:53 -08:00
Christian Blichmann ce26b55e26 clang_generator: Turn fatal error on struc-by-value into warning
An `absl::StatusCode` of `kCancelled` now indicates warnings inside the emitter.

PiperOrigin-RevId: 485851898
Change-Id: I10a57cbc1b6c2d4b708c3c19aa0fa71451845a22
2022-11-03 06:02:32 -07:00
Rebecca Chen 3abfefaf3b Silence some pytype errors.
PiperOrigin-RevId: 485526322
Change-Id: I55541b3f7c2f8e89efef0d37bbd2ef4e330bd3e7
2022-11-02 01:08:49 -07:00
Christian Blichmann 4b56d5606d clang_generator: Fail header generation with struct-by-value
The libclang based header generator disallows functions that pass structs (or
more generally "record types") by value. While this can be implemented, the
such functions as emitted by the clang_generator never worked.

We should revisit this when we implement support for passing 128-bit integer
types directly, as those will work the same as small structs.

PiperOrigin-RevId: 485522603
Change-Id: Iae8284720da52496d7a48fe3ca3c3c8605e6d19d
2022-11-02 00:43:26 -07:00
Christian Blichmann 7e0f72e445 clang_generator: Refactor frontend test utilities
This makes it easier to write tests with expected compilation/parsing failures.

PiperOrigin-RevId: 485331205
Change-Id: Ia545934849d38bded9320537c71e970fb4730cb6
2022-11-01 09:00:34 -07:00
Christian Blichmann 4c87556901 Use Abseil's log/flags instead of glog/gflags
Follow-up changes might be required to fully fix up the contrib sandboxes.

PiperOrigin-RevId: 482475998
Change-Id: Iff631eb838a024b2f047a1be61bb27e35a8ff2f4
2022-10-20 06:48:51 -07:00
Benjamin Kramer 7dca070083 Integrate LLVM at llvm/llvm-project@06da9b94ae
Updates LLVM usage to match
[06da9b94ae37](https://github.com/llvm/llvm-project/commit/06da9b94ae37)

PiperOrigin-RevId: 481181270
Change-Id: I8c015fbd95d7f5f543d4eed12ed5d4efc11ef9c3
2022-10-14 10:42:23 -07:00
Christian Blichmann 79b6784b82 #Cleanup: Consistently use `std::make_unique`
PiperOrigin-RevId: 480597371
Change-Id: I145586382ad7a7694384cc672986132376a47465
2022-10-12 05:23:42 -07:00
Christian Blichmann 39a1bc9d7a Skip system headers in Clang generator
When not requesting any particular function, `sapi_library()` will try and
make available _all_ functions it finds. In this case, system headers should
be skipped to avoid inflating the API surface. Standard library functions
can still be manually requested by adding them to the `functions` (Bazel)/
`FUNCTIONS` (CMake) argument.

PiperOrigin-RevId: 472272506
Change-Id: I8f8d79796d3044e598eebb7f87ce4cf464b47ed7
2022-09-05 07:15:46 -07:00
Sandboxed API Team 26b2519aed Integrate LLVM at llvm/llvm-project@ea460b7ddb
Updates LLVM usage to match
[ea460b7ddb8a](https://github.com/llvm/llvm-project/commit/ea460b7ddb8a)

PiperOrigin-RevId: 465026637
Change-Id: Ie32d8e89e2824b6e7d4b3f8f1588e55b6c72bb1e
2022-08-03 04:55:55 -07:00
Christian Blichmann 69ed3d6946 clang_generator: Fix build breakage with most recent LLVM
PiperOrigin-RevId: 447443100
Change-Id: I7b7ca475be159dc5e2c2f4e6f1596ff7bb0438e2
2022-05-09 05:24:25 -07:00
Christian Blichmann 6cbde854d6 #Cleanup: Consistently use `char* argv[]` instead of `char**`
PiperOrigin-RevId: 444782296
Change-Id: If8e7647be28f794392675ae001abbe9b809da0ac
2022-04-27 00:43:51 -07:00
Christian Blichmann 839914d6dd cmake: Rename build options to follow Abseil naming
`BUILD_TESTING` is a CMake provided option and we should use similar naming,
just like how Abseil does it.

- `SAPI_ENABLE_TESTS` -> `SAPI_BUILD_TESTING`
- `SAPI_ENABLE_CONTRIB_TESTS` -> `SAPI_CONTRIB_BUILD_TESTING`
- `SAPI_ENABLE_EXAMPLES` -> `SAPI_BUILD_EXAMPLES`

Drive-by:
- Fix option name in GitHub action
PiperOrigin-RevId: 443305932
Change-Id: Ice2b42be1229a0f9ae7c2ceda9ce87187baf22c4
2022-04-21 01:17:39 -07:00
Christian Blichmann c0cfeed925 cmake: Include `CTest` in all projects, honor `BUILD_TESTING` setting
Including the `CTest` modules ensures that the `BUILD_TESTING` option is
defined and automatically calls `enable_testing()` if needed. It does not
change the default or introduce any dependencies on its own.

This follows what Abseil already does in their top-level `CMakeLists.txt`.

PiperOrigin-RevId: 443305646
Change-Id: If067c17470f497437c7748aab4aab5227c26e84f
2022-04-21 01:15:34 -07:00
Christian Blichmann 0d833be608 clang_generator: Remove `.isystem` parameter file support
The corresponding command-line option `sapi_isystem` will be ignored for
compatibility.

PiperOrigin-RevId: 439806387
Change-Id: I8ad6d7feed2fba5fca9940281f03cfc757ada5be
2022-04-06 05:17:44 -07:00
Christian Blichmann 3f72b3cab7 bazel: Set generator version to default v1
No change in behavior.

PiperOrigin-RevId: 438766481
Change-Id: Iaf52d32856007fb83ee0032d4d5097482f45f47d
2022-04-01 01:45:50 -07:00
Christian Blichmann 4b2c730c5e Bazel: Make generator configurable
This change introduces an experimental `generator_version` attribute to the
`sapi_library()` rule. Version `1` will select the current interface
generator, which is based on libclang and Python. Setting the attribute to
version `2`, will select the newer interface generator written in C++ that
uses a full clang compiler frontend for parsing. Both emit equivalent header
output, differences in parsing and/or edge cases notwithstanding.

The default, as of now, is still the old version `1` generator.

Note: CMake allows to select the new interface generator globally by setting
`SAPI_ENABLE_GENERATOR`.
PiperOrigin-RevId: 438765013
Change-Id: I69c49a6bcf1751724edb0bce5c3b2beea2097138
2022-04-01 01:34:48 -07:00
Christian Blichmann 74bb2c35ca bazel: Add build rules for the new interface generator
This adds a workspace rule that inspects the current system first and
downloads a suitable version of LLVM/Clang from GitHub if it can't
find one. In the latter case, the necessary parts are build from source,
which can take a while (~10-15m, depending on the build machine).

In order to be found, LLVM/Clang system libraries must be version 11
or higher. On Debian/Ubuntu, install `llvm-13-dev` and `libclang-13-dev`.

The new `llvm_config.bzl` implements this logic. It is loosely based on
upstream's https://github.com/llvm/llvm-project/blob/main/utils/bazel/configure.bzl.
Note that due to the way Bazel separates local repositories, we have to
duplictate some of this code.

PiperOrigin-RevId: 438759950
Change-Id: Ia65f473b4cdef6507e3816bf09794ea10963d87a
2022-04-01 00:55:30 -07:00
Christian Blichmann dd512a1253 clang_generator: Map C's `_Bool` to `bool`
Fixes #154.

PiperOrigin-RevId: 436167183
Change-Id: I594e5b59c5b3aba640a98c399dcb994000dbffb0
2022-03-21 04:11:21 -07:00
Christian Blichmann 6d3adc45f7 clang_generator: Improve formatting of function prototype comments
This uses the Google formatting style to format the prototype comments, with an
internal line length of 75, which accomodates the indentation in the generated
API class.

PiperOrigin-RevId: 435303665
Change-Id: I4dcdf0ed773a79ebc55ead3843f07ca8556fd985
2022-03-17 03:53:13 -07:00
Christian Blichmann e8fe398340 clang_generator: Enable mixed header processing
This implements a custom compilation database to conditionally add the correct
language flags to the compiler frontend. Otherwise, a C header might receive
`--std=c++17` and fail.

Note: All headers are always processed in C++ mode. We expect that headers of
well-behaved C libraries contain `#ifdef __cplusplus`/`extern "C" {}` guards.
PiperOrigin-RevId: 435302048
Change-Id: Ib84e6e1f301ba434999846a012b3f8c16884648e
2022-03-17 03:41:53 -07:00
Christian Blichmann 4e71f1d0a3 clang_generator: Do not collect structs/unions if declared in another record
The enclosing type is enough to reconstruct the AST when writing the header and this
change avoids emitting the same struct twice.

PiperOrigin-RevId: 435300029
Change-Id: I34bd660db5ba5c68b64cce73ecf2f026727ac57b
2022-03-17 03:30:12 -07:00
Christian Blichmann 92ccfeae67 clang_generator: Correctly emit nested C-like structs
- Add more tests for this

PiperOrigin-RevId: 435296715
Change-Id: I7b42dbc58dc054d2565af9ad22498d98416b7af7
2022-03-17 03:13:25 -07:00
Christian Blichmann b8579e4746 clang_generator: Use a `btree_map` for fixed iteration order
PiperOrigin-RevId: 435291910
Change-Id: I198247409d095183849a221af9c3be21b5bb859b
2022-03-17 02:43:08 -07:00
Christian Blichmann d5ebb81598 clang_generator: Handle intrinsics directly
PiperOrigin-RevId: 435289903
Change-Id: I68a37bb7b25a7b77c046d00a2740aa9de2fcaa89
2022-03-17 02:31:57 -07:00