Creating a Simplified C++ Package Manager in Rust
Written on
Chapter 1: The Challenge of Dependency Management
Managing dependencies in C and C++ projects can be incredibly cumbersome. Each library tends to come with its own build system, and while many rely on CMake, there's no uniformity.
To work around this, I often vendor dependencies by utilizing git submodules and compiling them statically. However, as projects grow and include transitive dependencies alongside nested git submodules, the situation becomes increasingly complex.
For instance, if my project directly depends on a library that is also a transitive dependency, using add_subdirectory in CMake with nested submodules leads to the same library being pulled in and built multiple times—an inefficient use of resources.
Additionally, if a transitive dependency is updated, I am forced to refresh the entire tree of nested git submodules, which is a hassle. Compounding this, I must integrate each dependency's build system into my own to ensure everything operates smoothly—a tedious task indeed.
Thus, I've decided to develop my own package manager!
Section 1.1: Inspiration Behind the Package Manager
I aim for this package manager to be as user-friendly as Cargo, with the following features:
- Vendoring both direct and transitive dependencies, similar to npm.
- Ensuring a dependency is fetched and built just once.
- Generating a distributable tar.gz archive, akin to Mix.
- Being portable and versatile across multiple programming languages.
- Avoiding the need to create a new build system.
The last point is crucial; existing build systems like Make, CMake, Ninja, and Meson are already adept at handling complex workflows, such as conditional compilation and platform detection. I have no intention of reinventing the wheel. Instead, this package manager will leverage the build systems of existing libraries to manage dependency fetching and building.
Section 1.2: Introducing Shipp
Let’s name this package manager Shipp, as my goal is to efficiently ship software. To convert your project into a Shipp package, simply add a shipp.json file to the root of your Git repository:
{
"name": "mypackage",
"version": "0.1.0",
"scripts": {
"build": "command to build",
"install": "command to install"
},
"dependencies": [
{
"name": "libfoo",
"version": "v0.1.0"
}
]
}
For example, here’s the manifest for Shipp:
{
"name": "shipp",
"version": "0.1.0",
"scripts": {
"build": "cargo build --release",
"install": "cargo install --path . --root $SHIPP_DIST_DIR"
}
}
One major advantage of not being a build system is that Shipp isn’t restricted to C or C++; it can also build projects in Rust, Zig, Nim, or any other language.
Shipp provides four essential subcommands:
- shipp deps.get: This command will clone or pull each direct dependency and recursively fetch all transitive dependencies, placing them in the .shipp/deps/ folder.
- shipp deps.build: This subcommand builds each dependency in the correct order by executing the scripts.build command from the manifest and installs the dependencies in the .shipp/dist folder.
- The .shipp/dist folder will contain:
- bin: for executables
- lib: for static and/or shared libraries
- include: for C/C++ headers
- share: for additional resources
- The path to the .shipp/dist folder can be accessed via the $SHIPP_DIST_DIR environment variable, enabling your build system to utilize it for compiler flags like -Iincludedir or -Llibdir.
- The .shipp/dist folder will contain:
- shipp build: This command builds your project and installs it in the .shipp/dist folder, just like the dependencies.
- shipp dist: This will package the contents of the .shipp/dist folder into a tar.gz archive for distribution.
Chapter 2: Future Directions for Shipp
While I do not have a concrete roadmap, it would be exciting to introduce "features" for conditional compilation or to accommodate package sources beyond Git. So far, this development addresses a personal need, but if you see potential for growth, any contributions would be greatly appreciated!
The first video titled "C++ Package Manager - C++ Dependencies Don't Have To Be Painful!" by Augustin Popa discusses the challenges and solutions related to managing C++ dependencies efficiently.
In the second video, "C++ Package Manager - C++ Dependencies Don't Have To Be Painful! - ACCU 2023," Augustin Popa delves deeper into how a package manager can alleviate these burdens in C++ development.