- CramHacks
- Posts
- Gradle: Generating Multi-Project Lockfiles
Gradle: Generating Multi-Project Lockfiles
Doing things that aren't intended to be done but should be
If you’re reading this, there’s a good chance you are a Gradle user, and to that … I would say I’m sorry.
Disclaimer: I am not a Java developer, nor do I claim to be. I will say that I’ve had fewer headaches with Maven, but as I understand it, that is likely because Maven just tends to work for “standard” projects, whereas Gradle offers support for more complex builds.
This blog post will cover my journey to generate lockfiles for a Gradle multi-project, elasticsearch, in order to run Semgrep Supply Chain on the project. Did you know that the entire suite of Semgrep products is free to use for public repositories?
If you’re familiar with Multi-Project Builds, build.gradle
, and gradle.lockfile
- I’d suggest jumping to Scripting Lockfile generation: Elasticsearch.
Multi-Project Builds
In software development, one approach to managing complex applications is through multi-project builds.
This method involves dividing a larger project into several subprojects or modules. Each subproject typically has its own build.gradle
file and set of dependencies, allowing for more modular and compartmentalized development.
The structure of multi-project builds enables individual management and building of each component while maintaining their interconnections. This is especially useful in scenarios where a project consists of multiple interrelated modules that need to be integrated into a single cohesive system. Unlike smaller or monolithic projects that may use a single build file, this approach provides a way to organize and handle larger projects with interconnected parts, each requiring separate development and maintenance efforts.
build.gradle & gradle.lockfile
In the context of multi-project builds, the build.gradle
and gradle.lockfile
files play pivotal roles in managing and ensuring the consistency of these builds.
build.gradle
is the cornerstone of each subproject or module within a larger project. It acts as a script for the Gradle build system, defining how the project is built, tested, and packaged. Each build.gradle
file can specify its own set of dependencies, plugins, and build configurations, enabling modular and compartmentalized development. This modularity allows developers to tailor the build process to the specific needs of each subproject, from compiling code and packaging binaries to executing tests.
On the other hand, gradle.lockfile
plays a crucial role in maintaining the stability and consistency of the project. It is a part of Gradle's dependency-locking mechanism, which records the exact versions of dependencies used in the build. The lockfile ensures that every build uses the same versions of dependencies, preventing discrepancies that can arise from version updates or differences in local development environments. By doing so, gradle.lockfile
enhances the reliability and reproducibility of the builds across different machines and environments.
Together, build.gradle
and gradle.lockfile
forms a comprehensive system for managing multi-project builds. While build.gradle
allows for customized and independent development of each module, gradle.lockfile
ensures that this independence does not compromise the consistency and integrity of the overall project. This combination is particularly beneficial for larger projects with interconnected modules, as it provides both flexibility in development and stability in builds, ensuring that the integrated system functions as a cohesive whole.
Generating a gradle.lockfile
In smaller or monolithic projects, since you typically only have a single build file, this process is very straightforward.
In the build.gradle
file, add the following lines:
dependencyLocking {
lockAllConfigurations()
}
Once added, run the following command to then create a gradle.lockfile
gradle dependencies --write-locks
And you’re done. Now, I could live with that! However, the Gradle documentation has a special mention: “Note that in a multi project setup, dependencies only is executed on one project, the root one in this case.”
This led me to spend wayyyyy too much time trying to figure out how to quickly generate lockfiles for all the projects/subprojects in the elasticsearch project.
Scripting Lockfile generation: Elasticsearch
I’ll walk you through my journey, but I’ll leave out most of the parts where I was banging my head into my desk. I spent more time with Gradle today than the rest of my life combined, so I wouldn’t at all be surprised if there’s an easier way. If there is, I’d love to know about it!
Firstly, I cloned the elasticsearch repository. Out of curiosity, I checked how many total build.gradle
files were in the project.
find . -name 'build.gradle' | wc
450 450 21247
😲 seriously??? I don’t even know if this is a lot for a Gradle project, but that would be a lot of files to manually edit and then subsequently generate lockfiles.
So, what can we do? Well, there’s a file called settings.gradle
, which defines which modules go into the final build. When we check for these in the project, there are only 6
total - that’s a lot more manageable.
What’s even better, is when reviewing each of these, only the root settings.gradle
file seems to list projects. Unfortunately, I noticed that there were many projects indirectly added, meaning they are not in the convenient List projects = […]
So we’re back to square one. This led me to discover the gradle projects
command, which is incredibly frustrating because it works. Why would you create this and not support multi-project lockfile generations 🥹. Anyway, I used this and filtered out the project names to save them to a file projects.txt
. It’s worth noting that this only results in 428
projects - don’t ask me where the other 22
went.
gradle projects | grep "Project" | awk -F"'" '{gsub(/^:/, "", $2); print $2}' > projects.txt
Then, I created a script that would enable me to generate a gradle.lockfile
for each project. This script does the following:
Walks through the directories and finds every
build.gradle
and appendsdependencyLocking {lockAllConfigurations()}
Walks through the directories and finds every
verification-metadata.xml
file and deletes it
*** Projects maintained for build-reproducibility and already using lockfiles shouldn’t require thisThen we take each project name from our
projects.txt
file and append:dependencies --write-locks
to each of them before passing them toGradle
import os
import subprocess
def add_dependency_locking(build_file):
"""Add dependencyLocking block to build.gradle if not present."""
with open(build_file, 'r+') as file:
content = file.read()
if 'dependencyLocking {' not in content:
file.write("\ndependencyLocking {\n lockAllConfigurations() \n}// will lock all project configurations\n")
def run_gradle_command(command):
"""Run the gradle command with the specified argument."""
full_command = f"gradle {command}"
try:
result = subprocess.run(full_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
print(f"Command executed successfully: {command}")
print(result.stdout.decode())
except subprocess.CalledProcessError as e:
print(f"Error executing command: {command}")
print(e.stderr.decode())
def main():
# Delete verification-metadata.xml files
for root, dirs, files in os.walk('.'):
for file in files:
if file == 'verification-metadata.xml':
os.remove(os.path.join(root, file))
# Process each build.gradle file
for root, dirs, files in os.walk('.'):
for file in files:
if file == 'build.gradle':
build_file_path = os.path.join(root, file)
print(f"Processing {build_file_path}")
# Add dependency locking block if needed
add_dependency_locking(build_file_path)
print("All build.gradle files processed successfully.")
# Run commands from projects.txt
with open('projects.txt', 'r') as file:
for line in file:
line = line.strip() # Remove any leading/trailing whitespace
if line: # Check if the line is not empty
run_gradle_command(line + ":dependencies --write-locks")
print("All commands from projects.txt executed.")
if __name__ == "__main__":
main()
What you’re left with is not 450
gradle.lockfiles
, not 428
, but 359
grade.lockfiles
. Why? I’m not all too sure. But I’m okay with these results 🤣.
Now I can run semgrep ci --supply-chain
to detect each of these lockfiles, parse each dependency and version, and identify known vulnerabilities. With Semgrep, I was also able to generate a SBOM (Beta) in CycloneDX format.
GitHub makes it incredibly convenient to see known dependencies and export an SBOM for standard projects, but that definitely doesn’t work here. Semgrep reported >25,000
dependencies, whereas GitHub’s Dependency Graph for the project shows 6
- additionally, using the export SBOM button results in {"error":"Not Found"}
.
This isn’t to say Semgrep handled this Gradle multi-project repository particularly well. It didn’t handle it at all - hence why I’m writing this blog on how to script the nastiness 😆.