Gradle: Generating Multi-Project Lockfiles

If you’re reading this, there’s a good chance you are a Gradle user, and to that … I would say I’m sorry.

Disclaimer: I am not a Java developer, nor do I claim to be. I will say that I’ve had fewer headaches with Maven, but as I understand it, that is likely because Maven just tends to work for “standard” projects, whereas Gradle offers support for more complex builds.

This blog post will cover my journey to generate lockfiles for a Gradle multi-project, elasticsearch, in order to run Semgrep Supply Chain on the project. Did you know that the entire suite of Semgrep products is free to use for public repositories?

If you’re familiar with Multi-Project Builds, build.gradle, and gradle.lockfile - I’d suggest jumping to Scripting Lockfile generation: Elasticsearch.

Multi-Project Builds

In software development, one approach to managing complex applications is through multi-project builds.

This method involves dividing a larger project into several subprojects or modules. Each subproject typically has its own build.gradle file and set of dependencies, allowing for more modular and compartmentalized development.

The structure of multi-project builds enables individual management and building of each component while maintaining their interconnections. This is especially useful in scenarios where a project consists of multiple interrelated modules that need to be integrated into a single cohesive system. Unlike smaller or monolithic projects that may use a single build file, this approach provides a way to organize and handle larger projects with interconnected parts, each requiring separate development and maintenance efforts.

build.gradle & gradle.lockfile

In the context of multi-project builds, the build.gradle and gradle.lockfile files play pivotal roles in managing and ensuring the consistency of these builds.

build.gradle is the cornerstone of each subproject or module within a larger project. It acts as a script for the Gradle build system, defining how the project is built, tested, and packaged. Each build.gradle file can specify its own set of dependencies, plugins, and build configurations, enabling modular and compartmentalized development. This modularity allows developers to tailor the build process to the specific needs of each subproject, from compiling code and packaging binaries to executing tests.

On the other hand, gradle.lockfile plays a crucial role in maintaining the stability and consistency of the project. It is a part of Gradle's dependency-locking mechanism, which records the exact versions of dependencies used in the build. The lockfile ensures that every build uses the same versions of dependencies, preventing discrepancies that can arise from version updates or differences in local development environments. By doing so, gradle.lockfile enhances the reliability and reproducibility of the builds across different machines and environments.

Together, build.gradle and gradle.lockfile forms a comprehensive system for managing multi-project builds. While build.gradle allows for customized and independent development of each module, gradle.lockfile ensures that this independence does not compromise the consistency and integrity of the overall project. This combination is particularly beneficial for larger projects with interconnected modules, as it provides both flexibility in development and stability in builds, ensuring that the integrated system functions as a cohesive whole.

Generating a gradle.lockfile

In smaller or monolithic projects, since you typically only have a single build file, this process is very straightforward.

In the build.gradle file, add the following lines:

dependencyLocking {
    lockAllConfigurations()
}

Once added, run the following command to then create a gradle.lockfile

gradle dependencies --write-locks

And you’re done. Now, I could live with that! However, the Gradle documentation has a special mention: “Note that in a multi project setup, dependencies only is executed on one project, the root one in this case.”

This led me to spend wayyyyy too much time trying to figure out how to quickly generate lockfiles for all the projects/subprojects in the elasticsearch project.

Scripting Lockfile generation: Elasticsearch

I’ll walk you through my journey, but I’ll leave out most of the parts where I was banging my head into my desk. I spent more time with Gradle today than the rest of my life combined, so I wouldn’t at all be surprised if there’s an easier way. If there is, I’d love to know about it!

Firstly, I cloned the elasticsearch repository. Out of curiosity, I checked how many total build.gradle files were in the project.

find . -name 'build.gradle' | wc 
     450     450   21247

😲 seriously??? I don’t even know if this is a lot for a Gradle project, but that would be a lot of files to manually edit and then subsequently generate lockfiles.

So, what can we do? Well, there’s a file called settings.gradle, which defines which modules go into the final build. When we check for these in the project, there are only 6 total - that’s a lot more manageable.

What’s even better, is when reviewing each of these, only the root settings.gradle file seems to list projects. Unfortunately, I noticed that there were many projects indirectly added, meaning they are not in the convenient List projects = […]

So we’re back to square one. This led me to discover the gradle projects command, which is incredibly frustrating because it works. Why would you create this and not support multi-project lockfile generations 🥹. Anyway, I used this and filtered out the project names to save them to a file projects.txt. It’s worth noting that this only results in 428 projects - don’t ask me where the other 22 went.

gradle projects | grep "Project" | awk -F"'" '{gsub(/^:/, "", $2); print $2}' > projects.txt

Then, I created a script that would enable me to generate a gradle.lockfile for each project. This script does the following:

1. Walks through the directories and finds every build.gradle and appends dependencyLocking {lockAllConfigurations()}

2. Walks through the directories and finds every verification-metadata.xml file and deletes it
   *** Projects maintained for build-reproducibility and already using lockfiles shouldn’t require this

3. Then we take each project name from our projects.txt file and append :dependencies --write-locks to each of them before passing them to Gradle

import os
import subprocess

def add_dependency_locking(build_file):
    """Add dependencyLocking block to build.gradle if not present."""
    with open(build_file, 'r+') as file:
        content = file.read()
        if 'dependencyLocking {' not in content:
            file.write("\ndependencyLocking {\n    lockAllConfigurations() \n}// will lock all project configurations\n")

def run_gradle_command(command):
    """Run the gradle command with the specified argument."""
    full_command = f"gradle {command}"
    try:
        result = subprocess.run(full_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
        print(f"Command executed successfully: {command}")
        print(result.stdout.decode())
    except subprocess.CalledProcessError as e:
        print(f"Error executing command: {command}")
        print(e.stderr.decode())

def main():
    # Delete verification-metadata.xml files
    for root, dirs, files in os.walk('.'):
        for file in files:
            if file == 'verification-metadata.xml':
                os.remove(os.path.join(root, file))

    # Process each build.gradle file
    for root, dirs, files in os.walk('.'):
        for file in files:
            if file == 'build.gradle':
                build_file_path = os.path.join(root, file)
                print(f"Processing {build_file_path}")

                # Add dependency locking block if needed
                add_dependency_locking(build_file_path)

    print("All build.gradle files processed successfully.")

    # Run commands from projects.txt
    with open('projects.txt', 'r') as file:
        for line in file:
            line = line.strip()  # Remove any leading/trailing whitespace
            if line:  # Check if the line is not empty
                run_gradle_command(line + ":dependencies --write-locks")

    print("All commands from projects.txt executed.")

if __name__ == "__main__":
    main()

What you’re left with is not 450 gradle.lockfiles, not 428, but 359 grade.lockfiles. Why? I’m not all too sure. But I’m okay with these results 🤣.

Now I can run semgrep ci --supply-chain to detect each of these lockfiles, parse each dependency and version, and identify known vulnerabilities. With Semgrep, I was also able to generate a SBOM (Beta) in CycloneDX format.

GitHub makes it incredibly convenient to see known dependencies and export an SBOM for standard projects, but that definitely doesn’t work here. Semgrep reported >25,000 dependencies, whereas GitHub’s Dependency Graph for the project shows 6 - additionally, using the export SBOM button results in {"error":"Not Found"}.

This isn’t to say Semgrep handled this Gradle multi-project repository particularly well. It didn’t handle it at all - hence why I’m writing this blog on how to script the nastiness 😆.