When StackAid launched, we had first-class support for discovering dependencies in the JavaScript ecosystem. Today we're happy to announce the addition of Go, Java, PHP, Python, Ruby, and Rust!
While it was (and still is) technically possible to fund any dependency via our GitHub Action, it required additional setup for each of your repositories. This involved configuring a new GitHub workflow, with which some of our users had no prior experience. Others had hunderds of repositories and updating each one was a tedious and time consumig process. One of our core principles is to reduce friction as much as possible, so the need for additional work to fund dependencies never sat well with us.
Below are details about how this works under the covers, but the TLDR is that StackAid will now automatically discover dependencies for Go, Java, PHP, Python, Ruby and Rust-based projects. If you already have a StackAid account, we updated your existing projects. Of course, we're excited about being able to fund even more projects via StackAid.
Under the covers
To support JavaScript dependencies, we first had to index the roughly 2.2 million packages published on npm, mapping names and semvers to their source repositories. We continually consume the npm changefeed to stay updated on every new release. So, when we discover a package.json file in a subscriber's repository, we know the state of the world and can instantly show you the underlying direct and indirect dependency repositories you are using.
To support the new ecosystems announced today, we are instead leveraging the GitHub Dependency Graph API to get a list of manifest files for each user's repositories. For example for PHP, we look for composer.json files and cargo.toml files for Rust. For each file, the GitHub API will give us the repositories that map to the declared dependencies. We then crawl each of those repositories to get the indirect dependencies.
Go, a special case
While testing the GitHub API for Go-based repos, we noticed that for go.mod files it returns just a flat list of repositories. Here's a test go.mod file we created with just one dependency, but GitHub returns:
What is missing is the relationship between direct and indirect dependencies. Thankfully Go has some builtin utilities to understanding your dependency tree:
will list your project's direct dependencies and
will list the parent-child relationships for the entire dependency tree. Using the output of the go list command allows us to filter the output provided by go mod graph for the direct and indirect dependencies.
However, we now have a new problem. We need to execute these commands against your repository's source code. To do this, we use a custom image on Fly.io to check out the source, run the Go commands, and return the results via an API call. To check out the source, we create an encrypted JWT with the repository name and GitHub auth token, which allows us to clone the repo. This way, the fly instance has the minimum credentials necessary to check out the source and has no access to StackAid's database, API, etc.
Good, but not perfect
While the GitHub Dependency Graph API does a lot of heavy lifting for us, it's not without it's limitations such as with Go. Two other limitations are limited or no support for many ecosystems and being confined to analyzing projects on GitHub. Over the coming months, we plan to support additional ecosystems as well as fill in the gaps to provide the most accurate list of dependencies for your projects.