Differences in Open Source vulnerability counts across environments

Depending on the environment where the scan was done, snyk test may report different numbers of vulnerabilities.

Some dependencies found through the CLI may differ from those found through Source Git integrations that were set up using the Snyk Web UI.

There are also other possible causes of variation in vulnerabilities reported. These possible causes are summarized on this page.

Scanning a local repository using the CLI

When you scan a local repository using the CLI, snyk test builds a dependency graph based on the source files in your build environment. When you run a test with the CLI using snyk test or make a snapshot for ongoing testing using snyk monitor, Snyk looks at the dependencies as detected by the build tool (for example, Maven, npm, and so on) in order to see exactly what your Project relies on. Private dependencies and the specifics of your build environment are all available to the build tool and so to Snyk as well.

Scanning a remote repository using the Web UI

When you scan a remote repository through an SCM integration using the Snyk Web UI, Snyk infers the dependencies based on the files it can see. Therefore, a scan with the CLI may find more vulnerabilities than a scan through the Snyk Web UI.

Specifically, when Snyk runs a test using an SCM integration such as the GitHub, GitLab, or Bitbucket server integration, Snyk processes only the dependency manifest files for the Project, for example, the pom.xml, package-lock.json, Gemfile.lock or other supported manifest file.

The Snyk integration then infers the dependency graph of your Project, approximating the operation of the build tool. A Snyk SCM Integration does not have access to private dependencies or the specifics of your build environment, such as environment variables. Hence the results may be partial, or the results may be slightly different from those from a scan using the Snyk CLI.

If the Project was scanned with a lockfile, the results may be different from the results of scanning the Project using a manifest file, such as package.json. In general, the lockfile, if it exists, gives a much more deterministic view of the dependency graph. If a lockfile exists, it is always preferred by both the CLI and the SCM integrations, and it allows the SCM integrations, in particular, to be much more accurate and to be similar to the CLI results. Therefore Snyk recommends using a lockfile if possible.

Example of differing results from a CLI versus a Web UI scan

Consider this example:

• A sub-dependency requires foo ^0.0.5 and during build foo 0.0.6 was installed and had a vulnerability.

• Then foo 0.0.7 was released, patching the vulnerability after foo 0.0.6 had been installed.

• According to Snyk rules, an integration would assume that foo 0.0.7 was installed, but in scanning the local Project itself, foo 0.0.7 is not taken into account.

Note: If the Snyk Web UI finds vulnerabilities, the CLI does not try deleting the Project from the Project settings and re-importing the Project. If the problem persists, submit a request to Snyk support.

Differences when scanning Gradle and sbt Projects

In some package managers such as Gradle and sbt, the Snyk SCM Integrations with GitHub, GitLab, Bitbucket, and so on rely on parsing the manifest file or files to determine dependencies, whereas these package managers frequently rely on code to build the dependency tree.

Dependency overrides and mapping using code can sometimes cause the CLI to report a more accurate picture of dependencies than when Snyk scans a Project imported through the SCM.

Differences when scanning Golang Projects

For Go Modules Projects imported using a Git integration, dependencies are resolved at the module level rather than the package level, as Snyk does not have full access to the Project source code.

This means you may see more issues reported for Projects tested in the Git repository because Snyk reports all vulnerabilities for each module rather than for the package(s) referenced in the source code.

Differences when dev dependencies are being ignored

By default, Snyk does not scan for dependencies listed in the dev dependencies section of the manifest file. You can enable scanning of dev dependencies in the following ways:

• Using the Snyk CLI, add the command line option --dev.

• Using the Snyk Web UI, click on Settings > Languages and select the language, for example, JavaScript. Then select Scan and fix dev dependencies or the similar option for each language where such an option is available.

Differences between Snyk scans and tests by other tools

Snyk has more vulnerabilities in its database and may include them sooner than other databases do. Thus results may be different from tests done with other tools such as npm-audit.

When comparing results between tools, consider the following:

• Is what is being scanned the same? For example, were the projects built on different machines at different times without a lockfile that ensures similar dependencies?

• Are there "ignore" rules configured in one tool but not in the other?

• Does one tool evaluate developer dependencies while the other does not? Snyk does not evaluate developer dependencies by default to reduce noise not seen in production.

• Does the vendor "scrape databases" or does it curate and review issues and their severities like Snyk does?

• How does the tool count issues? Is one issue counted for every instance or path of a vulnerability? Or does the tool group the paths to the offending package?

All of these differences can lead to different package dependency trees being evaluated, resulting in different security and license issues being identified.

For more information, see Snyk Security Intelligence, Comprehensive security coverage section.

Differences based on ignores set in the Web UI

Consider the case of setting ignores for certain vulnerabilities in a GitHub-originated Project. When the same Project is scanned with the CLI, the vulnerabilities that were ignored in the Web UI were found again, and the build failed.

This occurred because ignores set in the Web UI are not duplicated in the local version of the Project.

When you ignore a vulnerability using the CLI command snyk ignore, a .snyk policy file is created and saved locally in a .snyk file in the folder of your Project. For information on how this file is created, see The .snyk file.

If you ignore a vulnerability through the Snyk Web UI, there is no mechanism for the ignore policy to be propagated from the Web UI to the local .snyk file in your Project folder. This is why when you run snyk test after ignoring a vulnerability using the Snyk Web UI, you see that vulnerability in the CLI test results.

That is, ignores are not showing because you created them in the Project imported from GitHub while the CLI is running through the CLI integration with snyk. For Snyk, the local Project and the Project in your account are two separate Projects, so the ignores work only for the scans of GitHub.

If you want the ignores to be effective in the CLI, create the Project from the CLI and apply the ignores in the CLI as follows:

1. Go to the root of your Project and run snyk monitor.

2. Go to the Snyk Web UI; there is a new Project that has the name of the Project in the package.json file with the CLI icon.

3. Use the snyk ignore command to ignore the issue in the new Project. You can also set ignores in the Web UI for a Project created with the CLI.

4. Run snyk test and snyk monitor and see if this solution ignores the issues you intend to ignore. Wait a minute or two to allow some caches to expire. Then look at the Projects screen to see if what you intended is ignored.

An alternative is to run the snyk ignore command in the CLI and commit the resulting .snyk policy file to your branch in GitHub to apply ignores for all Snyk environments.

Note that you must commit the .snyk policy file each time you make a change in the ignore settings.

Ignores and snyk monitor versus SCM Projects

If a Project is created with snyk monitor in the CLI and ignores are added in the Web UI, when you run snyk test on the local Project, the ignores created in the Web UI are taken into account.

If the Project is created by an SCM integration, there are two different Projects, one created using the CLI and the other by an SCM integration. These two Projects may have the same name but will have a different icon. If you add an Ignore in Web UI for the Git Project and test the Project locally, the ignores added in the Web UI are not taken into account.

When to scan with the CLI versus through the Snyk Web UI

When you need the most accurate possible results, scan using the CLI. When you can rely on an approximation of the vulnerabilities or have a lockfile that is supported, scan using one of the Git integrations in the Snyk Web UI.

Scanning through an SCM integration in the Web UI allows you to scan a lot of repositories quickly. Thus, for example, if a security manager scans 50 Projects and 45 repositories are found to have no vulnerabilities, and five have vulnerabilities, the manager can ask developers to scan those five with the CLI and explore the results in greater depth.

The drawback is that for the other 45 repositories, you may miss vulnerabilities because you are relying on an approximation.