Tool: snyk-api-import

Snyk helps you find, fix, and monitor for known vulnerabilities in your dependencies, both on an ad hoc basis and as part of your Continuous Integration (CI) (build) system.

snyk-api-import

The Snyk API Project importer, snyk-api-import, is a script intended to help import Projects into Snyk using available Snyk APIs at a controlled pace to avoid rate limiting from GitHub, GitLab, Bitbucket, and other systems and to provide a stable import. The script kicks off import in batches, waits for completion, and then keeps going. Any failed requests are retried before they are considered a failure and logged.

If you need to adjust concurrency, you can stop the script, change the concurrency variable, and start again. The tool skips previous repositories (Targets) that have been requested for import.

To use snyk-api-import you must do the following in advance:

• Set up your Snyk Organizations (Orgs) before running an import.

• Configure your Snyk Organizations with some connection to an SCM (GitHub, GitLab, Bitbucket, other) as you will need the integrationId to generate the import files.

• Use the Set notification settings API endpoint to disable notifications for emails and so on, to avoid receiving import notifications (recommended).

• Use the Update (integration settings) endpoint to disable the fix PRs and PR checks until import is complete to avoid sending extra requests to SCMs (GitHub, GitLab, Bitbucket, and so on).

Installation

snyk-api-import CLI can be installed through multiple channels.

Standalone executables (macOS, Linux, Windows)

Use the GitHub Releases to download a standalone executable of snyk-api-import CLI for your platform.

Install with npm or Yarn

snyk-api-import CLI is available as an npm package. If you have Node.js installed locally, you can install it by running:

npm install snyk-api-import@latest -g

If you are using Yarn, run:

yarn global add snyk-api-import

Usage

By default the import command will run if no command is specified.

• import - kick off an API-powered import of repos (Targets) into existing Snyk Organizations defined in the import configuration file. All support available for all Project types is provided through the Import API endpoints, Import targets and Get Import job details.

• help - show help and all available commands and their options.

• orgs:data utility - use to generate data required to create Organizations using the API.

• orgs:create utility - use to create the Organizations in Snyk based on the data file generated with orgs:data command.

• import:data utility - use to generate data required to kick off an import. Note that archived repositories are excluded by default.

• list:imported utility - use to generate data to help skip previously imported targets during import.

The logs can be explored using the Bunyan CLI

Contents of the snyk-api-import instructions

• Utilities

  • Creating Organizations in Snyk

  • Creating import targets and data for import

  • Mirroring GitHub.com and GitHub Enterprise organizations and repos in Snyk

  • Mirroring GitLab organizations and repos in Snyk

  • Mirroring Bitbucket Server organizations and repos in Snyk

  • Mirroring Bitbucket Cloud organizations and repos in Snyk

• Kicking off an import

• Contributing to snyk-api-import

• Sync: detecting changes in monitored repos and updating Snyk Projects

• Example workflow: AWS automation

FAQ

Error: ENFILE: file table overflow, open or Error: EMFILE, too many open files

If you see these errors, you may need to bump ulimit to allow more open file operations. In order to keep the operations performing well, the tool logs as soon as it is convenient rather than waiting until the very end of a loop and logging a huge data structure. This means that depending on the number of concurrent imports set, the tool may exceed the system default ulimit.

Some of these resources may help you bump the ulimit:

• ss64.com

• StackOverflow

• blog.mact.me

ERROR: HttpError: request to https://github.private.com failed, reason: self signed certificate in certificate chain

If your GitHub, GitLab, Bitbucket, or Azure instance is using a self-signed certificate, you can configure snyk-api-import to use this certificate when calling the HTTPS APIs.

export NODE_EXTRA_CA_CERTS=./path-to-ca

Does this work with brokered integrations?

Yes. Because Snyk reuses the existing integration with your SCM (Git) repository to perform the imports, the brokered connection will be used when configured.

What is supported for the import command?

snyk-api-import supports all of the same integration types and Project sources as identified in the Import API documentation. If an example for your use case is not in these instructions, see the API documentation.