Command-Line Usage

Each tool of nbtoolbelt is invoked as a subcommand of nbtb:

nbtb <tool> ...

nbtb handles the following arguments itself:

nbtb -version to obtain the current version, and exit
nbtb -h or nbtb --help to get a usage message, and exit
nbtb config to show the location of the embedded configuration file and the start-up configuration
nbtb config <tool> to show the start-up configuration for a specific tool

Processing Multiple Notebooks

When more than one notebook is given, or in verbose mode, the output related to each notebook nb.ipynb is preceded by a header of the form

:::::::::::::
nb.ipynb
:::::::::::::

Some data is collected over all notebooks and aggregated (also see the option output-json to Write JSON Output and the Examples below).

Common Tool Options

Many options, also for the tools themselves, have both a positive and a negated form. This is useful, because option values can also be read from configuration files.

The following options apply to multiple tools (and many of them apply to all tools):

-h, --help            show this help message and exit
-v, --verbose, -V, --no-verbose
                      verbose mode produces extra output (default: False)
-q, --quiet, -Q, --no-quiet
                      quiet mode produces less output (default: False)
--assert, --no-assert
                      assert mode: when processing fails, abort with exit
                      code 1 (default: True)
--validate, --no-validate
                      validate notebook before processing (default: False)
--run, --no-run       run notebook before processing (default: False)
--inplace, --no-inplace
                      replace original notebooks with processing result
                      (default: False)
--write-files, --no-write-files
                      do write result files (default: True)
-n                    short for --no-write-files: do processing but do not
                      write result files (dry run)
--config FILE.json    read configuration from FILE.json (in JSON)
--output-json FILE.json
                      write statistical output to FILE.json
-d, --debug           debug mode produces diagnostic output (default: False)

Help

Each command provides its own help option:

nbtb <tool> -h or nbtb command --help

The default option values shown, are after loading of the start-up configuration (see below).

Verbose Mode

Each command supprts verbose mode:

nbtb <tool> -v or nbtb <tool> --verbose

In verbose mode, the tool’s option values are shown, and the total number of processed notebooks is shown at the end, e.g.:

Notebooks processed: 2

Quiet Mode

Each command supprts quiet mode:

nbtb <tool> -q or nbtb <tool> --quiet

In quiet mode, output is reduced to a minimum.

Assert Mode

In assert mode (option -a or --assert), whenever an error occurs during processing, all further processing will be interrupted and the tool aborts.

Validate Mode

In validate mode (option --validate), each notebook is validated before processing.

Validation options can be set under the specific tool in a configuration file, but not on the command line.

Run Mode

In run mode (option --run), each notebook is run before processing (and after validation, if that applies).

Run options can be set under the specific tool in a configuration file, but not on the command line.

Inplace

For some tools (currently: clean and run), the option --inplace can be used to replace the input notebook with the resulting notebook.

Write Files and Dry Run

In dry-run mode (option --no-write-files, or -n for short), all processing and reporting is done, but resulting notebooks are not written to their destination file.

This allows one te experiment with options. In particular, this is useful in combination with verbose mode.

Read Configuration File

Initial option settings can be read from a configuration file (option --config FILE.json).

At start up, each tool gathers an initial configuration as follows.

The configuration file embedded in the package is loaded.
If present, the configuration file /etc/nbtoolbelt.json is loaded on top of this.
Finally, if present, the configuration file ~/.nbtoolbelt.json is loaded on top of that. Here, ~ stands for the user’s home directory.

The invokation

$ nbtb config [tool]

will show which configuration files are loaded, and what the resulting configuration is.

Each tool can read options from a configuration file:

nbtb <tool> ... --config file.json ...

The options from the configuration file override the preceding command-line options, and are, in turn, overridden by any subsequent command-line options.

See Configuration Files for details about configuration files.

Write JSON Output

Most tools gather some statistics as they process notebooks. These statistics are printed as notebooks are processed. For use with other tooling, such tool output can also be written to a file in JSON format (option --output-json FILE.json).

This JSON output file contains one array, where

item 0 concerns data aggregated over all notebooks that were processed;
item i (i from 1) concerns data collected for the i-th notebook argument;
each item is an object with further data appearing in various members, documented per tool;
one common member is present with name "file_name", and the file’s name as value.

Debug Mode

In debug mode, diagnostic output is produced, and in particular, all option settings are shown.

Available Tools

The following tools are available:

nbtb validate ... Notebook Validation
nbtb head ... Notebook Heads
nbtb dump ... Notebook Dump
nbtb stats ... Notebook Statistics
nbtb view ... Notebook Viewing
nbtb cat ... Notebook Catenation
nbtb clean ... Notebook Cleaning
nbtb run ... Notebook Execution
nbtb split ... Notebook Splitting
nbtb punch ... Notebook Punching

Examples

The following invocation shows

the options loaded at start up from the various configuration files,
the location where these files were found, and
what the resulting configuration is:

$ nbtb config run
Loading configuration for nbrun from: /Users/.../nbtoolbelt/data/nbtoolbelt.json
Skipping configuration in: /etc/nbtoolbelt.json (not present)
Loading configuration for nbrun from: /Users/.../.nbtoolbelt.json
Configuration accumulated for nbrun:
  allow_errors (bool): True
  append_cell (bool): False
  appended_cell (str): '# Automatically added code cell: lists all global names defined by this notebook.\n%whos'
  assert (bool): True
  clean_after (bool): True
  clean_after_metadata (list): ['collapsed', 'scrolled']
  clean_before (bool): True
  clean_before_metadata (list): ['ExecuteTime', 'execution']
  debug (bool): False
  inplace (bool): False
  interrupt_on_timeout (bool): True
  ipc (str): ''
  kernel_name (str): ''
  notebooks (list): []
  output_json (NoneType): None
  quiet (bool): False
  record_timing (bool): True
  run (bool): False
  run_path (str): ''
  run_result_name (str): '-run'
  streams_head (int): -1
  streams_truncate_message (str): '*** Output truncated ***'
  timeout (int): -1
  validate (bool): False
  verbose (bool): False
  write_files (bool): True

The following invocation prints statistics for two notebooks, and writes JSON output to file nbtb-stats-output.json.

$ nbtb stats short.ipynb test.ipynb --output-json nbtb-stats-output.json

::::::::::::::
short.ipynb
::::::::::::::
Notebook metadata:
         4.2 format version
     python3 kernel
  python 3.6.1 language
Cell types:
           1 code
           1 markdown
           2 total cell count

::::::::::::::
test.ipynb
::::::::::::::
Notebook metadata:
         4.2 format version
     python3 kernel
  python 3.6.1 language
Cell types:
          12 code
           7 markdown
          19 total cell count

Totals
======

Cell types:
          13 code
           8 markdown
          21 total cell count

The file nbtb-stats-output.json then contains:

[
  {
    "cell_types": {
      "code": {
        "25%": 3.75,
        "50%": 6.5,
        "75%": 9.25,
        "count": 2.0,
        "max": 12.0,
        "mean": 6.5,
        "min": 1.0,
        "std": 7.7781745930520225,
        "total": 13.0
      },
      "markdown": {
        "25%": 2.5,
        "50%": 4.0,
        "75%": 5.5,
        "count": 2.0,
        "max": 7.0,
        "mean": 4.0,
        "min": 1.0,
        "std": 4.242640687119285,
        "total": 8.0
      },
      "total cell count": {
        "25%": 6.25,
        "50%": 10.5,
        "75%": 14.75,
        "count": 2.0,
        "max": 19.0,
        "mean": 10.5,
        "min": 2.0,
        "std": 12.020815280171307,
        "total": 21.0
      }
    },
    "file_count": 2,
    "file_name": {
      "count": 2,
      "freq": 1,
      "top": "short.ipynb",
      "unique": 2
    },
    "notebook_metadata": {
      "format version": {
        "count": 2,
        "freq": 2,
        "top": "4.2",
        "unique": 1
      },
      "kernel": {
        "count": 2,
        "freq": 2,
        "top": "python3",
        "unique": 1
      },
      "language": {
        "count": 2,
        "freq": 2,
        "top": "python 3.6.1",
        "unique": 1
      }
    }
  },
  {
    "cell_types": {
      "code": 1,
      "markdown": 1,
      "total cell count": 2
    },
    "file_name": "short.ipynb",
    "notebook_metadata": {
      "format version": "4.2",
      "kernel": "python3",
      "language": "python 3.6.1"
    }
  },
  {
    "cell_types": {
      "code": 12,
      "markdown": 7,
      "total cell count": 19
    },
    "file_name": "test.ipynb",
    "notebook_metadata": {
      "format version": "4.2",
      "kernel": "python3",
      "language": "python 3.6.1"
    }
  }
]