Notebook Punching
=================

``punch`` punches holes in notebooks,
supporting two different approaches to defining the holes (see below).
The resulting holes can be **filled**
either with fixed content, or from a source notebook
(see options ``fill`` and ``source``).

``punch`` can also collect the content that has been punched out,
the so-called **chads** (see option ``chads``).


Result File Names
-----------------

The result of punching notebook ``nb.ipynb`` is written to ``nb-punched.ipynb``,
and the content that got punched out, the so-called *chads*, is written to ``nb-chads.ipynb``.

The result name additions can be adjusted in the configuration file
by setting ``punch_punched_result_name`` and ``punch_chads_result_name``;
see :doc:`configuration`.


Holes and Filling
-----------------

The two ways of defining holes are:

* Via **tags**:
  any cell with a tag contained in the trigger set (see option ``tags``),
  will be punched as a whole.
* Via **marker lines**:
  any source content appearing between
  *begin* and *end* marker lines will be punched.

Via Tags
^^^^^^^^
This approach is selected by providing a list of tags to the ``tags`` option.

In the tags approach, ``punch`` removes the source content of selected cells.
No cells will be removed;
only their source content will be emptied.

Cells are selected through the presence of cell metadata tags in a trigger set,
defined with the option ``tags``.

When the source of a *code* cell is cleared,
also its outputs and execution count are cleared.

Note that ``nbgrader`` uses the tag ``YourTurn``
to mark cells where students write their solutions;
see `documentation <https://nbgrader.readthedocs.org/>`__ of ``nbgrader``.

Via Marker Lines
^^^^^^^^^^^^^^^^
This approach applies when no tags are given (the default).

Lines to be punched are marked by enclosing them between
**begin** and **end** marker lines.

Such marker lines are recognized by a special character sequence
(default: ``#//`` for all cell types;
this can be overridden per cell type in the configuration file).
Marker lines are parsed through a regular expression (per cell type) into

* a **transition**, to distinguish *begin* and *end* marker lines
  (from outside to inside, and from inside to outside);
* a **label**, to distinguish holes, especially needed when filling them from a source notebook;
* an optional **description**, mainly as documentation, or as hint to students.

Replacement Content
^^^^^^^^^^^^^^^^^^^
When filling,
the default replacement content is defined in the embedded configuration file as

.. code-block:: json

  {
    "filling": {
      "markdown": "\n<div class='alert alert-warning' role='alert'>Replace this line by your text.</div>\n",
      "code": "\n# ===== =====> Replace this line by your code. <===== ===== #\n",
      "raw": "\n% ===== =====> Replace this line by your content. <===== =====\n"
    }
  }

These replacement strings can be redefined in your own configuration file;
see :doc:`configuration`.

In the replacement content,
two substitutions are done:

* ``{label}`` is replaced by the hole's label,
* ``{description}`` is replaced by the hole's description.

This is especially useful with the ``no-keep-marker-lines`` option.

With the ``source`` option,
the replacement content is taken from a *source* notebook.
This can only be used with *marker lines*.
The replacement chad in the source notebook is selected by matching the *labels*.
In this case,
there can be a mismatch between cell types in the hole to be filled and in the selected source chad
to be used as filling.

``punch`` ensures that content from source cells ends up
in cells of the same type in the punched notebook.
In particular,
when the ``source`` option is *not* used,
the punched notebook will have the same cells as the input notebook,
but possibly with different content inside the holes.

The following diagram illustrates how ``punch`` uses cells and cell source content
when filling from a source notebook.

.. image:: images/punch-filling-details.png
   :scale: 40 %
   :align: center
   :alt: relationship between input, source, and punched notebook when using ``punch``

Some notes:

* The number of cells in the source chad and input hole can differ.
  Both have at least one cell.
* The begin and end marker line can be in the same cell
  (in which case ``cells[0]`` and ``cells[-1]`` are identical).
  In fact, a cell can contain multiple holes.
* The cell types of the begin cells can differ,
  and so can the types of the end cells.
  For instance, the hole could start with a code cell,
  whereas the replacement source chad starts with a markdown cell.
  In such cases,
  ``punch`` will split cells as necessary.
* All cell metadata, attachments, and outputs are carried along.


Educational Use
---------------

``punch`` is intended for use in an educational setting.
The teacher prepares a *master notebook* **A** with questions and
tagged or marked solutions
(see diagram).

.. image:: images/punch-fill.png
   :scale: 40 %
   :align: center
   :alt: relationship between notebooks when using ``punch``

Using ``punch``,
the teacher then produces a *punched notebook* **B** (a framework, in software terms),
that can be given to students,
who need to fill the holes.
The teach can also produce a *chads notebook* **C**,
with just the solutions,
that can be given to assistants.
Students deliver their completed notebooks **D**.
When punching a student notebook **D**,
the *chads notebook* **E** will just contain the student's work.

With reference to the diagram, we have:

* **A** = ``master.ipynb`` with questions and solutions
* ``nbtb punch mater.ipynb --chads`` produces
* **B** = ``master-punched.ipynb``
* **C** = ``master-chads.ipynb``
* student produces
* **D** = ``student.ipynb`` delivered by student (from ``master-punched.ipynb`` with work in holes)
* ``nbtb punch student.ipynb --chads`` produces
* **E** = ``student-chads.ipynb`` with just the student's work

``punch`` can optionally fill the punched holes either with fixed content
(taken from the configuration file),
or (in the case of the approach via marker lines)
with content taken from another *source notebook* with correspondingly marked content.

The teacher can also prepare a *test notebook* **G**,
with correspondingly marked holes,
and punch it while using the student's notebook **D** as source to fill the holes.
In that way, the student's work is copied into the test notebook
in the appropriate places.
The resulting notebook **H** can be executed to provide feedback.

``punch`` can also be used to transfer the solutions from the master notebook **A**
into the test notebook **G**, obtaining a verification notebook **F**.

With reference to the diagram, we have:

* **F** = `test_frame.ipynb`` a test notebook to assess work
* ``nbtb punch test_frame.ipynb --fill --source master.ipynb`` produces
* **G** = ``test_frame-punched.ipynb`` tes notebook with teacher solutions for verification
* ``nbtb punch test_frame.ipynb --fill --source student.ipynb`` produces
* **H** = ``test_frame-punched.ipynb`` test notebook with student work

When filling the template notebook **B** with the solutions (chads) **C**,
the original master notebook **A** is obtained:

* ``nbtb punch master-punched.ipynb --fill --source master-chads.ipynb`` produces
* ``master-punched-punched.ipynb`` which is equivalent to ``master.ipynb``


Options
-------

The following options are supported by ``punch``:

.. code-block:: none

  -t TAGS, --tags TAGS  comma-separated list of tags that trigger removal of
                        source from cells; e.g. YourTurn (default: '')
  -p, --punched, -P, --no-punched
                        whether to write punched notebook (default: True)
  -c, --chads, -C, --no-chads
                        whether to write chads notebook (default: False)
  -m, --keep-marker-lines, -M, --no-keep-marker-lines
                        whether to keep marker lines (default: True)
  -f, --fill, -F, --no-fill
                        whether to fill punched holes (default: False)
  -s PUNCH_SOURCE, --source PUNCH_SOURCE
                        source notebook for filling holes; implies --fill
                        (default: "")
  -l, --list, -L, --no-list
                        list marker labels and descriptions (default: False)
  -e, --allow-errors, -E, --no-allow-errors
                        continue on parsing errors in marker lines (default:
                        False)
  --label-regex REGEX
                        only process marker lines whose labels match REGEX
                        (default: ''; empty regex matches all)

Tags
^^^^
The ``tags`` option defines the trigger set, using a comma-separated list.
It can also be defined as a list in the configuration file in your home directory,
with the key ``tags``.

Punched
^^^^^^^
By default, ``punch`` writes the punched notebook.
The option ``punched`` controls this.

Chads
^^^^^
By default, ``punch`` does not write out the chads notebook.
The option ``chads`` controls this.

Keep Marker Lines
^^^^^^^^^^^^^^^^^
By default, the marker lines are copied to both the punched notebook
and the chads notebook.
The option ``keep-marker-lines`` controls this.

Fill
^^^^
By default, ``punch`` will not fill the created holes.
The option ``fill`` controls whether ``punch`` fills holes.
Also see option ``source`` below.

Source
^^^^^^
When filling holes,
``punch`` will take fixed replacement content from the configuration file
if option ``source`` is not supplied;
otherwise, it will take replacement content from the source notebook,
by matching hole labels.
The hole labels must be unique in the source notebook.

The ``source`` option cannot be used together with the ``tags`` option.

The ``source`` options implies the ``fill`` option.

List
^^^^
The option ``list`` will list labels and descriptions of all holes.

Allow Errors
^^^^^^^^^^^^
By default, ``punch`` will abort execution at the first error
that is detected in the marker structure.
The option ``allow-errors`` can be used to let ``punch``
continue processing by assuming a corrected marker structure.
In particular,

* when a bad marker is encountered outside a hole, it is ignored, and
* when a bad marker is encountered inside a hole, a correct end marker is assumed,
  or inserted in case a begin marker was detected.

The number of errors is reported.
Do note, however, that when errors occurred,
the notebooks involved still need to be fixed.
This option is intended only as an aid to finding as many problems
in a single run of the tool.

Label Regex
^^^^^^^^^^^
The option ``label-regex`` can be used to restrict processing
to marker lines whose label matches a given regular expression.
This filtering affects production of punched and chads files, and to reading of source chads.
Regular expressions use `Python re syntax <https://docs.python.org/3/library/re.html#regular-expression-syntax>`__.
An empty string is interpreted as no filtering (processing all labels).


JSON Output
-----------

See :ref:`output-json` for general information about JSON output.

``punch`` produces the following members in the JSON output.
Note that members are absent when count is zero.

With option ``tags``:

  +------------------+----------------------------------------------------+
  | Name             | Value                                              |
  +==================+====================================================+
  | ``"CT"``         | count of cells with type ``CT``                    |
  +------------------+----------------------------------------------------+
  | ``"CT holes"``   | count of cells with type ``CT`` punched            |
  +------------------+----------------------------------------------------+

Cell types distinguished: ``markdown``, ``code``, ``raw``

With options ``punched`` and ``chads``:

  +------------------------------+-------------------------------------------------------+
  | Name                         | Value                                                 |
  +==============================+=======================================================+
  | ``"errors"``                 | count of errors in marker lines                       |
  +------------------------------+-------------------------------------------------------+
  | ``"holes"``                  | count of holes detected                               |
  +------------------------------+-------------------------------------------------------+
  | ``"labels with duplicates"`` | count of labels with duplicates                       |
  +------------------------------+-------------------------------------------------------+
  | ``"unfilled holes"``         | count of unfilled holes (with option ``fill``)        |
  +------------------------------+-------------------------------------------------------+
  | ``"unused source chads"``    | count of unused source chads (with option ``source``) |
  +------------------------------+-------------------------------------------------------+


Examples
--------

The following example punches all cells having a tag ``YourTurn``, in verbose mode.

.. code-block:: bash

  $ nbtb punch -t YourTurn test.ipynb -v
    Options for nbpunch:
      Tags that trigger punch: ['YourTurn']
      Writing punched notebook: True
      Writing chads notebook: False
      Keeping marker lines: True
      Do not fill punched holes
    ::::::::::::::
    test.ipynb
    ::::::::::::::
    Punch cell statistics:
              12 code
               2 code holes
               7 markdown
               1 markdown holes
    No file written

    Notebooks processed: 1

The following example uses a source notebook.
However, not all holes in the input notebook
have replacements in this source notebook.
Also the chads notebook is written.

.. code-block:: bash

  $ nbtb punch test-template.ipynb --chads --source test-source.ipynb
    Punch cell statistics:
               8 holes
               2 unfilled holes

To find out which holes were not filled, use verbose mode:

.. code-block:: bash

    $ nbtb punch test-template.ipynb --chads --source test-source.ipynb -v
    Options for nbpunch:
      Writing punched notebook: True
      Writing chads notebook: True
      Keeping marker lines: True
      Fill holes from source notebook: test-source.ipynb
    ::::::::::::::
    test-template.ipynb
    ::::::::::::::
    Labels of unfilled holes: ['Label_7', 'Label_8']
    Punch cell statistics:
               8 holes
               2 unfilled holes
      Files written: {'test-template-punched.ipynb', 'test-template-chads.ipynb'}

    Notebooks processed: 1

To verify the marker structure, use the options ``dry-run``, ``list``, ``allow-errors``.

.. code-block:: bash

    $ nbtb punch -nle test-template.ipynb test-code-marker-duplicate-label.ipynb test-code-marker-bad-transition-missing-end.ipynb
    Dry run (no files written)
    ::::::::::::::
    test-template.ipynb
    ::::::::::::::
    Listing labels and descriptions for "test-template.ipynb"
      [Label_1] Do first thing
      [Label_2] Do second, longer, thing
      [Label_3] Do third, even longer, thing
      [Label_4] Write a short text
      [Label_5] Write a medium text
      [Label_6] Write longer text
      [Label_7] Do seventh, mixed, thing.
      [Label_8] Do another mixed problem.
    Punch cell statistics:
               8 holes
    No file written
    ::::::::::::::
    test-code-marker-duplicate-label.ipynb
    ::::::::::::::
    Listing labels and descriptions for "test-code-marker-duplicate-label.ipynb"
      [Label_1] Begin of first hole
      [Label_1] Begin of second hole, with duplicate label
    Punch cell statistics:
               2 holes
               1 labels with duplicates
    No file written
    ::::::::::::::
    test-code-marker-bad-transition-missing-end.ipynb
    ::::::::::::::
    Listing labels and descriptions for "test-code-marker-bad-transition-missing-end.ipynb"
      [Label_1] Begin of first hole
    Unexpected transition in marker line of cell 2 on line 1:
      #// BEGIN_TODO [Label_2] Begin of second hole
      Expected END [Label_1], but found BEGIN.
      Inserting correct end marker
      [Label_2] Begin of second hole
    Punch cell statistics:
               1 errors
               2 holes
    No file written