Notebook Punching
punch
punches holes in notebooks,
supporting two different approaches to defining the holes (see below).
The resulting holes can be filled
either with fixed content, or from a source notebook
(see options fill
and source
).
punch
can also collect the content that has been punched out,
the so-called chads (see option chads
).
Result File Names
The result of punching notebook nb.ipynb
is written to nb-punched.ipynb
,
and the content that got punched out, the so-called chads, is written to nb-chads.ipynb
.
The result name additions can be adjusted in the configuration file
by setting punch_punched_result_name
and punch_chads_result_name
;
see Configuration Files.
Holes and Filling
The two ways of defining holes are:
Via tags: any cell with a tag contained in the trigger set (see option
tags
), will be punched as a whole.Via marker lines: any source content appearing between begin and end marker lines will be punched.
Via Marker Lines
This approach applies when no tags are given (the default).
Lines to be punched are marked by enclosing them between begin and end marker lines.
Such marker lines are recognized by a special character sequence
(default: #//
for all cell types;
this can be overridden per cell type in the configuration file).
Marker lines are parsed through a regular expression (per cell type) into
a transition, to distinguish begin and end marker lines (from outside to inside, and from inside to outside);
a label, to distinguish holes, especially needed when filling them from a source notebook;
an optional description, mainly as documentation, or as hint to students.
Replacement Content
When filling, the default replacement content is defined in the embedded configuration file as
{
"filling": {
"markdown": "\n<div class='alert alert-warning' role='alert'>Replace this line by your text.</div>\n",
"code": "\n# ===== =====> Replace this line by your code. <===== ===== #\n",
"raw": "\n% ===== =====> Replace this line by your content. <===== =====\n"
}
}
These replacement strings can be redefined in your own configuration file; see Configuration Files.
In the replacement content, two substitutions are done:
{label}
is replaced by the hole’s label,{description}
is replaced by the hole’s description.
This is especially useful with the no-keep-marker-lines
option.
With the source
option,
the replacement content is taken from a source notebook.
This can only be used with marker lines.
The replacement chad in the source notebook is selected by matching the labels.
In this case,
there can be a mismatch between cell types in the hole to be filled and in the selected source chad
to be used as filling.
punch
ensures that content from source cells ends up
in cells of the same type in the punched notebook.
In particular,
when the source
option is not used,
the punched notebook will have the same cells as the input notebook,
but possibly with different content inside the holes.
The following diagram illustrates how punch
uses cells and cell source content
when filling from a source notebook.
Some notes:
The number of cells in the source chad and input hole can differ. Both have at least one cell.
The begin and end marker line can be in the same cell (in which case
cells[0]
andcells[-1]
are identical). In fact, a cell can contain multiple holes.The cell types of the begin cells can differ, and so can the types of the end cells. For instance, the hole could start with a code cell, whereas the replacement source chad starts with a markdown cell. In such cases,
punch
will split cells as necessary.All cell metadata, attachments, and outputs are carried along.
Educational Use
punch
is intended for use in an educational setting.
The teacher prepares a master notebook A with questions and
tagged or marked solutions
(see diagram).
Using punch
,
the teacher then produces a punched notebook B (a framework, in software terms),
that can be given to students,
who need to fill the holes.
The teach can also produce a chads notebook C,
with just the solutions,
that can be given to assistants.
Students deliver their completed notebooks D.
When punching a student notebook D,
the chads notebook E will just contain the student’s work.
With reference to the diagram, we have:
A =
master.ipynb
with questions and solutionsnbtb punch mater.ipynb --chads
producesB =
master-punched.ipynb
C =
master-chads.ipynb
student produces
D =
student.ipynb
delivered by student (frommaster-punched.ipynb
with work in holes)nbtb punch student.ipynb --chads
producesE =
student-chads.ipynb
with just the student’s work
punch
can optionally fill the punched holes either with fixed content
(taken from the configuration file),
or (in the case of the approach via marker lines)
with content taken from another source notebook with correspondingly marked content.
The teacher can also prepare a test notebook G, with correspondingly marked holes, and punch it while using the student’s notebook D as source to fill the holes. In that way, the student’s work is copied into the test notebook in the appropriate places. The resulting notebook H can be executed to provide feedback.
punch
can also be used to transfer the solutions from the master notebook A
into the test notebook G, obtaining a verification notebook F.
With reference to the diagram, we have:
F = test_frame.ipynb` a test notebook to assess work
nbtb punch test_frame.ipynb --fill --source master.ipynb
producesG =
test_frame-punched.ipynb
tes notebook with teacher solutions for verificationnbtb punch test_frame.ipynb --fill --source student.ipynb
producesH =
test_frame-punched.ipynb
test notebook with student work
When filling the template notebook B with the solutions (chads) C, the original master notebook A is obtained:
nbtb punch master-punched.ipynb --fill --source master-chads.ipynb
producesmaster-punched-punched.ipynb
which is equivalent tomaster.ipynb
Options
The following options are supported by punch
:
-t TAGS, --tags TAGS comma-separated list of tags that trigger removal of
source from cells; e.g. YourTurn (default: '')
-p, --punched, -P, --no-punched
whether to write punched notebook (default: True)
-c, --chads, -C, --no-chads
whether to write chads notebook (default: False)
-m, --keep-marker-lines, -M, --no-keep-marker-lines
whether to keep marker lines (default: True)
-f, --fill, -F, --no-fill
whether to fill punched holes (default: False)
-s PUNCH_SOURCE, --source PUNCH_SOURCE
source notebook for filling holes; implies --fill
(default: "")
-l, --list, -L, --no-list
list marker labels and descriptions (default: False)
-e, --allow-errors, -E, --no-allow-errors
continue on parsing errors in marker lines (default:
False)
--label-regex REGEX
only process marker lines whose labels match REGEX
(default: ''; empty regex matches all)
Punched
By default, punch
writes the punched notebook.
The option punched
controls this.
Chads
By default, punch
does not write out the chads notebook.
The option chads
controls this.
Keep Marker Lines
By default, the marker lines are copied to both the punched notebook
and the chads notebook.
The option keep-marker-lines
controls this.
Fill
By default, punch
will not fill the created holes.
The option fill
controls whether punch
fills holes.
Also see option source
below.
Source
When filling holes,
punch
will take fixed replacement content from the configuration file
if option source
is not supplied;
otherwise, it will take replacement content from the source notebook,
by matching hole labels.
The hole labels must be unique in the source notebook.
The source
option cannot be used together with the tags
option.
The source
options implies the fill
option.
List
The option list
will list labels and descriptions of all holes.
Allow Errors
By default, punch
will abort execution at the first error
that is detected in the marker structure.
The option allow-errors
can be used to let punch
continue processing by assuming a corrected marker structure.
In particular,
when a bad marker is encountered outside a hole, it is ignored, and
when a bad marker is encountered inside a hole, a correct end marker is assumed, or inserted in case a begin marker was detected.
The number of errors is reported. Do note, however, that when errors occurred, the notebooks involved still need to be fixed. This option is intended only as an aid to finding as many problems in a single run of the tool.
Label Regex
The option label-regex
can be used to restrict processing
to marker lines whose label matches a given regular expression.
This filtering affects production of punched and chads files, and to reading of source chads.
Regular expressions use Python re syntax.
An empty string is interpreted as no filtering (processing all labels).
JSON Output
See Write JSON Output for general information about JSON output.
punch
produces the following members in the JSON output.
Note that members are absent when count is zero.
With option tags
:
Name
Value
"CT"
count of cells with type
CT
"CT holes"
count of cells with type
CT
punched
Cell types distinguished: markdown
, code
, raw
With options punched
and chads
:
Name
Value
"errors"
count of errors in marker lines
"holes"
count of holes detected
"labels with duplicates"
count of labels with duplicates
"unfilled holes"
count of unfilled holes (with option
fill
)
"unused source chads"
count of unused source chads (with option
source
)
Examples
The following example punches all cells having a tag YourTurn
, in verbose mode.
$ nbtb punch -t YourTurn test.ipynb -v
Options for nbpunch:
Tags that trigger punch: ['YourTurn']
Writing punched notebook: True
Writing chads notebook: False
Keeping marker lines: True
Do not fill punched holes
::::::::::::::
test.ipynb
::::::::::::::
Punch cell statistics:
12 code
2 code holes
7 markdown
1 markdown holes
No file written
Notebooks processed: 1
The following example uses a source notebook. However, not all holes in the input notebook have replacements in this source notebook. Also the chads notebook is written.
$ nbtb punch test-template.ipynb --chads --source test-source.ipynb
Punch cell statistics:
8 holes
2 unfilled holes
To find out which holes were not filled, use verbose mode:
$ nbtb punch test-template.ipynb --chads --source test-source.ipynb -v
Options for nbpunch:
Writing punched notebook: True
Writing chads notebook: True
Keeping marker lines: True
Fill holes from source notebook: test-source.ipynb
::::::::::::::
test-template.ipynb
::::::::::::::
Labels of unfilled holes: ['Label_7', 'Label_8']
Punch cell statistics:
8 holes
2 unfilled holes
Files written: {'test-template-punched.ipynb', 'test-template-chads.ipynb'}
Notebooks processed: 1
To verify the marker structure, use the options dry-run
, list
, allow-errors
.
$ nbtb punch -nle test-template.ipynb test-code-marker-duplicate-label.ipynb test-code-marker-bad-transition-missing-end.ipynb
Dry run (no files written)
::::::::::::::
test-template.ipynb
::::::::::::::
Listing labels and descriptions for "test-template.ipynb"
[Label_1] Do first thing
[Label_2] Do second, longer, thing
[Label_3] Do third, even longer, thing
[Label_4] Write a short text
[Label_5] Write a medium text
[Label_6] Write longer text
[Label_7] Do seventh, mixed, thing.
[Label_8] Do another mixed problem.
Punch cell statistics:
8 holes
No file written
::::::::::::::
test-code-marker-duplicate-label.ipynb
::::::::::::::
Listing labels and descriptions for "test-code-marker-duplicate-label.ipynb"
[Label_1] Begin of first hole
[Label_1] Begin of second hole, with duplicate label
Punch cell statistics:
2 holes
1 labels with duplicates
No file written
::::::::::::::
test-code-marker-bad-transition-missing-end.ipynb
::::::::::::::
Listing labels and descriptions for "test-code-marker-bad-transition-missing-end.ipynb"
[Label_1] Begin of first hole
Unexpected transition in marker line of cell 2 on line 1:
#// BEGIN_TODO [Label_2] Begin of second hole
Expected END [Label_1], but found BEGIN.
Inserting correct end marker
[Label_2] Begin of second hole
Punch cell statistics:
1 errors
2 holes
No file written