Commit a08c1040 for tesseract

commit a08c1040e2d6d45f85fa766db8dfeafd504fb549
Author: Stefan Weil <sw@weilnetz.de>
Date:   Sat Nov 14 09:59:33 2026 +0100

    Add benchmarks for the LSTM code

    Add the tools which were used to measure and to tune the speed of LSTM
    training on ARM64, and the results:

    - bench.sh times lstmtraining for a fixed number of training lines
      with a given build and optional settings (environment, CPU pinning).
    - sample-summary.py summarizes a profile made with the macOS tool
      sample.
    - neon-kernel-variants.cpp compares variants of the NEON functions for
      several float dot products at once.
    - neon-int-kernel-variants.cpp compares variants for the product of an
      int8_t matrix with a vector, which is used for the recognition with
      integer models. They are 3 to 5 times faster than the current code,
      but not implemented in the library yet.
    - README.md describes the usage and the results for an Apple M1 Pro
      and an NVIDIA DGX Spark, with and without OpenMP.

    Assisted-by: OpenCode / Claude Fable 5.2 (Anthropic)
    Signed-off-by: Stefan Weil <sw@weilnetz.de>

diff --git a/benchmark/lstm/README.md b/benchmark/lstm/README.md
new file mode 100644
index 00000000..8dc00964
--- /dev/null
+++ b/benchmark/lstm/README.md
@@ -0,0 +1,144 @@
+# Benchmarks for the LSTM code
+
+Tools to measure the speed of LSTM training (and of the matrix products
+which dominate training and recognition), and the results which were
+measured with them.
+
+## Files
+
+- `bench.sh` times `lstmtraining` for a fixed number of training lines.
+- `sample-summary.py` summarizes a profile which was made with the macOS tool
+  `sample`.
+- `neon-kernel-variants.cpp` is a microbenchmark for the float matrix products
+  (variants of `RowDotProductsNEON` and `CrossDotProductsNEON`).
+- `neon-int-kernel-variants.cpp` is a microbenchmark for the `int8_t` matrix
+  product which is used for the recognition with integer (fast) models.
+
+The two microbenchmarks are standalone programs for ARM64. How to build
+them is described at the beginning of each file.
+
+## Timing the training
+
+`bench.sh` trains the default network of
+[tesstrain](https://github.com/tesseract-ocr/tesstrain)
+(`[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx192 O1c111]`) from scratch
+on the 1036 lines of `test/testdata/eng.Arial.exp0.lstmf`. It needs a build
+directory with the training tools, the `test` submodule and the sibling
+directory `../langdata_lstm`.
+
+    benchmark/lstm/bench.sh build
+    NAME=generic benchmark/lstm/bench.sh build env DOTPRODUCT=generic
+    NAME=omp1 benchmark/lstm/bench.sh build-openmp env OMP_THREAD_LIMIT=1
+    NAME=x925 benchmark/lstm/bench.sh build taskset -c 5-9,15-19
+
+Each run prints one line with the times in seconds and the last progress
+message of the training, which shows whether two runs calculated the same
+results:
+
+    build:   real 15.83 user 15.41 sys 0.27 | 999/1000/1000, mean rms=5.173%, ...
+
+The number of iterations, the network and the working directory can be set
+with environment variables (see the comment in the script).
+
+Run only one benchmark at a time (several runs disturb each other, and
+tests which run at the same time do so, too) and repeat a measurement if
+the machine was busy.
+
+## Profiling
+
+On macOS, `sample` works without special rights:
+
+    ITER=2500 benchmark/lstm/bench.sh build &
+    sleep 4
+    sample $(pgrep -n lstmtraining) 20 1 -file sample.txt
+    benchmark/lstm/sample-summary.py sample.txt
+
+On Linux, `perf record` / `perf report` can be used if
+`/proc/sys/kernel/perf_event_paranoid` allows it (a value of 2 or less).
+
+## Results
+
+Measured in November 2026 with 1000 training lines, FAST_FLOAT, a Release
+build with CMake, and a single thread unless noted otherwise.
+
+- M1: Apple M1 Pro, macOS, Apple clang 21, performance cores.
+- X925: NVIDIA DGX Spark (10 Cortex-X925 and 10 Cortex-A725 cores),
+  Ubuntu 24.04, g++ 13, pinned to the Cortex-X925 cores.
+
+"After" is the commit "lstm: calculate several dot products at once with
+NEON", "before" is its parent.
+
+### Training time
+
+| Build and settings                          | M1 before | M1 after | X925 before | X925 after |
+| ------------------------------------------- | --------: | -------: | ----------: | ---------: |
+| Without OpenMP                              |    22.2 s |   15.9 s |      14.4 s |     10.2 s |
+| Without OpenMP, Cortex-A725 cores           |           |          |      31.3 s |     24.3 s |
+| Without OpenMP, `DOTPRODUCT=generic`        |           |   85.2 s |      51.5 s |     50.6 s |
+| OpenMP, `OMP_THREAD_LIMIT=1`                |    23.0 s |          |      15.5 s |            |
+| OpenMP, default (4 threads)                 | 56 - 98 s |          |      15.5 s |     12.8 s |
+| OpenMP, `OMP_PROC_BIND=true OMP_PLACES=cores` |         |          |      11.6 s |      9.6 s |
+| OpenMP, `OMP_WAIT_POLICY=passive`           |    79.4 s |          |      62.4 s |            |
+
+- The new functions make the training 1.4 times faster on both machines.
+- The results for `DOTPRODUCT=generic` are the same before and after.
+- OpenMP does not pay off. The LSTM code starts a parallel region for each
+  timestep, which costs more than the small matrices gain. On macOS (libomp)
+  it is several times slower. On Linux (libgomp) it is slower by default, and
+  with bound threads it is 1.06 (after) to 1.24 (before) times faster for
+  more than twice the CPU time.
+- The results with and without OpenMP are identical.
+- A process which is not pinned runs on the Cortex-X925 cores by itself.
+
+### Profile of the training (M1, share of the samples of the main thread)
+
+| Function (with everything it calls)           | Before | After |
+| --------------------------------------------- | -----: | ----: |
+| `WeightMatrix::SumOuterTransposed` (gradients) |  25 %  | 12 %  |
+| `WeightMatrix::MatrixDotVector` (forward)      |  22 %  | 24 %  |
+| `WeightMatrix::VectorDotMatrix` (backward)     |  21 %  | 20 %  |
+| `Plumbing::Update` (weight update)             | 3.5 %  | 4.9 % |
+| `LSTMTrainer::MaintainCheckpoints`             | 2.6 %  | 3.8 % |
+| `CTC::ComputeCTCTargets`                       | 1.8 %  | 2.4 % |
+| `ImageData::PreScale` (decode and scale image) | 0.8 %  |       |
+
+The shares after the change refer to a total which is 1.4 times smaller.
+`VectorDotMatrix` is inlined after the change, its share is the one of
+`RowDotProductsNEON` called from `LSTM::Backward`.
+
+### Float matrix products (`neon-kernel-variants.cpp`)
+
+Time for the largest layer (Lfx192, 192 x 289 weights for each of the four
+gates), single dot products compared with the implemented variants:
+
+| Product                                  | M1 single | M1 new | X925 single | X925 new |
+| ---------------------------------------- | --------: | -----: | ----------: | -------: |
+| Forward, 4 gates, per timestep (4x2)     |   17.9 us | 14.1 us |    11.0 us |   9.0 us |
+| Backward, 4 gates, per timestep (4x2)    |   17.0 us | 13.3 us |     9.5 us |   7.5 us |
+| Gradients, 1 gate, 300 timesteps (4x4)   |   1.39 ms | 0.39 ms |    0.78 ms |  0.25 ms |
+
+- The smaller layers gain more for the products with a vector (up to 1.9
+  times).
+- More accumulators for a single dot product gain much less in the context
+  of a matrix product (about 7 % on M1) than for a single call.
+- Variants which keep the accumulators in an array of vectors instead of
+  named variables were two to four times slower with g++ 13.
+
+### Integer matrix product (`neon-int-kernel-variants.cpp`)
+
+Recognition of three test images with `tessdata_fast/eng` on M1 spends 73 %
+of the time in `LSTMRecognizer::RecognizeLine` and 30 % in the integer
+`matrixDotVector` of `src/arch/intsimdmatrixneon.cpp`. The variants are not
+implemented in the library yet. Time for the four gates of a layer per
+timestep:
+
+| Layer  | M1 now | M1 padal | M1 sdot | X925 now | X925 padal | X925 sdot |
+| ------ | -----: | -------: | ------: | -------: | ---------: | --------: |
+| Lfys48 | 0.70 us |  0.23 us | 0.15 us |  0.45 us |    0.19 us |   0.09 us |
+| Lfx96  | 2.47 us |  0.89 us | 0.53 us |  2.00 us |    0.81 us |   0.35 us |
+| Lrx96  | 3.32 us |  1.17 us | 0.69 us |  2.68 us |    1.08 us |   0.50 us |
+| Lfx192 | 9.89 us |  3.45 us | 3.11 us |  8.00 us |    3.21 us |   1.75 us |
+| Lfx512 | 55.8 us |  19.6 us | 18.0 us |  45.2 us |    17.9 us |   10.5 us |
+
+"padal" works on any ARM64 CPU, "sdot" needs the dot product instructions of
+ARMv8.2. All variants give identical results.
diff --git a/benchmark/lstm/bench.sh b/benchmark/lstm/bench.sh
new file mode 100755
index 00000000..a8ed9e64
--- /dev/null
+++ b/benchmark/lstm/bench.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+# Times lstmtraining for a fixed number of training lines.
+#
+# Usage: bench.sh BUILDDIR [COMMAND PREFIX ...]
+#
+#   BUILDDIR  build directory with bin/lstmtraining and bin/combine_lang_model,
+#             absolute or relative to the Tesseract source directory.
+#   PREFIX    optional command which runs lstmtraining, for example
+#             "env OMP_THREAD_LIMIT=1" or "taskset -c 5-9,15-19".
+#
+# Environment variables:
+#   ITER      number of training iterations (default 1000)
+#   NAME      name of the run for the log and the output files
+#             (default: name of the build directory)
+#   NET_SPEC  network specification (default: the one of tesstrain)
+#   WORK      directory for the model, checkpoints and logs
+#             (default: $TMPDIR/tess-train-bench)
+#   REPO      Tesseract source directory (default: the one of this script)
+#
+# The training data is test/testdata/eng.Arial.exp0.lstmf (1036 lines), the
+# starter traineddata is made from test/testdata/eng/eng.unicharset and the
+# sibling directory ../langdata_lstm. The training starts from scratch, so
+# the times of different builds are comparable, and the last log line shows
+# whether the results are the same.
+#
+# Run only one benchmark at a time and look at the load of the machine
+# (uptime) before trusting a result.
+
+set -e
+
+if [ $# -lt 1 ]; then
+  sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'
+  exit 1
+fi
+
+REPO=${REPO:-$(cd "$(dirname "$0")/../.." && pwd)}
+ITER=${ITER:-1000}
+NET_SPEC=${NET_SPEC:-[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx192 O1c111]}
+WORK=${WORK:-${TMPDIR:-/tmp}/tess-train-bench}
+
+builddir=$1
+shift
+case $builddir in
+  /*) ;;
+  *) builddir=$REPO/$builddir ;;
+esac
+NAME=${NAME:-$(basename "$builddir")}
+if [ $# -eq 0 ]; then
+  set -- env
+fi
+
+lstmf=$REPO/test/testdata/eng.Arial.exp0.lstmf
+for f in "$builddir/bin/lstmtraining" "$builddir/bin/combine_lang_model" "$lstmf" \
+         "$REPO/test/testdata/eng/eng.unicharset" "$REPO/../langdata_lstm/eng"; do
+  if [ ! -e "$f" ]; then
+    echo "Missing $f" >&2
+    exit 1
+  fi
+done
+
+mkdir -p "$WORK/out" "$WORK/lang"
+if [ ! -f "$WORK/lang/eng/eng.traineddata" ]; then
+  "$builddir/bin/combine_lang_model" \
+    --input_unicharset "$REPO/test/testdata/eng/eng.unicharset" \
+    --script_dir "$REPO/../langdata_lstm" --output_dir "$WORK/lang" --lang eng \
+    > "$WORK/combine_lang_model.log" 2>&1 || {
+    tail -5 "$WORK/combine_lang_model.log" >&2
+    exit 1
+  }
+fi
+echo "$lstmf" > "$WORK/train.list"
+
+log=$WORK/$NAME.log
+rm -f "$WORK/out/$NAME"_* "$WORK/out/$NAME".*
+LC_ALL=C /usr/bin/time -p "$@" "$builddir/bin/lstmtraining" \
+  --traineddata "$WORK/lang/eng/eng.traineddata" --net_spec "$NET_SPEC" \
+  --model_output "$WORK/out/$NAME" --train_listfile "$WORK/train.list" \
+  --max_iterations "$ITER" > "$log" 2>&1 || {
+  tail -5 "$log" >&2
+  exit 1
+}
+
+times=$(grep -E '^(real|user|sys) ' "$log" | tr '\n' ' ')
+last=$(grep 'At iteration' "$log" | tail -1 | sed -e 's/^At iteration //' | cut -c1-76)
+printf '%-18s %s| %s\n' "$NAME:" "$times" "$last"
diff --git a/benchmark/lstm/neon-int-kernel-variants.cpp b/benchmark/lstm/neon-int-kernel-variants.cpp
new file mode 100644
index 00000000..bb28fe90
--- /dev/null
+++ b/benchmark/lstm/neon-int-kernel-variants.cpp
@@ -0,0 +1,209 @@
+///////////////////////////////////////////////////////////////////////
+// File:        neon-int-kernel-variants.cpp
+// Description: Microbenchmark for variants of the NEON function which
+//              multiplies an int8_t matrix with a vector.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+// http://www.apache.org/licenses/LICENSE-2.0
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+///////////////////////////////////////////////////////////////////////
+
+// This standalone program (ARM64 only) compares the core loop of
+// PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp, which is used
+// for the recognition with integer (fast) models, with two variants:
+// one for any ARM64 CPU and one with the dot product instruction SDOT,
+// which needs ARMv8.2 with the feature "asimddp" (see /proc/cpuinfo).
+// The scaling of the results and the bias are not included. All variants
+// give the same results, because the calculation with integers is exact.
+//
+//   c++ -std=c++17 -O3 -o neon-int-kernel-variants neon-int-kernel-variants.cpp
+//   ./neon-int-kernel-variants
+
+#include <arm_neon.h>
+#include <chrono>
+#include <cstdint>
+#include <cstdio>
+#include <vector>
+#define NOINLINE __attribute__((noinline))
+
+// Core loop of PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp
+// (8 outputs, weights interleaved in groups of 8 rows x 8 inputs).
+NOINLINE void Now8(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
+  int32x4_t result0123 = {0, 0, 0, 0};
+  int32x4_t result4567 = {0, 0, 0, 0};
+  for (int j = 0; j < num_in; j += 8) {
+    int8x8_t vu = vld1_s8(u);
+    int8x16_t vw01 = vld1q_s8(wi);
+    int8x16_t vw23 = vld1q_s8(wi + 8 * 2);
+    int8x16_t vw45 = vld1q_s8(wi + 8 * 4);
+    int8x16_t vw67 = vld1q_s8(wi + 8 * 6);
+    int32x4_t r0 = vpaddlq_s16(vmull_s8(vget_low_s8(vw01), vu));
+    int32x4_t r1 = vpaddlq_s16(vmull_s8(vget_high_s8(vw01), vu));
+    int32x4_t r2 = vpaddlq_s16(vmull_s8(vget_low_s8(vw23), vu));
+    int32x4_t r3 = vpaddlq_s16(vmull_s8(vget_high_s8(vw23), vu));
+    int32x4_t r4 = vpaddlq_s16(vmull_s8(vget_low_s8(vw45), vu));
+    int32x4_t r5 = vpaddlq_s16(vmull_s8(vget_high_s8(vw45), vu));
+    int32x4_t r6 = vpaddlq_s16(vmull_s8(vget_low_s8(vw67), vu));
+    int32x4_t r7 = vpaddlq_s16(vmull_s8(vget_high_s8(vw67), vu));
+    r0 = vcombine_s32(vpadd_s32(vget_low_s32(r0), vget_high_s32(r0)),
+                      vpadd_s32(vget_low_s32(r1), vget_high_s32(r1)));
+    r2 = vcombine_s32(vpadd_s32(vget_low_s32(r2), vget_high_s32(r2)),
+                      vpadd_s32(vget_low_s32(r3), vget_high_s32(r3)));
+    r4 = vcombine_s32(vpadd_s32(vget_low_s32(r4), vget_high_s32(r4)),
+                      vpadd_s32(vget_low_s32(r5), vget_high_s32(r5)));
+    r6 = vcombine_s32(vpadd_s32(vget_low_s32(r6), vget_high_s32(r6)),
+                      vpadd_s32(vget_low_s32(r7), vget_high_s32(r7)));
+    r0 = vcombine_s32(vpadd_s32(vget_low_s32(r0), vget_high_s32(r0)),
+                      vpadd_s32(vget_low_s32(r2), vget_high_s32(r2)));
+    r4 = vcombine_s32(vpadd_s32(vget_low_s32(r4), vget_high_s32(r4)),
+                      vpadd_s32(vget_low_s32(r6), vget_high_s32(r6)));
+    result0123 = vaddq_s32(result0123, r0);
+    result4567 = vaddq_s32(result4567, r4);
+    u += 8;
+    wi += 64;
+  }
+  vst1q_s32(v, result0123);
+  vst1q_s32(v + 4, result4567);
+}
+NOINLINE void NowMatrix(const int8_t *wi, int rows, int num_in, const int8_t *u,
+                        int32_t *v) {
+  for (int i = 0; i < rows; i += 8)
+    Now8(wi + static_cast<size_t>(i / 8) * num_in * 8, u, num_in, v + i);
+}
+
+// Without SDOT: widening multiply and pairwise accumulate, 4 rows per pass.
+NOINLINE void Padal4(const int8_t *w, int rows, int num_in, const int8_t *u,
+                     int32_t *v) {
+  for (int i = 0; i < rows; i += 4) {
+    const int8_t *w0 = w + static_cast<size_t>(i) * num_in, *w1 = w0 + num_in,
+                 *w2 = w1 + num_in, *w3 = w2 + num_in;
+    int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
+    for (int j = 0; j < num_in; j += 16) {
+      int8x16_t x = vld1q_s8(u + j);
+      int8x16_t y;
+      y = vld1q_s8(w0 + j);
+      a0 = vpadalq_s16(
+          a0, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+      y = vld1q_s8(w1 + j);
+      a1 = vpadalq_s16(
+          a1, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+      y = vld1q_s8(w2 + j);
+      a2 = vpadalq_s16(
+          a2, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+      y = vld1q_s8(w3 + j);
+      a3 = vpadalq_s16(
+          a3, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+    }
+    v[i] = vaddvq_s32(a0);
+    v[i + 1] = vaddvq_s32(a1);
+    v[i + 2] = vaddvq_s32(a2);
+    v[i + 3] = vaddvq_s32(a3);
+  }
+}
+
+// With SDOT (ARMv8.2 dot product), 4 rows per pass, 16 inputs per instruction.
+__attribute__((target("arch=armv8.2-a+dotprod"))) NOINLINE void Sdot4(
+    const int8_t *w, int rows, int num_in, const int8_t *u, int32_t *v) {
+  for (int i = 0; i < rows; i += 4) {
+    const int8_t *w0 = w + static_cast<size_t>(i) * num_in, *w1 = w0 + num_in,
+                 *w2 = w1 + num_in, *w3 = w2 + num_in;
+    int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
+    for (int j = 0; j < num_in; j += 16) {
+      int8x16_t x = vld1q_s8(u + j);
+      a0 = vdotq_s32(a0, vld1q_s8(w0 + j), x);
+      a1 = vdotq_s32(a1, vld1q_s8(w1 + j), x);
+      a2 = vdotq_s32(a2, vld1q_s8(w2 + j), x);
+      a3 = vdotq_s32(a3, vld1q_s8(w3 + j), x);
+    }
+    v[i] = vaddvq_s32(a0);
+    v[i + 1] = vaddvq_s32(a1);
+    v[i + 2] = vaddvq_s32(a2);
+    v[i + 3] = vaddvq_s32(a3);
+  }
+}
+
+template <class F>
+static double BestNs(F f, int reps) {
+  double best = 1e30;
+  for (int r = 0; r < 9; r++) {
+    auto t0 = std::chrono::steady_clock::now();
+    for (int k = 0; k < reps; k++)
+      f();
+    double ns = std::chrono::duration<double, std::nano>(
+                    std::chrono::steady_clock::now() - t0)
+                    .count() /
+                reps;
+    if (ns < best)
+      best = ns;
+  }
+  return best;
+}
+
+// Returns deterministic pseudo-random numbers with 24 bits for the test data.
+// A linear congruential generator gives the same data on all platforms.
+static uint32_t Random() {
+  static uint32_t state = 1;
+  state = state * 1664525u + 1013904223u;
+  return state >> 8;
+}
+
+int main() {
+  struct {
+    const char *name;
+    int ns, na;
+  } layers[] = {{"Lfys48", 48, 64},
+                {"Lfx96", 96, 144},
+                {"Lrx96", 96, 192},
+                {"Lfx192", 192, 288},
+                {"Lfx512", 512, 608}};
+  printf(
+      "int8 W*u (4 gate matrices), us per timestep\n"
+      " layer      now  padal   sdot   now/sdot\n");
+  for (auto &L : layers) {
+    int rows = L.ns, cols = (L.na + 15) / 16 * 16;
+    std::vector<int8_t> w(4 * rows * cols), wi(4 * rows * cols), u(cols);
+    for (auto &e : w)
+      e = Random() % 255 - 127;
+    for (auto &e : u)
+      e = Random() % 255 - 127;
+    // Interleaved copy for the current code: blocks of 8 rows, groups of 8
+    // inputs.
+    for (int g = 0; g < 4; g++)
+      for (int i = 0; i < rows; i++)
+        for (int j = 0; j < cols; j++)
+          wi[g * rows * cols + (i / 8) * cols * 8 + (j / 8) * 64 + (i % 8) * 8 +
+             j % 8] = w[g * rows * cols + i * cols + j];
+    std::vector<int32_t> v0(4 * rows), v1(4 * rows), v2(4 * rows);
+    double t0 = BestNs(
+        [&] {
+          for (int g = 0; g < 4; g++)
+            NowMatrix(wi.data() + g * rows * cols, rows, cols, u.data(),
+                      v0.data() + g * rows);
+        },
+        5000);
+    double t1 = BestNs(
+        [&] {
+          for (int g = 0; g < 4; g++)
+            Padal4(w.data() + g * rows * cols, rows, cols, u.data(),
+                   v1.data() + g * rows);
+        },
+        5000);
+    double t2 = BestNs(
+        [&] {
+          for (int g = 0; g < 4; g++)
+            Sdot4(w.data() + g * rows * cols, rows, cols, u.data(),
+                  v2.data() + g * rows);
+        },
+        5000);
+    printf(" %-7s %6.2f %6.2f %6.2f   %.1fx %s\n", L.name, t0 / 1e3, t1 / 1e3,
+           t2 / 1e3, t0 / t2,
+           (v0 == v1 && v0 == v2) ? "(identical results)" : "(RESULTS DIFFER)");
+  }
+  return 0;
+}
diff --git a/benchmark/lstm/neon-kernel-variants.cpp b/benchmark/lstm/neon-kernel-variants.cpp
new file mode 100644
index 00000000..d251eafa
--- /dev/null
+++ b/benchmark/lstm/neon-kernel-variants.cpp
@@ -0,0 +1,444 @@
+///////////////////////////////////////////////////////////////////////
+// File:        neon-kernel-variants.cpp
+// Description: Microbenchmark for variants of the NEON functions which
+//              calculate several dot products at once.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+// http://www.apache.org/licenses/LICENSE-2.0
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+///////////////////////////////////////////////////////////////////////
+
+// This standalone program (ARM64 only) compares the time of single dot
+// products, called by a function pointer like in the generic code, with
+// variants which handle several rows per pass. The sizes are those of the
+// layers of the default network of tesstrain. It was used to choose the
+// implementations of RowDotProductsNEON (Rows4x2) and CrossDotProductsNEON
+// (Cross4x4) in src/arch/dotproductneon.cpp and can be used to try other
+// variants on a new CPU or with a new compiler:
+//
+//   c++ -std=c++17 -O3 -o neon-kernel-variants neon-kernel-variants.cpp
+//   ./neon-kernel-variants
+//
+// The accumulators are named variables. Variants with arrays of vectors were
+// much slower with g++ 13, which did not keep them in registers.
+
+#include <arm_neon.h>
+#include <chrono>
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <vector>
+
+#define NOINLINE __attribute__((noinline))
+
+// Dot product like DotProductNEON for FAST_FLOAT.
+NOINLINE float Dot2(const float *u, const float *v, int n) {
+  float32x4_t r0 = vdupq_n_f32(0.0f), r1 = vdupq_n_f32(0.0f);
+  while (n > 7) {
+    r0 = vfmaq_f32(r0, vld1q_f32(u), vld1q_f32(v));
+    r1 = vfmaq_f32(r1, vld1q_f32(u + 4), vld1q_f32(v + 4));
+    u += 8;
+    v += 8;
+    n -= 8;
+  }
+  float total = vaddvq_f32(r0);
+  total += vaddvq_f32(r1);
+  while (n > 0) {
+    total += *u++ * *v++;
+    n--;
+  }
+  return total;
+}
+typedef float (*DotFn)(const float *, const float *, int);
+volatile DotFn g_dot = Dot2;
+
+// Single dot products for each row, like the generic code.
+NOINLINE void RowsNow(const float *w, int stride, int rows, const float *u,
+                      int n, float *v) {
+  DotFn dot = g_dot;
+  for (int i = 0; i < rows; i++)
+    v[i] = dot(w + i * stride, u, n);
+}
+
+// 4 rows per pass, 2 accumulators per row (8 columns per iteration).
+NOINLINE void Rows4x2(const float *w, int stride, int rows, const float *u,
+                      int n, float *v) {
+  int i = 0;
+  for (; i + 4 <= rows; i += 4) {
+    const float *w0 = w + static_cast<size_t>(i) * stride, *w1 = w0 + stride,
+                *w2 = w1 + stride, *w3 = w2 + stride;
+    float32x4_t a0 = vdupq_n_f32(0), a1 = a0, a2 = a0, a3 = a0, b0 = a0,
+                b1 = a0, b2 = a0, b3 = a0;
+    int j = 0;
+    for (; j + 8 <= n; j += 8) {
+      float32x4_t x = vld1q_f32(u + j), y = vld1q_f32(u + j + 4);
+      a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+      b0 = vfmaq_f32(b0, vld1q_f32(w0 + j + 4), y);
+      a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+      b1 = vfmaq_f32(b1, vld1q_f32(w1 + j + 4), y);
+      a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+      b2 = vfmaq_f32(b2, vld1q_f32(w2 + j + 4), y);
+      a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+      b3 = vfmaq_f32(b3, vld1q_f32(w3 + j + 4), y);
+    }
+    if (j + 4 <= n) {
+      float32x4_t x = vld1q_f32(u + j);
+      a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+      a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+      a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+      a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+      j += 4;
+    }
+    float t0 = vaddvq_f32(vaddq_f32(a0, b0)),
+          t1 = vaddvq_f32(vaddq_f32(a1, b1));
+    float t2 = vaddvq_f32(vaddq_f32(a2, b2)),
+          t3 = vaddvq_f32(vaddq_f32(a3, b3));
+    for (; j < n; j++) {
+      t0 += w0[j] * u[j];
+      t1 += w1[j] * u[j];
+      t2 += w2[j] * u[j];
+      t3 += w3[j] * u[j];
+    }
+    v[i] = t0;
+    v[i + 1] = t1;
+    v[i + 2] = t2;
+    v[i + 3] = t3;
+  }
+  for (; i < rows; i++)
+    v[i] = Dot2(w + static_cast<size_t>(i) * stride, u, n);
+}
+
+// 4 rows per pass, 4 accumulators per row (16 columns per iteration).
+NOINLINE void Rows4x4(const float *w, int stride, int rows, const float *u,
+                      int n, float *v) {
+  int i = 0;
+  for (; i + 4 <= rows; i += 4) {
+    const float *w0 = w + static_cast<size_t>(i) * stride, *w1 = w0 + stride,
+                *w2 = w1 + stride, *w3 = w2 + stride;
+    float32x4_t z = vdupq_n_f32(0);
+    float32x4_t a0 = z, a1 = z, a2 = z, a3 = z, b0 = z, b1 = z, b2 = z, b3 = z;
+    float32x4_t c0 = z, c1 = z, c2 = z, c3 = z, d0 = z, d1 = z, d2 = z, d3 = z;
+    int j = 0;
+    for (; j + 16 <= n; j += 16) {
+      float32x4_t x = vld1q_f32(u + j), y = vld1q_f32(u + j + 4),
+                  p = vld1q_f32(u + j + 8), q = vld1q_f32(u + j + 12);
+      a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+      b0 = vfmaq_f32(b0, vld1q_f32(w0 + j + 4), y);
+      c0 = vfmaq_f32(c0, vld1q_f32(w0 + j + 8), p);
+      d0 = vfmaq_f32(d0, vld1q_f32(w0 + j + 12), q);
+      a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+      b1 = vfmaq_f32(b1, vld1q_f32(w1 + j + 4), y);
+      c1 = vfmaq_f32(c1, vld1q_f32(w1 + j + 8), p);
+      d1 = vfmaq_f32(d1, vld1q_f32(w1 + j + 12), q);
+      a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+      b2 = vfmaq_f32(b2, vld1q_f32(w2 + j + 4), y);
+      c2 = vfmaq_f32(c2, vld1q_f32(w2 + j + 8), p);
+      d2 = vfmaq_f32(d2, vld1q_f32(w2 + j + 12), q);
+      a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+      b3 = vfmaq_f32(b3, vld1q_f32(w3 + j + 4), y);
+      c3 = vfmaq_f32(c3, vld1q_f32(w3 + j + 8), p);
+      d3 = vfmaq_f32(d3, vld1q_f32(w3 + j + 12), q);
+    }
+    for (; j + 4 <= n; j += 4) {
+      float32x4_t x = vld1q_f32(u + j);
+      a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+      a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+      a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+      a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+    }
+    float t0 = vaddvq_f32(vaddq_f32(vaddq_f32(a0, b0), vaddq_f32(c0, d0))),
+          t1 = vaddvq_f32(vaddq_f32(vaddq_f32(a1, b1), vaddq_f32(c1, d1)));
+    float t2 = vaddvq_f32(vaddq_f32(vaddq_f32(a2, b2), vaddq_f32(c2, d2))),
+          t3 = vaddvq_f32(vaddq_f32(vaddq_f32(a3, b3), vaddq_f32(c3, d3)));
+    for (; j < n; j++) {
+      t0 += w0[j] * u[j];
+      t1 += w1[j] * u[j];
+      t2 += w2[j] * u[j];
+      t3 += w3[j] * u[j];
+    }
+    v[i] = t0;
+    v[i + 1] = t1;
+    v[i + 2] = t2;
+    v[i + 3] = t3;
+  }
+  for (; i < rows; i++)
+    v[i] = Dot2(w + static_cast<size_t>(i) * stride, u, n);
+}
+
+// Single dot products for each pair of rows, like the generic code.
+NOINLINE void CrossNow(const float *u, int nu, const float *v, int nv, int n,
+                       float *out, int os) {
+  DotFn dot = g_dot;
+  for (int i = 0; i < nu; i++)
+    for (int j = 0; j < nv; j++)
+      out[static_cast<size_t>(i) * os + j] = dot(
+          u + static_cast<size_t>(i) * n, v + static_cast<size_t>(j) * n, n);
+}
+
+// 2 rows of u and 4 rows of v per pass.
+NOINLINE void Cross2x4(const float *u, int nu, const float *v, int nv, int n,
+                       float *out, int os) {
+  int i = 0;
+  for (; i + 2 <= nu; i += 2) {
+    const float *u0 = u + static_cast<size_t>(i) * n, *u1 = u0 + n;
+    float *o0 = out + static_cast<size_t>(i) * os, *o1 = o0 + os;
+    int j = 0;
+    for (; j + 4 <= nv; j += 4) {
+      const float *v0 = v + static_cast<size_t>(j) * n, *v1 = v0 + n,
+                  *v2 = v1 + n, *v3 = v2 + n;
+      float32x4_t z = vdupq_n_f32(0);
+      float32x4_t a0 = z, a1 = z, a2 = z, a3 = z, b0 = z, b1 = z, b2 = z,
+                  b3 = z;
+      int t = 0;
+      for (; t + 4 <= n; t += 4) {
+        float32x4_t x0 = vld1q_f32(u0 + t), x1 = vld1q_f32(u1 + t);
+        float32x4_t y0 = vld1q_f32(v0 + t), y1 = vld1q_f32(v1 + t),
+                    y2 = vld1q_f32(v2 + t), y3 = vld1q_f32(v3 + t);
+        a0 = vfmaq_f32(a0, x0, y0);
+        a1 = vfmaq_f32(a1, x0, y1);
+        a2 = vfmaq_f32(a2, x0, y2);
+        a3 = vfmaq_f32(a3, x0, y3);
+        b0 = vfmaq_f32(b0, x1, y0);
+        b1 = vfmaq_f32(b1, x1, y1);
+        b2 = vfmaq_f32(b2, x1, y2);
+        b3 = vfmaq_f32(b3, x1, y3);
+      }
+      float r00 = vaddvq_f32(a0), r01 = vaddvq_f32(a1), r02 = vaddvq_f32(a2),
+            r03 = vaddvq_f32(a3);
+      float r10 = vaddvq_f32(b0), r11 = vaddvq_f32(b1), r12 = vaddvq_f32(b2),
+            r13 = vaddvq_f32(b3);
+      for (; t < n; t++) {
+        r00 += u0[t] * v0[t];
+        r01 += u0[t] * v1[t];
+        r02 += u0[t] * v2[t];
+        r03 += u0[t] * v3[t];
+        r10 += u1[t] * v0[t];
+        r11 += u1[t] * v1[t];
+        r12 += u1[t] * v2[t];
+        r13 += u1[t] * v3[t];
+      }
+      o0[j] = r00;
+      o0[j + 1] = r01;
+      o0[j + 2] = r02;
+      o0[j + 3] = r03;
+      o1[j] = r10;
+      o1[j + 1] = r11;
+      o1[j + 2] = r12;
+      o1[j + 3] = r13;
+    }
+    for (; j < nv; j++) {
+      o0[j] = Dot2(u0, v + static_cast<size_t>(j) * n, n);
+      o1[j] = Dot2(u1, v + static_cast<size_t>(j) * n, n);
+    }
+  }
+  for (; i < nu; i++)
+    Rows4x2(v, n, nv, u + static_cast<size_t>(i) * n, n,
+            out + static_cast<size_t>(i) * os);
+}
+
+// 4 rows of u and 4 rows of v per pass.
+NOINLINE void Cross4x4(const float *u, int nu, const float *v, int nv, int n,
+                       float *out, int os) {
+  int i = 0;
+  for (; i + 4 <= nu; i += 4) {
+    const float *u0 = u + static_cast<size_t>(i) * n, *u1 = u0 + n,
+                *u2 = u1 + n, *u3 = u2 + n;
+    float *o0 = out + static_cast<size_t>(i) * os, *o1 = o0 + os, *o2 = o1 + os,
+          *o3 = o2 + os;
+    int j = 0;
+    for (; j + 4 <= nv; j += 4) {
+      const float *v0 = v + static_cast<size_t>(j) * n, *v1 = v0 + n,
+                  *v2 = v1 + n, *v3 = v2 + n;
+      float32x4_t z = vdupq_n_f32(0);
+      float32x4_t a0 = z, a1 = z, a2 = z, a3 = z, b0 = z, b1 = z, b2 = z,
+                  b3 = z;
+      float32x4_t c0 = z, c1 = z, c2 = z, c3 = z, d0 = z, d1 = z, d2 = z,
+                  d3 = z;
+      int t = 0;
+      for (; t + 4 <= n; t += 4) {
+        float32x4_t y0 = vld1q_f32(v0 + t), y1 = vld1q_f32(v1 + t),
+                    y2 = vld1q_f32(v2 + t), y3 = vld1q_f32(v3 + t);
+        float32x4_t x = vld1q_f32(u0 + t);
+        a0 = vfmaq_f32(a0, x, y0);
+        a1 = vfmaq_f32(a1, x, y1);
+        a2 = vfmaq_f32(a2, x, y2);
+        a3 = vfmaq_f32(a3, x, y3);
+        x = vld1q_f32(u1 + t);
+        b0 = vfmaq_f32(b0, x, y0);
+        b1 = vfmaq_f32(b1, x, y1);
+        b2 = vfmaq_f32(b2, x, y2);
+        b3 = vfmaq_f32(b3, x, y3);
+        x = vld1q_f32(u2 + t);
+        c0 = vfmaq_f32(c0, x, y0);
+        c1 = vfmaq_f32(c1, x, y1);
+        c2 = vfmaq_f32(c2, x, y2);
+        c3 = vfmaq_f32(c3, x, y3);
+        x = vld1q_f32(u3 + t);
+        d0 = vfmaq_f32(d0, x, y0);
+        d1 = vfmaq_f32(d1, x, y1);
+        d2 = vfmaq_f32(d2, x, y2);
+        d3 = vfmaq_f32(d3, x, y3);
+      }
+      float r[4][4] = {
+          {vaddvq_f32(a0), vaddvq_f32(a1), vaddvq_f32(a2), vaddvq_f32(a3)},
+          {vaddvq_f32(b0), vaddvq_f32(b1), vaddvq_f32(b2), vaddvq_f32(b3)},
+          {vaddvq_f32(c0), vaddvq_f32(c1), vaddvq_f32(c2), vaddvq_f32(c3)},
+          {vaddvq_f32(d0), vaddvq_f32(d1), vaddvq_f32(d2), vaddvq_f32(d3)}};
+      for (; t < n; t++) {
+        const float x[4] = {u0[t], u1[t], u2[t], u3[t]},
+                    y[4] = {v0[t], v1[t], v2[t], v3[t]};
+        for (int a = 0; a < 4; a++)
+          for (int b = 0; b < 4; b++)
+            r[a][b] += x[a] * y[b];
+      }
+      for (int b = 0; b < 4; b++) {
+        o0[j + b] = r[0][b];
+        o1[j + b] = r[1][b];
+        o2[j + b] = r[2][b];
+        o3[j + b] = r[3][b];
+      }
+    }
+    for (; j < nv; j++) {
+      const float *vj = v + static_cast<size_t>(j) * n;
+      o0[j] = Dot2(u0, vj, n);
+      o1[j] = Dot2(u1, vj, n);
+      o2[j] = Dot2(u2, vj, n);
+      o3[j] = Dot2(u3, vj, n);
+    }
+  }
+  for (; i < nu; i++)
+    Rows4x2(v, n, nv, u + static_cast<size_t>(i) * n, n,
+            out + static_cast<size_t>(i) * os);
+}
+
+// Returns the best time per call in ns of several measurements.
+template <class F>
+static double BestNs(F f, int reps) {
+  double best = 1e30;
+  for (int r = 0; r < 9; r++) {
+    auto t0 = std::chrono::steady_clock::now();
+    for (int k = 0; k < reps; k++)
+      f();
+    double ns = std::chrono::duration<double, std::nano>(
+                    std::chrono::steady_clock::now() - t0)
+                    .count() /
+                reps;
+    if (ns < best)
+      best = ns;
+  }
+  return best;
+}
+// Returns the largest difference of two results.
+static double MaxDiff(const std::vector<float> &a,
+                      const std::vector<float> &b) {
+  double m = 0;
+  for (size_t i = 0; i < a.size(); i++)
+    m = std::fmax(m, std::fabs(a[i] - b[i]));
+  return m;
+}
+
+// Returns deterministic pseudo-random numbers with 24 bits for the test data.
+// A linear congruential generator gives the same data on all platforms.
+static uint32_t Random() {
+  static uint32_t state = 1;
+  state = state * 1664525u + 1013904223u;
+  return state >> 8;
+}
+
+int main() {
+  auto fill = [](std::vector<float> &x) {
+    for (auto &e : x)
+      e = Random() / (float)0x1000000 - 0.5f;
+  };
+  struct {
+    const char *name;
+    int ns, na;
+  } layers[] = {{"Conv16", 16, 9},  {"Lfys48", 48, 64},   {"Lfx96", 96, 144},
+                {"Lrx96", 96, 192}, {"Lfx192", 192, 288}, {"Out111", 111, 192},
+                {"odd", 113, 195}};
+  typedef void (*RowFn)(const float *, int, int, const float *, int, float *);
+  RowFn rf[] = {RowsNow, Rows4x2, Rows4x4};
+  printf(
+      "W*u forward (4 gate matrices in turn), us per timestep\n"
+      " layer   single    4x2    4x4\n");
+  for (auto &L : layers) {
+    int stride = L.na + 1;
+    std::vector<float> w(4 * L.ns * stride), u(L.na), ref(4 * L.ns),
+        v(4 * L.ns);
+    fill(w);
+    fill(u);
+    printf(" %-7s", L.name);
+    for (int k = 0; k < 3; k++) {
+      auto &dst = k == 0 ? ref : v;
+      double t = BestNs(
+          [&] {
+            for (int g = 0; g < 4; g++)
+              rf[k](w.data() + g * L.ns * stride, stride, L.ns, u.data(), L.na,
+                    dst.data() + g * L.ns);
+          },
+          5000);
+      printf(" %6.2f", t / 1e3);
+      if (k && MaxDiff(ref, v) > 1e-3)
+        printf("(DIFF %g)", MaxDiff(ref, v));
+    }
+    printf("\n");
+  }
+  printf(
+      "u*W backward (4 transposed gate matrices), us per timestep\n"
+      " layer   single    4x2    4x4\n");
+  for (auto &L : layers) {
+    int stride = L.ns;
+    std::vector<float> w(4 * (L.na + 1) * stride), u(L.ns), ref(4 * L.na),
+        v(4 * L.na);
+    fill(w);
+    fill(u);
+    printf(" %-7s", L.name);
+    for (int k = 0; k < 3; k++) {
+      auto &dst = k == 0 ? ref : v;
+      double t = BestNs(
+          [&] {
+            for (int g = 0; g < 4; g++)
+              rf[k](w.data() + g * (L.na + 1) * stride, stride, L.na, u.data(),
+                    L.ns, dst.data() + g * L.na);
+          },
+          5000);
+      printf(" %6.2f", t / 1e3);
+      if (k && MaxDiff(ref, v) > 1e-3)
+        printf("(DIFF %g)", MaxDiff(ref, v));
+    }
+    printf("\n");
+  }
+  typedef void (*CrossFn)(const float *, int, const float *, int, int, float *,
+                          int);
+  CrossFn cf[] = {CrossNow, Cross2x4, Cross4x4};
+  printf(
+      "gradient sum (one gate) for T timesteps, ms per call\n"
+      " layer     T  single    2x4    4x4\n");
+  for (int T : {37, 100, 300, 1000}) {
+    for (auto &L : layers) {
+      int os = L.na + 1;
+      std::vector<float> u(L.ns * T), vv(L.na * T), ref(L.ns * os),
+          d(L.ns * os);
+      fill(u);
+      fill(vv);
+      printf(" %-7s %4d", L.name, T);
+      for (int k = 0; k < 3; k++) {
+        auto &dst = k == 0 ? ref : d;
+        double t = BestNs(
+            [&] { cf[k](u.data(), L.ns, vv.data(), L.na, T, dst.data(), os); },
+            40);
+        printf(" %6.3f", t / 1e6);
+        if (k && MaxDiff(ref, d) > 2e-3)
+          printf("(DIFF %g)", MaxDiff(ref, d));
+      }
+      printf("\n");
+    }
+  }
+  return 0;
+}
diff --git a/benchmark/lstm/sample-summary.py b/benchmark/lstm/sample-summary.py
new file mode 100755
index 00000000..6a4f429f
--- /dev/null
+++ b/benchmark/lstm/sample-summary.py
@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+
+# Summarize a call graph which was written by the macOS tool "sample".
+#
+# Usage:
+#   sample PID SECONDS 1 -file sample.txt
+#   sample-summary.py sample.txt [NUMBER_OF_FUNCTIONS]
+#
+# Prints the functions of the main thread with the largest inclusive sample
+# counts (a function and everything which it calls), and the callers of the
+# dot product functions. "sample" splits the counts of a function by call
+# site, so its own summary does not show the totals.
+
+import collections
+import re
+import sys
+
+# Functions for which the direct callers are listed.
+LEAF_FUNCTIONS = ('DotProductNEON', 'RowDotProductsNEON',
+                  'CrossDotProductsNEON')
+
+
+def main():
+    if len(sys.argv) < 2:
+        sys.exit(f'Usage: {sys.argv[0]} sample.txt [NUMBER_OF_FUNCTIONS]')
+    limit = int(sys.argv[2]) if len(sys.argv) > 2 else 40
+    with open(sys.argv[1]) as f:
+        lines = f.read().split('\n')
+    start = next(i for i, l in enumerate(lines) if l.startswith('Call graph:'))
+    end = next(i for i, l in enumerate(lines)
+               if l.startswith('Total number in stack'))
+    # Each line is the indentation (which gives the depth), the count and
+    # the name of the function.
+    pattern = re.compile(r'^([ +!:|]*?)(\d+) (.*)$')
+    inclusive = collections.Counter()
+    callers = collections.Counter()
+    stack = []  # (depth, name) of the callers of the current line
+    total = 0
+    in_main_thread = False
+    for line in lines[start + 1:end]:
+        match = pattern.match(line)
+        if not match:
+            continue
+        depth = len(match.group(1))
+        count = int(match.group(2))
+        name = re.sub(r'  \(in .*$', '', match.group(3))
+        name = re.sub(r'\(.*$', '', name).replace('tesseract::', '')
+        if name.startswith('Thread_'):
+            in_main_thread = 'main-thread' in line
+            if in_main_thread:
+                total = count
+            stack = []
+            continue
+        if not in_main_thread:
+            continue
+        while stack and stack[-1][0] >= depth:
+            stack.pop()
+        ancestors = [n for _, n in stack]
+        # Count recursive calls only once.
+        if name not in ancestors:
+            inclusive[name] += count
+        if name in LEAF_FUNCTIONS and ancestors:
+            caller = ancestors[-1]
+            caller2 = ancestors[-2] if len(ancestors) > 1 else ''
+            callers[(name, caller, caller2)] += count
+        stack.append((depth, name))
+    if total == 0:
+        sys.exit('No samples for the main thread found')
+    print(f'Samples of the main thread: {total}')
+    for name, count in inclusive.most_common(limit):
+        print(f'{count:7d} {100.0 * count / total:5.1f}%  {name}')
+    if callers:
+        print('Callers of the dot product functions:')
+        for (name, caller, caller2), count in callers.most_common(12):
+            print(f'{count:7d} {100.0 * count / total:5.1f}%  '
+                  f'{name} <- {caller} <- {caller2}')
+
+
+if __name__ == '__main__':
+    main()