Commit a08c1040 for tesseract
commit a08c1040e2d6d45f85fa766db8dfeafd504fb549
Author: Stefan Weil <sw@weilnetz.de>
Date: Sat Nov 14 09:59:33 2026 +0100
Add benchmarks for the LSTM code
Add the tools which were used to measure and to tune the speed of LSTM
training on ARM64, and the results:
- bench.sh times lstmtraining for a fixed number of training lines
with a given build and optional settings (environment, CPU pinning).
- sample-summary.py summarizes a profile made with the macOS tool
sample.
- neon-kernel-variants.cpp compares variants of the NEON functions for
several float dot products at once.
- neon-int-kernel-variants.cpp compares variants for the product of an
int8_t matrix with a vector, which is used for the recognition with
integer models. They are 3 to 5 times faster than the current code,
but not implemented in the library yet.
- README.md describes the usage and the results for an Apple M1 Pro
and an NVIDIA DGX Spark, with and without OpenMP.
Assisted-by: OpenCode / Claude Fable 5.2 (Anthropic)
Signed-off-by: Stefan Weil <sw@weilnetz.de>
diff --git a/benchmark/lstm/README.md b/benchmark/lstm/README.md
new file mode 100644
index 00000000..8dc00964
--- /dev/null
+++ b/benchmark/lstm/README.md
@@ -0,0 +1,144 @@
+# Benchmarks for the LSTM code
+
+Tools to measure the speed of LSTM training (and of the matrix products
+which dominate training and recognition), and the results which were
+measured with them.
+
+## Files
+
+- `bench.sh` times `lstmtraining` for a fixed number of training lines.
+- `sample-summary.py` summarizes a profile which was made with the macOS tool
+ `sample`.
+- `neon-kernel-variants.cpp` is a microbenchmark for the float matrix products
+ (variants of `RowDotProductsNEON` and `CrossDotProductsNEON`).
+- `neon-int-kernel-variants.cpp` is a microbenchmark for the `int8_t` matrix
+ product which is used for the recognition with integer (fast) models.
+
+The two microbenchmarks are standalone programs for ARM64. How to build
+them is described at the beginning of each file.
+
+## Timing the training
+
+`bench.sh` trains the default network of
+[tesstrain](https://github.com/tesseract-ocr/tesstrain)
+(`[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx192 O1c111]`) from scratch
+on the 1036 lines of `test/testdata/eng.Arial.exp0.lstmf`. It needs a build
+directory with the training tools, the `test` submodule and the sibling
+directory `../langdata_lstm`.
+
+ benchmark/lstm/bench.sh build
+ NAME=generic benchmark/lstm/bench.sh build env DOTPRODUCT=generic
+ NAME=omp1 benchmark/lstm/bench.sh build-openmp env OMP_THREAD_LIMIT=1
+ NAME=x925 benchmark/lstm/bench.sh build taskset -c 5-9,15-19
+
+Each run prints one line with the times in seconds and the last progress
+message of the training, which shows whether two runs calculated the same
+results:
+
+ build: real 15.83 user 15.41 sys 0.27 | 999/1000/1000, mean rms=5.173%, ...
+
+The number of iterations, the network and the working directory can be set
+with environment variables (see the comment in the script).
+
+Run only one benchmark at a time (several runs disturb each other, and
+tests which run at the same time do so, too) and repeat a measurement if
+the machine was busy.
+
+## Profiling
+
+On macOS, `sample` works without special rights:
+
+ ITER=2500 benchmark/lstm/bench.sh build &
+ sleep 4
+ sample $(pgrep -n lstmtraining) 20 1 -file sample.txt
+ benchmark/lstm/sample-summary.py sample.txt
+
+On Linux, `perf record` / `perf report` can be used if
+`/proc/sys/kernel/perf_event_paranoid` allows it (a value of 2 or less).
+
+## Results
+
+Measured in November 2026 with 1000 training lines, FAST_FLOAT, a Release
+build with CMake, and a single thread unless noted otherwise.
+
+- M1: Apple M1 Pro, macOS, Apple clang 21, performance cores.
+- X925: NVIDIA DGX Spark (10 Cortex-X925 and 10 Cortex-A725 cores),
+ Ubuntu 24.04, g++ 13, pinned to the Cortex-X925 cores.
+
+"After" is the commit "lstm: calculate several dot products at once with
+NEON", "before" is its parent.
+
+### Training time
+
+| Build and settings | M1 before | M1 after | X925 before | X925 after |
+| ------------------------------------------- | --------: | -------: | ----------: | ---------: |
+| Without OpenMP | 22.2 s | 15.9 s | 14.4 s | 10.2 s |
+| Without OpenMP, Cortex-A725 cores | | | 31.3 s | 24.3 s |
+| Without OpenMP, `DOTPRODUCT=generic` | | 85.2 s | 51.5 s | 50.6 s |
+| OpenMP, `OMP_THREAD_LIMIT=1` | 23.0 s | | 15.5 s | |
+| OpenMP, default (4 threads) | 56 - 98 s | | 15.5 s | 12.8 s |
+| OpenMP, `OMP_PROC_BIND=true OMP_PLACES=cores` | | | 11.6 s | 9.6 s |
+| OpenMP, `OMP_WAIT_POLICY=passive` | 79.4 s | | 62.4 s | |
+
+- The new functions make the training 1.4 times faster on both machines.
+- The results for `DOTPRODUCT=generic` are the same before and after.
+- OpenMP does not pay off. The LSTM code starts a parallel region for each
+ timestep, which costs more than the small matrices gain. On macOS (libomp)
+ it is several times slower. On Linux (libgomp) it is slower by default, and
+ with bound threads it is 1.06 (after) to 1.24 (before) times faster for
+ more than twice the CPU time.
+- The results with and without OpenMP are identical.
+- A process which is not pinned runs on the Cortex-X925 cores by itself.
+
+### Profile of the training (M1, share of the samples of the main thread)
+
+| Function (with everything it calls) | Before | After |
+| --------------------------------------------- | -----: | ----: |
+| `WeightMatrix::SumOuterTransposed` (gradients) | 25 % | 12 % |
+| `WeightMatrix::MatrixDotVector` (forward) | 22 % | 24 % |
+| `WeightMatrix::VectorDotMatrix` (backward) | 21 % | 20 % |
+| `Plumbing::Update` (weight update) | 3.5 % | 4.9 % |
+| `LSTMTrainer::MaintainCheckpoints` | 2.6 % | 3.8 % |
+| `CTC::ComputeCTCTargets` | 1.8 % | 2.4 % |
+| `ImageData::PreScale` (decode and scale image) | 0.8 % | |
+
+The shares after the change refer to a total which is 1.4 times smaller.
+`VectorDotMatrix` is inlined after the change, its share is the one of
+`RowDotProductsNEON` called from `LSTM::Backward`.
+
+### Float matrix products (`neon-kernel-variants.cpp`)
+
+Time for the largest layer (Lfx192, 192 x 289 weights for each of the four
+gates), single dot products compared with the implemented variants:
+
+| Product | M1 single | M1 new | X925 single | X925 new |
+| ---------------------------------------- | --------: | -----: | ----------: | -------: |
+| Forward, 4 gates, per timestep (4x2) | 17.9 us | 14.1 us | 11.0 us | 9.0 us |
+| Backward, 4 gates, per timestep (4x2) | 17.0 us | 13.3 us | 9.5 us | 7.5 us |
+| Gradients, 1 gate, 300 timesteps (4x4) | 1.39 ms | 0.39 ms | 0.78 ms | 0.25 ms |
+
+- The smaller layers gain more for the products with a vector (up to 1.9
+ times).
+- More accumulators for a single dot product gain much less in the context
+ of a matrix product (about 7 % on M1) than for a single call.
+- Variants which keep the accumulators in an array of vectors instead of
+ named variables were two to four times slower with g++ 13.
+
+### Integer matrix product (`neon-int-kernel-variants.cpp`)
+
+Recognition of three test images with `tessdata_fast/eng` on M1 spends 73 %
+of the time in `LSTMRecognizer::RecognizeLine` and 30 % in the integer
+`matrixDotVector` of `src/arch/intsimdmatrixneon.cpp`. The variants are not
+implemented in the library yet. Time for the four gates of a layer per
+timestep:
+
+| Layer | M1 now | M1 padal | M1 sdot | X925 now | X925 padal | X925 sdot |
+| ------ | -----: | -------: | ------: | -------: | ---------: | --------: |
+| Lfys48 | 0.70 us | 0.23 us | 0.15 us | 0.45 us | 0.19 us | 0.09 us |
+| Lfx96 | 2.47 us | 0.89 us | 0.53 us | 2.00 us | 0.81 us | 0.35 us |
+| Lrx96 | 3.32 us | 1.17 us | 0.69 us | 2.68 us | 1.08 us | 0.50 us |
+| Lfx192 | 9.89 us | 3.45 us | 3.11 us | 8.00 us | 3.21 us | 1.75 us |
+| Lfx512 | 55.8 us | 19.6 us | 18.0 us | 45.2 us | 17.9 us | 10.5 us |
+
+"padal" works on any ARM64 CPU, "sdot" needs the dot product instructions of
+ARMv8.2. All variants give identical results.
diff --git a/benchmark/lstm/bench.sh b/benchmark/lstm/bench.sh
new file mode 100755
index 00000000..a8ed9e64
--- /dev/null
+++ b/benchmark/lstm/bench.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+# Times lstmtraining for a fixed number of training lines.
+#
+# Usage: bench.sh BUILDDIR [COMMAND PREFIX ...]
+#
+# BUILDDIR build directory with bin/lstmtraining and bin/combine_lang_model,
+# absolute or relative to the Tesseract source directory.
+# PREFIX optional command which runs lstmtraining, for example
+# "env OMP_THREAD_LIMIT=1" or "taskset -c 5-9,15-19".
+#
+# Environment variables:
+# ITER number of training iterations (default 1000)
+# NAME name of the run for the log and the output files
+# (default: name of the build directory)
+# NET_SPEC network specification (default: the one of tesstrain)
+# WORK directory for the model, checkpoints and logs
+# (default: $TMPDIR/tess-train-bench)
+# REPO Tesseract source directory (default: the one of this script)
+#
+# The training data is test/testdata/eng.Arial.exp0.lstmf (1036 lines), the
+# starter traineddata is made from test/testdata/eng/eng.unicharset and the
+# sibling directory ../langdata_lstm. The training starts from scratch, so
+# the times of different builds are comparable, and the last log line shows
+# whether the results are the same.
+#
+# Run only one benchmark at a time and look at the load of the machine
+# (uptime) before trusting a result.
+
+set -e
+
+if [ $# -lt 1 ]; then
+ sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'
+ exit 1
+fi
+
+REPO=${REPO:-$(cd "$(dirname "$0")/../.." && pwd)}
+ITER=${ITER:-1000}
+NET_SPEC=${NET_SPEC:-[1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx192 O1c111]}
+WORK=${WORK:-${TMPDIR:-/tmp}/tess-train-bench}
+
+builddir=$1
+shift
+case $builddir in
+ /*) ;;
+ *) builddir=$REPO/$builddir ;;
+esac
+NAME=${NAME:-$(basename "$builddir")}
+if [ $# -eq 0 ]; then
+ set -- env
+fi
+
+lstmf=$REPO/test/testdata/eng.Arial.exp0.lstmf
+for f in "$builddir/bin/lstmtraining" "$builddir/bin/combine_lang_model" "$lstmf" \
+ "$REPO/test/testdata/eng/eng.unicharset" "$REPO/../langdata_lstm/eng"; do
+ if [ ! -e "$f" ]; then
+ echo "Missing $f" >&2
+ exit 1
+ fi
+done
+
+mkdir -p "$WORK/out" "$WORK/lang"
+if [ ! -f "$WORK/lang/eng/eng.traineddata" ]; then
+ "$builddir/bin/combine_lang_model" \
+ --input_unicharset "$REPO/test/testdata/eng/eng.unicharset" \
+ --script_dir "$REPO/../langdata_lstm" --output_dir "$WORK/lang" --lang eng \
+ > "$WORK/combine_lang_model.log" 2>&1 || {
+ tail -5 "$WORK/combine_lang_model.log" >&2
+ exit 1
+ }
+fi
+echo "$lstmf" > "$WORK/train.list"
+
+log=$WORK/$NAME.log
+rm -f "$WORK/out/$NAME"_* "$WORK/out/$NAME".*
+LC_ALL=C /usr/bin/time -p "$@" "$builddir/bin/lstmtraining" \
+ --traineddata "$WORK/lang/eng/eng.traineddata" --net_spec "$NET_SPEC" \
+ --model_output "$WORK/out/$NAME" --train_listfile "$WORK/train.list" \
+ --max_iterations "$ITER" > "$log" 2>&1 || {
+ tail -5 "$log" >&2
+ exit 1
+}
+
+times=$(grep -E '^(real|user|sys) ' "$log" | tr '\n' ' ')
+last=$(grep 'At iteration' "$log" | tail -1 | sed -e 's/^At iteration //' | cut -c1-76)
+printf '%-18s %s| %s\n' "$NAME:" "$times" "$last"
diff --git a/benchmark/lstm/neon-int-kernel-variants.cpp b/benchmark/lstm/neon-int-kernel-variants.cpp
new file mode 100644
index 00000000..bb28fe90
--- /dev/null
+++ b/benchmark/lstm/neon-int-kernel-variants.cpp
@@ -0,0 +1,209 @@
+///////////////////////////////////////////////////////////////////////
+// File: neon-int-kernel-variants.cpp
+// Description: Microbenchmark for variants of the NEON function which
+// multiplies an int8_t matrix with a vector.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+// http://www.apache.org/licenses/LICENSE-2.0
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+///////////////////////////////////////////////////////////////////////
+
+// This standalone program (ARM64 only) compares the core loop of
+// PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp, which is used
+// for the recognition with integer (fast) models, with two variants:
+// one for any ARM64 CPU and one with the dot product instruction SDOT,
+// which needs ARMv8.2 with the feature "asimddp" (see /proc/cpuinfo).
+// The scaling of the results and the bias are not included. All variants
+// give the same results, because the calculation with integers is exact.
+//
+// c++ -std=c++17 -O3 -o neon-int-kernel-variants neon-int-kernel-variants.cpp
+// ./neon-int-kernel-variants
+
+#include <arm_neon.h>
+#include <chrono>
+#include <cstdint>
+#include <cstdio>
+#include <vector>
+#define NOINLINE __attribute__((noinline))
+
+// Core loop of PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp
+// (8 outputs, weights interleaved in groups of 8 rows x 8 inputs).
+NOINLINE void Now8(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
+ int32x4_t result0123 = {0, 0, 0, 0};
+ int32x4_t result4567 = {0, 0, 0, 0};
+ for (int j = 0; j < num_in; j += 8) {
+ int8x8_t vu = vld1_s8(u);
+ int8x16_t vw01 = vld1q_s8(wi);
+ int8x16_t vw23 = vld1q_s8(wi + 8 * 2);
+ int8x16_t vw45 = vld1q_s8(wi + 8 * 4);
+ int8x16_t vw67 = vld1q_s8(wi + 8 * 6);
+ int32x4_t r0 = vpaddlq_s16(vmull_s8(vget_low_s8(vw01), vu));
+ int32x4_t r1 = vpaddlq_s16(vmull_s8(vget_high_s8(vw01), vu));
+ int32x4_t r2 = vpaddlq_s16(vmull_s8(vget_low_s8(vw23), vu));
+ int32x4_t r3 = vpaddlq_s16(vmull_s8(vget_high_s8(vw23), vu));
+ int32x4_t r4 = vpaddlq_s16(vmull_s8(vget_low_s8(vw45), vu));
+ int32x4_t r5 = vpaddlq_s16(vmull_s8(vget_high_s8(vw45), vu));
+ int32x4_t r6 = vpaddlq_s16(vmull_s8(vget_low_s8(vw67), vu));
+ int32x4_t r7 = vpaddlq_s16(vmull_s8(vget_high_s8(vw67), vu));
+ r0 = vcombine_s32(vpadd_s32(vget_low_s32(r0), vget_high_s32(r0)),
+ vpadd_s32(vget_low_s32(r1), vget_high_s32(r1)));
+ r2 = vcombine_s32(vpadd_s32(vget_low_s32(r2), vget_high_s32(r2)),
+ vpadd_s32(vget_low_s32(r3), vget_high_s32(r3)));
+ r4 = vcombine_s32(vpadd_s32(vget_low_s32(r4), vget_high_s32(r4)),
+ vpadd_s32(vget_low_s32(r5), vget_high_s32(r5)));
+ r6 = vcombine_s32(vpadd_s32(vget_low_s32(r6), vget_high_s32(r6)),
+ vpadd_s32(vget_low_s32(r7), vget_high_s32(r7)));
+ r0 = vcombine_s32(vpadd_s32(vget_low_s32(r0), vget_high_s32(r0)),
+ vpadd_s32(vget_low_s32(r2), vget_high_s32(r2)));
+ r4 = vcombine_s32(vpadd_s32(vget_low_s32(r4), vget_high_s32(r4)),
+ vpadd_s32(vget_low_s32(r6), vget_high_s32(r6)));
+ result0123 = vaddq_s32(result0123, r0);
+ result4567 = vaddq_s32(result4567, r4);
+ u += 8;
+ wi += 64;
+ }
+ vst1q_s32(v, result0123);
+ vst1q_s32(v + 4, result4567);
+}
+NOINLINE void NowMatrix(const int8_t *wi, int rows, int num_in, const int8_t *u,
+ int32_t *v) {
+ for (int i = 0; i < rows; i += 8)
+ Now8(wi + static_cast<size_t>(i / 8) * num_in * 8, u, num_in, v + i);
+}
+
+// Without SDOT: widening multiply and pairwise accumulate, 4 rows per pass.
+NOINLINE void Padal4(const int8_t *w, int rows, int num_in, const int8_t *u,
+ int32_t *v) {
+ for (int i = 0; i < rows; i += 4) {
+ const int8_t *w0 = w + static_cast<size_t>(i) * num_in, *w1 = w0 + num_in,
+ *w2 = w1 + num_in, *w3 = w2 + num_in;
+ int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
+ for (int j = 0; j < num_in; j += 16) {
+ int8x16_t x = vld1q_s8(u + j);
+ int8x16_t y;
+ y = vld1q_s8(w0 + j);
+ a0 = vpadalq_s16(
+ a0, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+ y = vld1q_s8(w1 + j);
+ a1 = vpadalq_s16(
+ a1, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+ y = vld1q_s8(w2 + j);
+ a2 = vpadalq_s16(
+ a2, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+ y = vld1q_s8(w3 + j);
+ a3 = vpadalq_s16(
+ a3, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
+ }
+ v[i] = vaddvq_s32(a0);
+ v[i + 1] = vaddvq_s32(a1);
+ v[i + 2] = vaddvq_s32(a2);
+ v[i + 3] = vaddvq_s32(a3);
+ }
+}
+
+// With SDOT (ARMv8.2 dot product), 4 rows per pass, 16 inputs per instruction.
+__attribute__((target("arch=armv8.2-a+dotprod"))) NOINLINE void Sdot4(
+ const int8_t *w, int rows, int num_in, const int8_t *u, int32_t *v) {
+ for (int i = 0; i < rows; i += 4) {
+ const int8_t *w0 = w + static_cast<size_t>(i) * num_in, *w1 = w0 + num_in,
+ *w2 = w1 + num_in, *w3 = w2 + num_in;
+ int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
+ for (int j = 0; j < num_in; j += 16) {
+ int8x16_t x = vld1q_s8(u + j);
+ a0 = vdotq_s32(a0, vld1q_s8(w0 + j), x);
+ a1 = vdotq_s32(a1, vld1q_s8(w1 + j), x);
+ a2 = vdotq_s32(a2, vld1q_s8(w2 + j), x);
+ a3 = vdotq_s32(a3, vld1q_s8(w3 + j), x);
+ }
+ v[i] = vaddvq_s32(a0);
+ v[i + 1] = vaddvq_s32(a1);
+ v[i + 2] = vaddvq_s32(a2);
+ v[i + 3] = vaddvq_s32(a3);
+ }
+}
+
+template <class F>
+static double BestNs(F f, int reps) {
+ double best = 1e30;
+ for (int r = 0; r < 9; r++) {
+ auto t0 = std::chrono::steady_clock::now();
+ for (int k = 0; k < reps; k++)
+ f();
+ double ns = std::chrono::duration<double, std::nano>(
+ std::chrono::steady_clock::now() - t0)
+ .count() /
+ reps;
+ if (ns < best)
+ best = ns;
+ }
+ return best;
+}
+
+// Returns deterministic pseudo-random numbers with 24 bits for the test data.
+// A linear congruential generator gives the same data on all platforms.
+static uint32_t Random() {
+ static uint32_t state = 1;
+ state = state * 1664525u + 1013904223u;
+ return state >> 8;
+}
+
+int main() {
+ struct {
+ const char *name;
+ int ns, na;
+ } layers[] = {{"Lfys48", 48, 64},
+ {"Lfx96", 96, 144},
+ {"Lrx96", 96, 192},
+ {"Lfx192", 192, 288},
+ {"Lfx512", 512, 608}};
+ printf(
+ "int8 W*u (4 gate matrices), us per timestep\n"
+ " layer now padal sdot now/sdot\n");
+ for (auto &L : layers) {
+ int rows = L.ns, cols = (L.na + 15) / 16 * 16;
+ std::vector<int8_t> w(4 * rows * cols), wi(4 * rows * cols), u(cols);
+ for (auto &e : w)
+ e = Random() % 255 - 127;
+ for (auto &e : u)
+ e = Random() % 255 - 127;
+ // Interleaved copy for the current code: blocks of 8 rows, groups of 8
+ // inputs.
+ for (int g = 0; g < 4; g++)
+ for (int i = 0; i < rows; i++)
+ for (int j = 0; j < cols; j++)
+ wi[g * rows * cols + (i / 8) * cols * 8 + (j / 8) * 64 + (i % 8) * 8 +
+ j % 8] = w[g * rows * cols + i * cols + j];
+ std::vector<int32_t> v0(4 * rows), v1(4 * rows), v2(4 * rows);
+ double t0 = BestNs(
+ [&] {
+ for (int g = 0; g < 4; g++)
+ NowMatrix(wi.data() + g * rows * cols, rows, cols, u.data(),
+ v0.data() + g * rows);
+ },
+ 5000);
+ double t1 = BestNs(
+ [&] {
+ for (int g = 0; g < 4; g++)
+ Padal4(w.data() + g * rows * cols, rows, cols, u.data(),
+ v1.data() + g * rows);
+ },
+ 5000);
+ double t2 = BestNs(
+ [&] {
+ for (int g = 0; g < 4; g++)
+ Sdot4(w.data() + g * rows * cols, rows, cols, u.data(),
+ v2.data() + g * rows);
+ },
+ 5000);
+ printf(" %-7s %6.2f %6.2f %6.2f %.1fx %s\n", L.name, t0 / 1e3, t1 / 1e3,
+ t2 / 1e3, t0 / t2,
+ (v0 == v1 && v0 == v2) ? "(identical results)" : "(RESULTS DIFFER)");
+ }
+ return 0;
+}
diff --git a/benchmark/lstm/neon-kernel-variants.cpp b/benchmark/lstm/neon-kernel-variants.cpp
new file mode 100644
index 00000000..d251eafa
--- /dev/null
+++ b/benchmark/lstm/neon-kernel-variants.cpp
@@ -0,0 +1,444 @@
+///////////////////////////////////////////////////////////////////////
+// File: neon-kernel-variants.cpp
+// Description: Microbenchmark for variants of the NEON functions which
+// calculate several dot products at once.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+// http://www.apache.org/licenses/LICENSE-2.0
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+///////////////////////////////////////////////////////////////////////
+
+// This standalone program (ARM64 only) compares the time of single dot
+// products, called by a function pointer like in the generic code, with
+// variants which handle several rows per pass. The sizes are those of the
+// layers of the default network of tesstrain. It was used to choose the
+// implementations of RowDotProductsNEON (Rows4x2) and CrossDotProductsNEON
+// (Cross4x4) in src/arch/dotproductneon.cpp and can be used to try other
+// variants on a new CPU or with a new compiler:
+//
+// c++ -std=c++17 -O3 -o neon-kernel-variants neon-kernel-variants.cpp
+// ./neon-kernel-variants
+//
+// The accumulators are named variables. Variants with arrays of vectors were
+// much slower with g++ 13, which did not keep them in registers.
+
+#include <arm_neon.h>
+#include <chrono>
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <vector>
+
+#define NOINLINE __attribute__((noinline))
+
+// Dot product like DotProductNEON for FAST_FLOAT.
+NOINLINE float Dot2(const float *u, const float *v, int n) {
+ float32x4_t r0 = vdupq_n_f32(0.0f), r1 = vdupq_n_f32(0.0f);
+ while (n > 7) {
+ r0 = vfmaq_f32(r0, vld1q_f32(u), vld1q_f32(v));
+ r1 = vfmaq_f32(r1, vld1q_f32(u + 4), vld1q_f32(v + 4));
+ u += 8;
+ v += 8;
+ n -= 8;
+ }
+ float total = vaddvq_f32(r0);
+ total += vaddvq_f32(r1);
+ while (n > 0) {
+ total += *u++ * *v++;
+ n--;
+ }
+ return total;
+}
+typedef float (*DotFn)(const float *, const float *, int);
+volatile DotFn g_dot = Dot2;
+
+// Single dot products for each row, like the generic code.
+NOINLINE void RowsNow(const float *w, int stride, int rows, const float *u,
+ int n, float *v) {
+ DotFn dot = g_dot;
+ for (int i = 0; i < rows; i++)
+ v[i] = dot(w + i * stride, u, n);
+}
+
+// 4 rows per pass, 2 accumulators per row (8 columns per iteration).
+NOINLINE void Rows4x2(const float *w, int stride, int rows, const float *u,
+ int n, float *v) {
+ int i = 0;
+ for (; i + 4 <= rows; i += 4) {
+ const float *w0 = w + static_cast<size_t>(i) * stride, *w1 = w0 + stride,
+ *w2 = w1 + stride, *w3 = w2 + stride;
+ float32x4_t a0 = vdupq_n_f32(0), a1 = a0, a2 = a0, a3 = a0, b0 = a0,
+ b1 = a0, b2 = a0, b3 = a0;
+ int j = 0;
+ for (; j + 8 <= n; j += 8) {
+ float32x4_t x = vld1q_f32(u + j), y = vld1q_f32(u + j + 4);
+ a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+ b0 = vfmaq_f32(b0, vld1q_f32(w0 + j + 4), y);
+ a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+ b1 = vfmaq_f32(b1, vld1q_f32(w1 + j + 4), y);
+ a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+ b2 = vfmaq_f32(b2, vld1q_f32(w2 + j + 4), y);
+ a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+ b3 = vfmaq_f32(b3, vld1q_f32(w3 + j + 4), y);
+ }
+ if (j + 4 <= n) {
+ float32x4_t x = vld1q_f32(u + j);
+ a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+ a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+ a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+ a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+ j += 4;
+ }
+ float t0 = vaddvq_f32(vaddq_f32(a0, b0)),
+ t1 = vaddvq_f32(vaddq_f32(a1, b1));
+ float t2 = vaddvq_f32(vaddq_f32(a2, b2)),
+ t3 = vaddvq_f32(vaddq_f32(a3, b3));
+ for (; j < n; j++) {
+ t0 += w0[j] * u[j];
+ t1 += w1[j] * u[j];
+ t2 += w2[j] * u[j];
+ t3 += w3[j] * u[j];
+ }
+ v[i] = t0;
+ v[i + 1] = t1;
+ v[i + 2] = t2;
+ v[i + 3] = t3;
+ }
+ for (; i < rows; i++)
+ v[i] = Dot2(w + static_cast<size_t>(i) * stride, u, n);
+}
+
+// 4 rows per pass, 4 accumulators per row (16 columns per iteration).
+NOINLINE void Rows4x4(const float *w, int stride, int rows, const float *u,
+ int n, float *v) {
+ int i = 0;
+ for (; i + 4 <= rows; i += 4) {
+ const float *w0 = w + static_cast<size_t>(i) * stride, *w1 = w0 + stride,
+ *w2 = w1 + stride, *w3 = w2 + stride;
+ float32x4_t z = vdupq_n_f32(0);
+ float32x4_t a0 = z, a1 = z, a2 = z, a3 = z, b0 = z, b1 = z, b2 = z, b3 = z;
+ float32x4_t c0 = z, c1 = z, c2 = z, c3 = z, d0 = z, d1 = z, d2 = z, d3 = z;
+ int j = 0;
+ for (; j + 16 <= n; j += 16) {
+ float32x4_t x = vld1q_f32(u + j), y = vld1q_f32(u + j + 4),
+ p = vld1q_f32(u + j + 8), q = vld1q_f32(u + j + 12);
+ a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+ b0 = vfmaq_f32(b0, vld1q_f32(w0 + j + 4), y);
+ c0 = vfmaq_f32(c0, vld1q_f32(w0 + j + 8), p);
+ d0 = vfmaq_f32(d0, vld1q_f32(w0 + j + 12), q);
+ a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+ b1 = vfmaq_f32(b1, vld1q_f32(w1 + j + 4), y);
+ c1 = vfmaq_f32(c1, vld1q_f32(w1 + j + 8), p);
+ d1 = vfmaq_f32(d1, vld1q_f32(w1 + j + 12), q);
+ a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+ b2 = vfmaq_f32(b2, vld1q_f32(w2 + j + 4), y);
+ c2 = vfmaq_f32(c2, vld1q_f32(w2 + j + 8), p);
+ d2 = vfmaq_f32(d2, vld1q_f32(w2 + j + 12), q);
+ a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+ b3 = vfmaq_f32(b3, vld1q_f32(w3 + j + 4), y);
+ c3 = vfmaq_f32(c3, vld1q_f32(w3 + j + 8), p);
+ d3 = vfmaq_f32(d3, vld1q_f32(w3 + j + 12), q);
+ }
+ for (; j + 4 <= n; j += 4) {
+ float32x4_t x = vld1q_f32(u + j);
+ a0 = vfmaq_f32(a0, vld1q_f32(w0 + j), x);
+ a1 = vfmaq_f32(a1, vld1q_f32(w1 + j), x);
+ a2 = vfmaq_f32(a2, vld1q_f32(w2 + j), x);
+ a3 = vfmaq_f32(a3, vld1q_f32(w3 + j), x);
+ }
+ float t0 = vaddvq_f32(vaddq_f32(vaddq_f32(a0, b0), vaddq_f32(c0, d0))),
+ t1 = vaddvq_f32(vaddq_f32(vaddq_f32(a1, b1), vaddq_f32(c1, d1)));
+ float t2 = vaddvq_f32(vaddq_f32(vaddq_f32(a2, b2), vaddq_f32(c2, d2))),
+ t3 = vaddvq_f32(vaddq_f32(vaddq_f32(a3, b3), vaddq_f32(c3, d3)));
+ for (; j < n; j++) {
+ t0 += w0[j] * u[j];
+ t1 += w1[j] * u[j];
+ t2 += w2[j] * u[j];
+ t3 += w3[j] * u[j];
+ }
+ v[i] = t0;
+ v[i + 1] = t1;
+ v[i + 2] = t2;
+ v[i + 3] = t3;
+ }
+ for (; i < rows; i++)
+ v[i] = Dot2(w + static_cast<size_t>(i) * stride, u, n);
+}
+
+// Single dot products for each pair of rows, like the generic code.
+NOINLINE void CrossNow(const float *u, int nu, const float *v, int nv, int n,
+ float *out, int os) {
+ DotFn dot = g_dot;
+ for (int i = 0; i < nu; i++)
+ for (int j = 0; j < nv; j++)
+ out[static_cast<size_t>(i) * os + j] = dot(
+ u + static_cast<size_t>(i) * n, v + static_cast<size_t>(j) * n, n);
+}
+
+// 2 rows of u and 4 rows of v per pass.
+NOINLINE void Cross2x4(const float *u, int nu, const float *v, int nv, int n,
+ float *out, int os) {
+ int i = 0;
+ for (; i + 2 <= nu; i += 2) {
+ const float *u0 = u + static_cast<size_t>(i) * n, *u1 = u0 + n;
+ float *o0 = out + static_cast<size_t>(i) * os, *o1 = o0 + os;
+ int j = 0;
+ for (; j + 4 <= nv; j += 4) {
+ const float *v0 = v + static_cast<size_t>(j) * n, *v1 = v0 + n,
+ *v2 = v1 + n, *v3 = v2 + n;
+ float32x4_t z = vdupq_n_f32(0);
+ float32x4_t a0 = z, a1 = z, a2 = z, a3 = z, b0 = z, b1 = z, b2 = z,
+ b3 = z;
+ int t = 0;
+ for (; t + 4 <= n; t += 4) {
+ float32x4_t x0 = vld1q_f32(u0 + t), x1 = vld1q_f32(u1 + t);
+ float32x4_t y0 = vld1q_f32(v0 + t), y1 = vld1q_f32(v1 + t),
+ y2 = vld1q_f32(v2 + t), y3 = vld1q_f32(v3 + t);
+ a0 = vfmaq_f32(a0, x0, y0);
+ a1 = vfmaq_f32(a1, x0, y1);
+ a2 = vfmaq_f32(a2, x0, y2);
+ a3 = vfmaq_f32(a3, x0, y3);
+ b0 = vfmaq_f32(b0, x1, y0);
+ b1 = vfmaq_f32(b1, x1, y1);
+ b2 = vfmaq_f32(b2, x1, y2);
+ b3 = vfmaq_f32(b3, x1, y3);
+ }
+ float r00 = vaddvq_f32(a0), r01 = vaddvq_f32(a1), r02 = vaddvq_f32(a2),
+ r03 = vaddvq_f32(a3);
+ float r10 = vaddvq_f32(b0), r11 = vaddvq_f32(b1), r12 = vaddvq_f32(b2),
+ r13 = vaddvq_f32(b3);
+ for (; t < n; t++) {
+ r00 += u0[t] * v0[t];
+ r01 += u0[t] * v1[t];
+ r02 += u0[t] * v2[t];
+ r03 += u0[t] * v3[t];
+ r10 += u1[t] * v0[t];
+ r11 += u1[t] * v1[t];
+ r12 += u1[t] * v2[t];
+ r13 += u1[t] * v3[t];
+ }
+ o0[j] = r00;
+ o0[j + 1] = r01;
+ o0[j + 2] = r02;
+ o0[j + 3] = r03;
+ o1[j] = r10;
+ o1[j + 1] = r11;
+ o1[j + 2] = r12;
+ o1[j + 3] = r13;
+ }
+ for (; j < nv; j++) {
+ o0[j] = Dot2(u0, v + static_cast<size_t>(j) * n, n);
+ o1[j] = Dot2(u1, v + static_cast<size_t>(j) * n, n);
+ }
+ }
+ for (; i < nu; i++)
+ Rows4x2(v, n, nv, u + static_cast<size_t>(i) * n, n,
+ out + static_cast<size_t>(i) * os);
+}
+
+// 4 rows of u and 4 rows of v per pass.
+NOINLINE void Cross4x4(const float *u, int nu, const float *v, int nv, int n,
+ float *out, int os) {
+ int i = 0;
+ for (; i + 4 <= nu; i += 4) {
+ const float *u0 = u + static_cast<size_t>(i) * n, *u1 = u0 + n,
+ *u2 = u1 + n, *u3 = u2 + n;
+ float *o0 = out + static_cast<size_t>(i) * os, *o1 = o0 + os, *o2 = o1 + os,
+ *o3 = o2 + os;
+ int j = 0;
+ for (; j + 4 <= nv; j += 4) {
+ const float *v0 = v + static_cast<size_t>(j) * n, *v1 = v0 + n,
+ *v2 = v1 + n, *v3 = v2 + n;
+ float32x4_t z = vdupq_n_f32(0);
+ float32x4_t a0 = z, a1 = z, a2 = z, a3 = z, b0 = z, b1 = z, b2 = z,
+ b3 = z;
+ float32x4_t c0 = z, c1 = z, c2 = z, c3 = z, d0 = z, d1 = z, d2 = z,
+ d3 = z;
+ int t = 0;
+ for (; t + 4 <= n; t += 4) {
+ float32x4_t y0 = vld1q_f32(v0 + t), y1 = vld1q_f32(v1 + t),
+ y2 = vld1q_f32(v2 + t), y3 = vld1q_f32(v3 + t);
+ float32x4_t x = vld1q_f32(u0 + t);
+ a0 = vfmaq_f32(a0, x, y0);
+ a1 = vfmaq_f32(a1, x, y1);
+ a2 = vfmaq_f32(a2, x, y2);
+ a3 = vfmaq_f32(a3, x, y3);
+ x = vld1q_f32(u1 + t);
+ b0 = vfmaq_f32(b0, x, y0);
+ b1 = vfmaq_f32(b1, x, y1);
+ b2 = vfmaq_f32(b2, x, y2);
+ b3 = vfmaq_f32(b3, x, y3);
+ x = vld1q_f32(u2 + t);
+ c0 = vfmaq_f32(c0, x, y0);
+ c1 = vfmaq_f32(c1, x, y1);
+ c2 = vfmaq_f32(c2, x, y2);
+ c3 = vfmaq_f32(c3, x, y3);
+ x = vld1q_f32(u3 + t);
+ d0 = vfmaq_f32(d0, x, y0);
+ d1 = vfmaq_f32(d1, x, y1);
+ d2 = vfmaq_f32(d2, x, y2);
+ d3 = vfmaq_f32(d3, x, y3);
+ }
+ float r[4][4] = {
+ {vaddvq_f32(a0), vaddvq_f32(a1), vaddvq_f32(a2), vaddvq_f32(a3)},
+ {vaddvq_f32(b0), vaddvq_f32(b1), vaddvq_f32(b2), vaddvq_f32(b3)},
+ {vaddvq_f32(c0), vaddvq_f32(c1), vaddvq_f32(c2), vaddvq_f32(c3)},
+ {vaddvq_f32(d0), vaddvq_f32(d1), vaddvq_f32(d2), vaddvq_f32(d3)}};
+ for (; t < n; t++) {
+ const float x[4] = {u0[t], u1[t], u2[t], u3[t]},
+ y[4] = {v0[t], v1[t], v2[t], v3[t]};
+ for (int a = 0; a < 4; a++)
+ for (int b = 0; b < 4; b++)
+ r[a][b] += x[a] * y[b];
+ }
+ for (int b = 0; b < 4; b++) {
+ o0[j + b] = r[0][b];
+ o1[j + b] = r[1][b];
+ o2[j + b] = r[2][b];
+ o3[j + b] = r[3][b];
+ }
+ }
+ for (; j < nv; j++) {
+ const float *vj = v + static_cast<size_t>(j) * n;
+ o0[j] = Dot2(u0, vj, n);
+ o1[j] = Dot2(u1, vj, n);
+ o2[j] = Dot2(u2, vj, n);
+ o3[j] = Dot2(u3, vj, n);
+ }
+ }
+ for (; i < nu; i++)
+ Rows4x2(v, n, nv, u + static_cast<size_t>(i) * n, n,
+ out + static_cast<size_t>(i) * os);
+}
+
+// Returns the best time per call in ns of several measurements.
+template <class F>
+static double BestNs(F f, int reps) {
+ double best = 1e30;
+ for (int r = 0; r < 9; r++) {
+ auto t0 = std::chrono::steady_clock::now();
+ for (int k = 0; k < reps; k++)
+ f();
+ double ns = std::chrono::duration<double, std::nano>(
+ std::chrono::steady_clock::now() - t0)
+ .count() /
+ reps;
+ if (ns < best)
+ best = ns;
+ }
+ return best;
+}
+// Returns the largest difference of two results.
+static double MaxDiff(const std::vector<float> &a,
+ const std::vector<float> &b) {
+ double m = 0;
+ for (size_t i = 0; i < a.size(); i++)
+ m = std::fmax(m, std::fabs(a[i] - b[i]));
+ return m;
+}
+
+// Returns deterministic pseudo-random numbers with 24 bits for the test data.
+// A linear congruential generator gives the same data on all platforms.
+static uint32_t Random() {
+ static uint32_t state = 1;
+ state = state * 1664525u + 1013904223u;
+ return state >> 8;
+}
+
+int main() {
+ auto fill = [](std::vector<float> &x) {
+ for (auto &e : x)
+ e = Random() / (float)0x1000000 - 0.5f;
+ };
+ struct {
+ const char *name;
+ int ns, na;
+ } layers[] = {{"Conv16", 16, 9}, {"Lfys48", 48, 64}, {"Lfx96", 96, 144},
+ {"Lrx96", 96, 192}, {"Lfx192", 192, 288}, {"Out111", 111, 192},
+ {"odd", 113, 195}};
+ typedef void (*RowFn)(const float *, int, int, const float *, int, float *);
+ RowFn rf[] = {RowsNow, Rows4x2, Rows4x4};
+ printf(
+ "W*u forward (4 gate matrices in turn), us per timestep\n"
+ " layer single 4x2 4x4\n");
+ for (auto &L : layers) {
+ int stride = L.na + 1;
+ std::vector<float> w(4 * L.ns * stride), u(L.na), ref(4 * L.ns),
+ v(4 * L.ns);
+ fill(w);
+ fill(u);
+ printf(" %-7s", L.name);
+ for (int k = 0; k < 3; k++) {
+ auto &dst = k == 0 ? ref : v;
+ double t = BestNs(
+ [&] {
+ for (int g = 0; g < 4; g++)
+ rf[k](w.data() + g * L.ns * stride, stride, L.ns, u.data(), L.na,
+ dst.data() + g * L.ns);
+ },
+ 5000);
+ printf(" %6.2f", t / 1e3);
+ if (k && MaxDiff(ref, v) > 1e-3)
+ printf("(DIFF %g)", MaxDiff(ref, v));
+ }
+ printf("\n");
+ }
+ printf(
+ "u*W backward (4 transposed gate matrices), us per timestep\n"
+ " layer single 4x2 4x4\n");
+ for (auto &L : layers) {
+ int stride = L.ns;
+ std::vector<float> w(4 * (L.na + 1) * stride), u(L.ns), ref(4 * L.na),
+ v(4 * L.na);
+ fill(w);
+ fill(u);
+ printf(" %-7s", L.name);
+ for (int k = 0; k < 3; k++) {
+ auto &dst = k == 0 ? ref : v;
+ double t = BestNs(
+ [&] {
+ for (int g = 0; g < 4; g++)
+ rf[k](w.data() + g * (L.na + 1) * stride, stride, L.na, u.data(),
+ L.ns, dst.data() + g * L.na);
+ },
+ 5000);
+ printf(" %6.2f", t / 1e3);
+ if (k && MaxDiff(ref, v) > 1e-3)
+ printf("(DIFF %g)", MaxDiff(ref, v));
+ }
+ printf("\n");
+ }
+ typedef void (*CrossFn)(const float *, int, const float *, int, int, float *,
+ int);
+ CrossFn cf[] = {CrossNow, Cross2x4, Cross4x4};
+ printf(
+ "gradient sum (one gate) for T timesteps, ms per call\n"
+ " layer T single 2x4 4x4\n");
+ for (int T : {37, 100, 300, 1000}) {
+ for (auto &L : layers) {
+ int os = L.na + 1;
+ std::vector<float> u(L.ns * T), vv(L.na * T), ref(L.ns * os),
+ d(L.ns * os);
+ fill(u);
+ fill(vv);
+ printf(" %-7s %4d", L.name, T);
+ for (int k = 0; k < 3; k++) {
+ auto &dst = k == 0 ? ref : d;
+ double t = BestNs(
+ [&] { cf[k](u.data(), L.ns, vv.data(), L.na, T, dst.data(), os); },
+ 40);
+ printf(" %6.3f", t / 1e6);
+ if (k && MaxDiff(ref, d) > 2e-3)
+ printf("(DIFF %g)", MaxDiff(ref, d));
+ }
+ printf("\n");
+ }
+ }
+ return 0;
+}
diff --git a/benchmark/lstm/sample-summary.py b/benchmark/lstm/sample-summary.py
new file mode 100755
index 00000000..6a4f429f
--- /dev/null
+++ b/benchmark/lstm/sample-summary.py
@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+
+# Summarize a call graph which was written by the macOS tool "sample".
+#
+# Usage:
+# sample PID SECONDS 1 -file sample.txt
+# sample-summary.py sample.txt [NUMBER_OF_FUNCTIONS]
+#
+# Prints the functions of the main thread with the largest inclusive sample
+# counts (a function and everything which it calls), and the callers of the
+# dot product functions. "sample" splits the counts of a function by call
+# site, so its own summary does not show the totals.
+
+import collections
+import re
+import sys
+
+# Functions for which the direct callers are listed.
+LEAF_FUNCTIONS = ('DotProductNEON', 'RowDotProductsNEON',
+ 'CrossDotProductsNEON')
+
+
+def main():
+ if len(sys.argv) < 2:
+ sys.exit(f'Usage: {sys.argv[0]} sample.txt [NUMBER_OF_FUNCTIONS]')
+ limit = int(sys.argv[2]) if len(sys.argv) > 2 else 40
+ with open(sys.argv[1]) as f:
+ lines = f.read().split('\n')
+ start = next(i for i, l in enumerate(lines) if l.startswith('Call graph:'))
+ end = next(i for i, l in enumerate(lines)
+ if l.startswith('Total number in stack'))
+ # Each line is the indentation (which gives the depth), the count and
+ # the name of the function.
+ pattern = re.compile(r'^([ +!:|]*?)(\d+) (.*)$')
+ inclusive = collections.Counter()
+ callers = collections.Counter()
+ stack = [] # (depth, name) of the callers of the current line
+ total = 0
+ in_main_thread = False
+ for line in lines[start + 1:end]:
+ match = pattern.match(line)
+ if not match:
+ continue
+ depth = len(match.group(1))
+ count = int(match.group(2))
+ name = re.sub(r' \(in .*$', '', match.group(3))
+ name = re.sub(r'\(.*$', '', name).replace('tesseract::', '')
+ if name.startswith('Thread_'):
+ in_main_thread = 'main-thread' in line
+ if in_main_thread:
+ total = count
+ stack = []
+ continue
+ if not in_main_thread:
+ continue
+ while stack and stack[-1][0] >= depth:
+ stack.pop()
+ ancestors = [n for _, n in stack]
+ # Count recursive calls only once.
+ if name not in ancestors:
+ inclusive[name] += count
+ if name in LEAF_FUNCTIONS and ancestors:
+ caller = ancestors[-1]
+ caller2 = ancestors[-2] if len(ancestors) > 1 else ''
+ callers[(name, caller, caller2)] += count
+ stack.append((depth, name))
+ if total == 0:
+ sys.exit('No samples for the main thread found')
+ print(f'Samples of the main thread: {total}')
+ for name, count in inclusive.most_common(limit):
+ print(f'{count:7d} {100.0 * count / total:5.1f}% {name}')
+ if callers:
+ print('Callers of the dot product functions:')
+ for (name, caller, caller2), count in callers.most_common(12):
+ print(f'{count:7d} {100.0 * count / total:5.1f}% '
+ f'{name} <- {caller} <- {caller2}')
+
+
+if __name__ == '__main__':
+ main()