Commit fca28346 for tesseract
commit fca2834671d9a7341f4adb613785dac72c6f8261
Author: Stefan Weil <sw@weilnetz.de>
Date: Sat Nov 14 10:25:28 2026 +0100
arch: speed up the integer matrix product with NEON
The NEON code for the product of an int8_t matrix with a vector, which
is used for the recognition with integer (fast) models, multiplied 8
weights of 8 rows with 8 inputs and added the products with a tree of
pairwise additions. That needs many instructions for each product.
Use groups of 16 inputs and add the widened products directly to the
partial sums of each row (vmull_s8 and vpadalq_s16). This is about two
times faster.
Add a second implementation for ARM64 which uses the dot product
instruction SDOT, an optional feature of ARMv8.2, and which is 3.5 to
7 times faster than the old code. It is selected at runtime if the
CPU supports it (HWCAP_ASIMDDP on Linux) or if the code is built for
such a CPU anyway (__ARM_FEATURE_DOTPROD, for example on macOS). The
function is built with a target attribute, so no new compiler options
are needed. Compilers which are too old for that use the first
implementation.
Recognition of 36 pages with tessdata_fast/eng is 1.27 times faster on
an Apple M1 Pro (22.2 s -> 17.4 s) and on a Cortex-X925 (NVIDIA DGX
Spark, g++ 13: 18.7 s -> 14.8 s), and 1.55 times faster on a
Cortex-A725 (43.1 s -> 27.8 s).
The calculation with integers is exact, so the results do not change.
Both implementations are also exact for the extreme values of weights
and inputs. A faster variant which adds two products with 16 bits
overflows for -128 * -128 and is therefore not used.
Add unit tests for the NEON implementations, which were not tested
until now, including a test with the extreme values.
Update the microbenchmark and the results in benchmark/lstm.
Assisted-by: OpenCode / Claude Fable 5.2 (Anthropic)
Signed-off-by: Stefan Weil <sw@weilnetz.de>
diff --git a/benchmark/lstm/README.md b/benchmark/lstm/README.md
index 8dc00964..f881cf5f 100644
--- a/benchmark/lstm/README.md
+++ b/benchmark/lstm/README.md
@@ -12,7 +12,8 @@ measured with them.
- `neon-kernel-variants.cpp` is a microbenchmark for the float matrix products
(variants of `RowDotProductsNEON` and `CrossDotProductsNEON`).
- `neon-int-kernel-variants.cpp` is a microbenchmark for the `int8_t` matrix
- product which is used for the recognition with integer (fast) models.
+ product which is used for the recognition with integer (fast) models
+ (variants of `PartialMatrixDotVector8`).
The two microbenchmarks are standalone programs for ARM64. How to build
them is described at the beginning of each file.
@@ -126,19 +127,43 @@ gates), single dot products compared with the implemented variants:
### Integer matrix product (`neon-int-kernel-variants.cpp`)
-Recognition of three test images with `tessdata_fast/eng` on M1 spends 73 %
+Recognition of three test images with `tessdata_fast/eng` on M1 spent 73 %
of the time in `LSTMRecognizer::RecognizeLine` and 30 % in the integer
-`matrixDotVector` of `src/arch/intsimdmatrixneon.cpp`. The variants are not
-implemented in the library yet. Time for the four gates of a layer per
-timestep:
-
-| Layer | M1 now | M1 padal | M1 sdot | X925 now | X925 padal | X925 sdot |
-| ------ | -----: | -------: | ------: | -------: | ---------: | --------: |
-| Lfys48 | 0.70 us | 0.23 us | 0.15 us | 0.45 us | 0.19 us | 0.09 us |
-| Lfx96 | 2.47 us | 0.89 us | 0.53 us | 2.00 us | 0.81 us | 0.35 us |
-| Lrx96 | 3.32 us | 1.17 us | 0.69 us | 2.68 us | 1.08 us | 0.50 us |
-| Lfx192 | 9.89 us | 3.45 us | 3.11 us | 8.00 us | 3.21 us | 1.75 us |
-| Lfx512 | 55.8 us | 19.6 us | 18.0 us | 45.2 us | 17.9 us | 10.5 us |
-
-"padal" works on any ARM64 CPU, "sdot" needs the dot product instructions of
-ARMv8.2. All variants give identical results.
+`matrixDotVector` of `src/arch/intsimdmatrixneon.cpp` with the old code.
+
+Time for the four gates of a layer per timestep in microseconds. "old" is
+the old code, "mul8" the new code for any ARM CPU with NEON
+(`intSimdMatrixNEON`) and "dot8" the new code for CPUs with the dot product
+instructions of ARMv8.2 (`intSimdMatrixNEONDotProd`). A725 is a Cortex-A725
+core of the same NVIDIA DGX Spark.
+
+| Layer | M1 old | M1 mul8 | M1 dot8 | X925 old | X925 mul8 | X925 dot8 | A725 old | A725 mul8 | A725 dot8 |
+| ------ | -----: | ------: | ------: | -------: | --------: | --------: | -------: | --------: | --------: |
+| Lfys48 | 0.60 | 0.32 | 0.14 | 0.46 | 0.29 | 0.12 | 1.67 | 0.77 | 0.34 |
+| Lfx96 | 2.48 | 1.25 | 0.53 | 2.02 | 1.12 | 0.46 | 7.44 | 2.94 | 1.05 |
+| Lrx96 | 3.33 | 1.61 | 0.66 | 2.69 | 1.43 | 0.55 | 9.92 | 3.77 | 1.41 |
+| Lfx192 | 9.93 | 4.75 | 2.85 | 8.04 | 4.12 | 1.78 | 29.7 | 10.8 | 3.88 |
+| Lfx512 | 56.0 | 25.9 | 15.7 | 45.4 | 22.5 | 10.3 | 167 | 58.1 | 21.6 |
+
+- All variants give identical results, also for the extreme values of
+ weights and inputs.
+- 8 rows per pass are a bit faster than 4 rows.
+- A variant of "mul" which adds two products with 16 bits before they are
+ widened was about 1.5 times faster, but it overflows if weights and inputs
+ are -128, so it is not used.
+
+### Recognition time
+
+Time for 36 pages (12 times `phototest.tif`, `eurotext.tif` and
+`8087_054.3B.tif` from `test/testing`, listed in a text file) with
+`tesseract pages.txt out --tessdata-dir ../MODELS -l eng --oem 1`, before
+and after the new NEON functions for the integer matrix product:
+
+| Models | M1 before | M1 after | X925 before | X925 after | A725 before | A725 after |
+| --------------- | --------: | -------: | ----------: | ---------: | ----------: | ---------: |
+| `tessdata_fast` | 22.2 s | 17.4 s | 18.7 s | 14.8 s | 43.1 s | 27.8 s |
+| `tessdata_best` | 48.7 s | 48.6 s | 36.3 s | 36.2 s | | |
+
+- Recognition with the integer models is about 1.27 times faster on M1 and
+ X925 and 1.55 times faster on A725. The text results are identical.
+- The float models do not use the integer matrix product.
diff --git a/benchmark/lstm/neon-int-kernel-variants.cpp b/benchmark/lstm/neon-int-kernel-variants.cpp
index bb28fe90..303db3fd 100644
--- a/benchmark/lstm/neon-int-kernel-variants.cpp
+++ b/benchmark/lstm/neon-int-kernel-variants.cpp
@@ -14,13 +14,23 @@
// limitations under the License.
///////////////////////////////////////////////////////////////////////
-// This standalone program (ARM64 only) compares the core loop of
-// PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp, which is used
-// for the recognition with integer (fast) models, with two variants:
-// one for any ARM64 CPU and one with the dot product instruction SDOT,
-// which needs ARMv8.2 with the feature "asimddp" (see /proc/cpuinfo).
-// The scaling of the results and the bias are not included. All variants
-// give the same results, because the calculation with integers is exact.
+// This standalone program (ARM64 only) compares variants of the core loop
+// of PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp, which is
+// used for the recognition with integer (fast) models:
+//
+// old 8 rows per pass, groups of 8 inputs, widening multiplication and
+// a tree of pairwise additions (the code until November 2026)
+// mul4 4 rows per pass, groups of 16 inputs, widening multiplication and
+// pairwise accumulation
+// mul8 the same with 8 rows per pass (implemented as intSimdMatrixNEON)
+// dot4 4 rows per pass, groups of 16 inputs, dot product instruction SDOT
+// dot8 the same with 8 rows per pass (implemented as
+// intSimdMatrixNEONDotProd)
+//
+// SDOT needs a CPU with the feature "asimddp" (see /proc/cpuinfo), an
+// optional part of ARMv8.2. The scaling of the results and the bias are not
+// included. The weights and the inputs use all values of int8_t, and the
+// results of all variants are compared.
//
// c++ -std=c++17 -O3 -o neon-int-kernel-variants neon-int-kernel-variants.cpp
// ./neon-int-kernel-variants
@@ -31,10 +41,14 @@
#include <cstdio>
#include <vector>
#define NOINLINE __attribute__((noinline))
+#if defined(__clang__)
+# define TARGET_DOTPROD __attribute__((target("dotprod")))
+#else
+# define TARGET_DOTPROD __attribute__((target("+dotprod")))
+#endif
-// Core loop of PartialMatrixDotVector8 in src/arch/intsimdmatrixneon.cpp
-// (8 outputs, weights interleaved in groups of 8 rows x 8 inputs).
-NOINLINE void Now8(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
+// Old code: 8 rows, groups of 8 inputs.
+NOINLINE void Old8(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
int32x4_t result0123 = {0, 0, 0, 0};
int32x4_t result4567 = {0, 0, 0, 0};
for (int j = 0; j < num_in; j += 8) {
@@ -71,63 +85,97 @@ NOINLINE void Now8(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
vst1q_s32(v, result0123);
vst1q_s32(v + 4, result4567);
}
-NOINLINE void NowMatrix(const int8_t *wi, int rows, int num_in, const int8_t *u,
- int32_t *v) {
- for (int i = 0; i < rows; i += 8)
- Now8(wi + static_cast<size_t>(i / 8) * num_in * 8, u, num_in, v + i);
+
+// Returns the sum of the four elements of a.
+static inline int32_t Sum(int32x4_t a) {
+ int32x2_t s = vadd_s32(vget_low_s32(a), vget_high_s32(a));
+ return vget_lane_s32(s, 0) + vget_lane_s32(s, 1);
}
-// Without SDOT: widening multiply and pairwise accumulate, 4 rows per pass.
-NOINLINE void Padal4(const int8_t *w, int rows, int num_in, const int8_t *u,
- int32_t *v) {
- for (int i = 0; i < rows; i += 4) {
- const int8_t *w0 = w + static_cast<size_t>(i) * num_in, *w1 = w0 + num_in,
- *w2 = w1 + num_in, *w3 = w2 + num_in;
- int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
- for (int j = 0; j < num_in; j += 16) {
- int8x16_t x = vld1q_s8(u + j);
- int8x16_t y;
- y = vld1q_s8(w0 + j);
- a0 = vpadalq_s16(
- a0, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
- y = vld1q_s8(w1 + j);
- a1 = vpadalq_s16(
- a1, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
- y = vld1q_s8(w2 + j);
- a2 = vpadalq_s16(
- a2, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
- y = vld1q_s8(w3 + j);
- a3 = vpadalq_s16(
- a3, vmlal_high_s8(vmull_s8(vget_low_s8(y), vget_low_s8(x)), y, x));
- }
- v[i] = vaddvq_s32(a0);
- v[i + 1] = vaddvq_s32(a1);
- v[i + 2] = vaddvq_s32(a2);
- v[i + 3] = vaddvq_s32(a3);
+// Adds the products of 16 weights of row k and 16 inputs to acc.
+#define MUL(acc, k) \
+ { \
+ int8x16_t w = vld1q_s8(wi + 16 * (k)); \
+ acc = vpadalq_s16(acc, vmull_s8(vget_low_s8(w), x_low)); \
+ acc = vpadalq_s16(acc, vmull_s8(vget_high_s8(w), x_high)); \
}
-}
+#define DOT(acc, k) acc = vdotq_s32(acc, vld1q_s8(wi + 16 * (k)), x);
-// With SDOT (ARMv8.2 dot product), 4 rows per pass, 16 inputs per instruction.
-__attribute__((target("arch=armv8.2-a+dotprod"))) NOINLINE void Sdot4(
- const int8_t *w, int rows, int num_in, const int8_t *u, int32_t *v) {
- for (int i = 0; i < rows; i += 4) {
- const int8_t *w0 = w + static_cast<size_t>(i) * num_in, *w1 = w0 + num_in,
- *w2 = w1 + num_in, *w3 = w2 + num_in;
- int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
- for (int j = 0; j < num_in; j += 16) {
- int8x16_t x = vld1q_s8(u + j);
- a0 = vdotq_s32(a0, vld1q_s8(w0 + j), x);
- a1 = vdotq_s32(a1, vld1q_s8(w1 + j), x);
- a2 = vdotq_s32(a2, vld1q_s8(w2 + j), x);
- a3 = vdotq_s32(a3, vld1q_s8(w3 + j), x);
- }
- v[i] = vaddvq_s32(a0);
- v[i + 1] = vaddvq_s32(a1);
- v[i + 2] = vaddvq_s32(a2);
- v[i + 3] = vaddvq_s32(a3);
+NOINLINE void Mul4(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
+ int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
+ for (int j = 0; j < num_in; j += 16) {
+ int8x16_t x = vld1q_s8(u + j);
+ int8x8_t x_low = vget_low_s8(x), x_high = vget_high_s8(x);
+ MUL(a0, 0) MUL(a1, 1) MUL(a2, 2) MUL(a3, 3) wi += 64;
+ }
+ v[0] = Sum(a0);
+ v[1] = Sum(a1);
+ v[2] = Sum(a2);
+ v[3] = Sum(a3);
+}
+NOINLINE void Mul8(const int8_t *wi, const int8_t *u, int num_in, int32_t *v) {
+ int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0, a4 = a0, a5 = a0,
+ a6 = a0, a7 = a0;
+ for (int j = 0; j < num_in; j += 16) {
+ int8x16_t x = vld1q_s8(u + j);
+ int8x8_t x_low = vget_low_s8(x), x_high = vget_high_s8(x);
+ MUL(a0, 0)
+ MUL(a1, 1) MUL(a2, 2) MUL(a3, 3) MUL(a4, 4) MUL(a5, 5) MUL(a6, 6) MUL(a7, 7)
+ wi += 128;
+ }
+ v[0] = Sum(a0);
+ v[1] = Sum(a1);
+ v[2] = Sum(a2);
+ v[3] = Sum(a3);
+ v[4] = Sum(a4);
+ v[5] = Sum(a5);
+ v[6] = Sum(a6);
+ v[7] = Sum(a7);
+}
+TARGET_DOTPROD NOINLINE void Dot4(const int8_t *wi, const int8_t *u, int num_in,
+ int32_t *v) {
+ int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0;
+ for (int j = 0; j < num_in; j += 16) {
+ int8x16_t x = vld1q_s8(u + j);
+ DOT(a0, 0) DOT(a1, 1) DOT(a2, 2) DOT(a3, 3) wi += 64;
+ }
+ v[0] = Sum(a0);
+ v[1] = Sum(a1);
+ v[2] = Sum(a2);
+ v[3] = Sum(a3);
+}
+TARGET_DOTPROD NOINLINE void Dot8(const int8_t *wi, const int8_t *u, int num_in,
+ int32_t *v) {
+ int32x4_t a0 = vdupq_n_s32(0), a1 = a0, a2 = a0, a3 = a0, a4 = a0, a5 = a0,
+ a6 = a0, a7 = a0;
+ for (int j = 0; j < num_in; j += 16) {
+ int8x16_t x = vld1q_s8(u + j);
+ DOT(a0, 0)
+ DOT(a1, 1) DOT(a2, 2) DOT(a3, 3) DOT(a4, 4) DOT(a5, 5) DOT(a6, 6) DOT(a7, 7)
+ wi += 128;
}
+ v[0] = Sum(a0);
+ v[1] = Sum(a1);
+ v[2] = Sum(a2);
+ v[3] = Sum(a3);
+ v[4] = Sum(a4);
+ v[5] = Sum(a5);
+ v[6] = Sum(a6);
+ v[7] = Sum(a7);
}
+typedef void (*Fn)(const int8_t *, const int8_t *, int, int32_t *);
+// Interleaves w (rows x cols) for sets of n rows and groups of g inputs.
+static std::vector<int8_t> Shape(const std::vector<int8_t> &w, int rows,
+ int cols, int n, int g) {
+ std::vector<int8_t> s(w.size());
+ for (int i = 0; i < rows; i++)
+ for (int j = 0; j < cols; j++)
+ s[(i / n) * cols * n + (j / g) * n * g + (i % n) * g + j % g] =
+ w[i * cols + j];
+ return s;
+}
+// Returns the best time per call in ns of several measurements.
template <class F>
static double BestNs(F f, int reps) {
double best = 1e30;
@@ -144,7 +192,6 @@ static double BestNs(F f, int reps) {
}
return best;
}
-
// Returns deterministic pseudo-random numbers with 24 bits for the test data.
// A linear congruential generator gives the same data on all platforms.
static uint32_t Random() {
@@ -157,53 +204,54 @@ int main() {
struct {
const char *name;
int ns, na;
- } layers[] = {{"Lfys48", 48, 64},
- {"Lfx96", 96, 144},
- {"Lrx96", 96, 192},
- {"Lfx192", 192, 288},
- {"Lfx512", 512, 608}};
- printf(
- "int8 W*u (4 gate matrices), us per timestep\n"
- " layer now padal sdot now/sdot\n");
+ } layers[] = {{"Conv16", 16, 16}, {"Lfys48", 48, 64}, {"Lfx96", 96, 144},
+ {"Lrx96", 96, 192}, {"Lfx192", 192, 288}, {"Lfx512", 512, 608}};
+ struct {
+ const char *name;
+ Fn f;
+ int n, g;
+ } variants[] = {{"old", Old8, 8, 8},
+ {"mul4", Mul4, 4, 16},
+ {"mul8", Mul8, 8, 16},
+ {"dot4", Dot4, 4, 16},
+ {"dot8", Dot8, 8, 16}};
+ printf("int8 W*u (4 gate matrices), us per timestep\n layer ");
+ for (auto &V : variants) {
+ printf(" %6s", V.name);
+ }
+ printf("\n");
for (auto &L : layers) {
int rows = L.ns, cols = (L.na + 15) / 16 * 16;
- std::vector<int8_t> w(4 * rows * cols), wi(4 * rows * cols), u(cols);
+ std::vector<int8_t> w(rows * cols), u(cols);
for (auto &e : w)
- e = Random() % 255 - 127;
+ e = Random() % 256 - 128;
for (auto &e : u)
- e = Random() % 255 - 127;
- // Interleaved copy for the current code: blocks of 8 rows, groups of 8
- // inputs.
- for (int g = 0; g < 4; g++)
- for (int i = 0; i < rows; i++)
- for (int j = 0; j < cols; j++)
- wi[g * rows * cols + (i / 8) * cols * 8 + (j / 8) * 64 + (i % 8) * 8 +
- j % 8] = w[g * rows * cols + i * cols + j];
- std::vector<int32_t> v0(4 * rows), v1(4 * rows), v2(4 * rows);
- double t0 = BestNs(
- [&] {
- for (int g = 0; g < 4; g++)
- NowMatrix(wi.data() + g * rows * cols, rows, cols, u.data(),
- v0.data() + g * rows);
- },
- 5000);
- double t1 = BestNs(
- [&] {
- for (int g = 0; g < 4; g++)
- Padal4(w.data() + g * rows * cols, rows, cols, u.data(),
- v1.data() + g * rows);
- },
- 5000);
- double t2 = BestNs(
- [&] {
- for (int g = 0; g < 4; g++)
- Sdot4(w.data() + g * rows * cols, rows, cols, u.data(),
- v2.data() + g * rows);
- },
- 5000);
- printf(" %-7s %6.2f %6.2f %6.2f %.1fx %s\n", L.name, t0 / 1e3, t1 / 1e3,
- t2 / 1e3, t0 / t2,
- (v0 == v1 && v0 == v2) ? "(identical results)" : "(RESULTS DIFFER)");
+ e = Random() % 256 - 128;
+ std::vector<int32_t> ref;
+ printf(" %-7s", L.name);
+ for (auto &V : variants) {
+ std::vector<int8_t> s = Shape(w, rows, cols, V.n, V.g);
+ std::vector<int8_t> s4;
+ for (int g = 0; g < 4; g++)
+ s4.insert(s4.end(), s.begin(), s.end());
+ std::vector<int32_t> v(4 * rows);
+ double t = BestNs(
+ [&] {
+ for (int g = 0; g < 4; g++)
+ for (int i = 0; i < rows; i += V.n)
+ V.f(s4.data() + (size_t)g * rows * cols +
+ (size_t)(i / V.n) * cols * V.n,
+ u.data(), cols, v.data() + g * rows + i);
+ },
+ 5000);
+ printf(" %6.2f", t / 1e3);
+ if (ref.empty()) {
+ ref = v;
+ } else if (ref != v) {
+ printf("(RESULTS DIFFER)");
+ }
+ }
+ printf("\n");
}
return 0;
}
diff --git a/src/arch/intsimdmatrix.h b/src/arch/intsimdmatrix.h
index af88ab49..def6b025 100644
--- a/src/arch/intsimdmatrix.h
+++ b/src/arch/intsimdmatrix.h
@@ -115,6 +115,9 @@ struct TESS_API IntSimdMatrix {
static const IntSimdMatrix *intSimdMatrix;
// Only available with NEON.
static const IntSimdMatrix intSimdMatrixNEON;
+ // Only available with NEON. Must only be used if the CPU supports the dot
+ // product instructions of ARMv8.2 (see SIMDDetect::IsNEONDotProdAvailable).
+ static const IntSimdMatrix intSimdMatrixNEONDotProd;
// Only available with RVV.
static const IntSimdMatrix intSimdMatrixRVV;
// Only available with AVX2 / AVX / FMA / SSE.
diff --git a/src/arch/intsimdmatrixneon.cpp b/src/arch/intsimdmatrixneon.cpp
index 6dadd887..24d00c0f 100644
--- a/src/arch/intsimdmatrixneon.cpp
+++ b/src/arch/intsimdmatrixneon.cpp
@@ -26,17 +26,37 @@
# include <vector>
# include "arm_neon.h"
+// The dot product instruction SDOT is an optional feature of ARMv8.2.
+// A function with a target attribute can use it even if the rest of the code
+// is built for a CPU without it. Such a function must only be called if the
+// CPU supports the instruction (see SIMDDetect).
+# if defined(__ARM_FEATURE_DOTPROD)
+// The compiler may use the instruction anywhere (for example on macOS).
+# define HAVE_NEON_DOTPROD
+# define TARGET_DOTPROD
+# elif defined(__aarch64__) && defined(__clang__)
+# if __clang_major__ >= 17
+# define HAVE_NEON_DOTPROD
+# define TARGET_DOTPROD __attribute__((target("+dotprod")))
+# endif
+# elif defined(__aarch64__) && defined(__GNUC__)
+# if __GNUC__ >= 10
+# define HAVE_NEON_DOTPROD
+# define TARGET_DOTPROD __attribute__((target("+dotprod")))
+# endif
+# endif
+
namespace tesseract {
-// Number of outputs held in each register. (Actually, we use a
-// pair of 4x32 registers, so 8 x 32 bit ints).
+// Number of outputs which are computed in one pass over the inputs. Each of
+// them uses one register for four partial sums.
constexpr int kNumOutputsPerRegister = 8;
// Maximum number of registers that we will use.
constexpr int kMaxOutputRegisters = 1;
// Number of inputs in the inputs register.
-constexpr int kNumInputsPerRegister = 8;
+constexpr int kNumInputsPerRegister = 16;
// Number of inputs in each weight group.
-constexpr int kNumInputsPerGroup = 8;
+constexpr int kNumInputsPerGroup = 16;
// Function to compute part of a matrix.vector multiplication. The weights
// are in a very specific order (see above) in w, which is multiplied by
@@ -44,128 +64,141 @@ constexpr int kNumInputsPerGroup = 8;
// by the corresponding member of scales.
// The amount of w and scales consumed is fixed and not available to the
// caller.
+using PartialFunction = void (*)(const int8_t *wi, const TFloat *scales,
+ const int8_t *u, int num_in, TFloat *v,
+ int num_out);
+
+// Returns the sums of the four elements of each of the four arguments.
+static inline int32x4_t HorizontalSums(int32x4_t a, int32x4_t b, int32x4_t c,
+ int32x4_t d) {
+ int32x2_t a2 = vpadd_s32(vget_low_s32(a), vget_high_s32(a)); // a0+a1 a2+a3
+ int32x2_t b2 = vpadd_s32(vget_low_s32(b), vget_high_s32(b));
+ int32x2_t c2 = vpadd_s32(vget_low_s32(c), vget_high_s32(c));
+ int32x2_t d2 = vpadd_s32(vget_low_s32(d), vget_high_s32(d));
+ return vcombine_s32(vpadd_s32(a2, b2), vpadd_s32(c2, d2));
+}
+
+// Adds the bias weights (which follow the other weights in wi) to the 8
+// integer results, scales them and writes num_out of them to v.
+static inline void WriteResults(int32x4_t result0123, int32x4_t result4567,
+ const int8_t *wi, const TFloat *scales,
+ TFloat *v, int num_out) {
+ int8x8_t bias_scale = {127, 127, 127, 127, 127, 127, 127, 127};
+ int8x8_t bias = vld1_s8(wi); // vw0 = b0 b1 b2 b3 b4 b5 b6 b7
+ int16x8_t scaled_bias = vmull_s8(bias, bias_scale);
+ result0123 = vaddw_s16(result0123, vget_low_s16(scaled_bias));
+ result4567 = vaddw_s16(result4567, vget_high_s16(scaled_bias));
+ *v++ = vget_lane_s32(vget_low_s32(result0123), 0) * *scales++;
+ if (num_out > 1)
+ *v++ = vget_lane_s32(vget_low_s32(result0123), 1) * *scales++;
+ if (num_out > 2)
+ *v++ = vget_lane_s32(vget_high_s32(result0123), 0) * *scales++;
+ if (num_out > 3)
+ *v++ = vget_lane_s32(vget_high_s32(result0123), 1) * *scales++;
+ if (num_out > 4)
+ *v++ = vget_lane_s32(vget_low_s32(result4567), 0) * *scales++;
+ if (num_out > 5)
+ *v++ = vget_lane_s32(vget_low_s32(result4567), 1) * *scales++;
+ if (num_out > 6)
+ *v++ = vget_lane_s32(vget_high_s32(result4567), 0) * *scales++;
+ if (num_out > 7)
+ *v = vget_lane_s32(vget_high_s32(result4567), 1) * *scales;
+}
+
+// Adds the 16 products of the elements of w and u to the four partial sums
+// in result. The products are widened to 32 bits when they are added, so the
+// result is exact for all values of w and u.
+static inline int32x4_t AddProducts(int32x4_t result, int8x16_t w,
+ int8x8_t u_low, int8x8_t u_high) {
+ result = vpadalq_s16(result, vmull_s8(vget_low_s8(w), u_low));
+ return vpadalq_s16(result, vmull_s8(vget_high_s8(w), u_high));
+}
// Computes part of matrix.vector v = Wu. Computes N=8 results.
// The weights *must* be arranged so that consecutive reads from wi
// provides (num_in/kNumInputsPerGroup groups of (N output dim groups of
// (kNumInputsPerGroup inputs))). After that there must be N consecutive
// bias weights, before continuing with any more weights.
-// u must be padded out with zeros to
-// kNumInputsPerGroup*ceil(num_in/kNumInputsPerGroup) elements.
-static inline void PartialMatrixDotVector8(const int8_t *__restrict wi,
- const TFloat *__restrict scales,
- const int8_t *__restrict u, int num_in,
- TFloat *__restrict v, int num_out) {
+// u must be padded out to
+// kNumInputsPerGroup*ceil(num_in/kNumInputsPerGroup) elements. The weights
+// for the padding are 0, so the values of the padding do not matter.
+static void PartialMatrixDotVector8(const int8_t *__restrict wi,
+ const TFloat *__restrict scales,
+ const int8_t *__restrict u, int num_in,
+ TFloat *__restrict v, int num_out) {
// Initialize all the results to 0.
- int32x4_t result0123 = {0, 0, 0, 0};
- int32x4_t result4567 = {0, 0, 0, 0};
- int8x8_t bias_scale = {127, 127, 127, 127, 127, 127, 127, 127};
+ int32x4_t result0 = vdupq_n_s32(0);
+ int32x4_t result1 = result0;
+ int32x4_t result2 = result0;
+ int32x4_t result3 = result0;
+ int32x4_t result4 = result0;
+ int32x4_t result5 = result0;
+ int32x4_t result6 = result0;
+ int32x4_t result7 = result0;
// Iterate over the input (u), one registerful at a time.
- for (int j = 0; j < num_in; j += 8) {
- int8x8_t vu = vld1_s8(u); // vu = u0 u1 u2 u3 u4 u5 u6 u7
- int8x16_t vw01 = vld1q_s8(wi); // vw0 = w00 w01 w02 w03 w04 w05 w06 w07
- // w10 w11 w12 w13 w14 w15 w16 w17
- int8x16_t vw23 = vld1q_s8(wi + 8 * 2); // vw2 = w20 w21 w22 w23 w24 w25 w26 w27 w30
- // w31 w32 w33 w34 w35 w36 w37
- int8x16_t vw45 = vld1q_s8(wi + 8 * 4); // vw4 = w40 w41 w42 w43 w44 w45 w46 w47 w50
- // w51 w52 w53 w54 w55 w56 w57
- int8x16_t vw67 = vld1q_s8(wi + 8 * 6); // vw6 = w60 w61 w62 w63 w64 w65 w66 w67 w70
- // w71 w72 w73 w74 w75 w76 w77
-
- int16x8_t vrow0q = vmull_s8(vget_low_s8(vw01), vu); // vrow0q = vw00.u0 w01.u1 w02.u2
- // w03.u3 vw04.u4 w05.u5 w06.u6 w07.u7
- int16x8_t vrow1q = vmull_s8(vget_high_s8(vw01),
- vu); // vrow1q = vw10.u0 w11.u1 w12.u2 w13.u3
- // vw14.u4 w15.u5 w16.u6 w17.u7
- int16x8_t vrow2q = vmull_s8(vget_low_s8(vw23), vu); // vrow2q = vw20.u0 w21.u1 w22.u2
- // w23.u3 vw24.u4 w25.u5 w26.u6 w27.u7
- int16x8_t vrow3q = vmull_s8(vget_high_s8(vw23),
- vu); // vrow3q = vw30.u0 w31.u1 w32.u2 w33.u3
- // vw34.u4 w35.u5 w36.u6 w37.u7
- int16x8_t vrow4q = vmull_s8(vget_low_s8(vw45), vu); // vrow4q = vw40.u0 w41.u1 w42.u2
- // w43.u3 vw44.u4 w45.u5 w46.u6 w47.u7
- int16x8_t vrow5q = vmull_s8(vget_high_s8(vw45),
- vu); // vrow5q = vw50.u0 w51.u1 w52.u2 w53.u3
- // vw54.u4 w55.u5 w56.u6 w57.u7
- int16x8_t vrow6q = vmull_s8(vget_low_s8(vw67), vu); // vrow6q = vw60.u0 w61.u1 w62.u2
- // w63.u3 vw64.u4 w65.u5 w66.u6 w67.u7
- int16x8_t vrow7q = vmull_s8(vget_high_s8(vw67),
- vu); // vrow7q = vw70.u0 w71.u1 w72.u2 w73.u3
- // vw74.u4 w75.u5 w76.u6 w77.u7
-
- int32x4_t vrow0q2 = vpaddlq_s16(vrow0q); // vrow0q2 = vw00.u0+w01.u1 w02.u2+w03.u3
- // vw04.u4+w05.u5 w06.u6+w07.u7
- int32x4_t vrow1q2 = vpaddlq_s16(vrow1q); // vrow1q2 = vw10.u0+w11.u1 w12.u2+w13.u3
- // vw14.u4+w15.u5 w16.u6+w17.u7
- int32x4_t vrow2q2 = vpaddlq_s16(vrow2q); // vrow2q2 = vw20.u0+w21.u1 w22.u2+w23.u3
- // vw24.u4+w25.u5 w26.u6+w27.u7
- int32x4_t vrow3q2 = vpaddlq_s16(vrow3q); // vrow3q2 = vw30.u0+w31.u1 w32.u2+w33.u3
- // vw34.u4+w35.u5 w36.u6+w37.u7
- int32x4_t vrow4q2 = vpaddlq_s16(vrow4q); // vrow4q2 = vw40.u0+w41.u1 w42.u2+w43.u3
- // vw44.u4+w45.u5 w46.u6+w47.u7
- int32x4_t vrow5q2 = vpaddlq_s16(vrow5q); // vrow5q2 = vw50.u0+w51.u1 w52.u2+w53.u3
- // vw54.u4+w55.u5 w56.u6+w57.u7
- int32x4_t vrow6q2 = vpaddlq_s16(vrow6q); // vrow6q2 = vw60.u0+w61.u1 w62.u2+w63.u3
- // vw64.u4+w65.u5 w66.u6+w67.u7
- int32x4_t vrow7q2 = vpaddlq_s16(vrow7q); // vrow7q2 = vw70.u0+w71.u1 w72.u2+w73.u3
- // vw74.u4+w75.u5 w76.u6+w77.u7
-
- vrow0q2 = vcombine_s32(vpadd_s32(vget_low_s32(vrow0q2), vget_high_s32(vrow0q2)),
- vpadd_s32(vget_low_s32(vrow1q2), vget_high_s32(vrow1q2)));
- // vrow0q2 = vw00.u0+...+w03.u3 vw04.u4+...+w07.u7 vw10.u0+...+w13.u3
- // vw14.u4+...+w17.u7
- vrow2q2 = vcombine_s32(vpadd_s32(vget_low_s32(vrow2q2), vget_high_s32(vrow2q2)),
- vpadd_s32(vget_low_s32(vrow3q2), vget_high_s32(vrow3q2)));
- // vrow0q2 = vw20.u0+...+w23.u3 vw24.u4+...+w27.u7 vw30.u0+...+w33.u3
- // vw34.u4+...+w37.u7
- vrow4q2 = vcombine_s32(vpadd_s32(vget_low_s32(vrow4q2), vget_high_s32(vrow4q2)),
- vpadd_s32(vget_low_s32(vrow5q2), vget_high_s32(vrow5q2)));
- // vrow0q2 = vw40.u0+...+w43.u3 vw44.u4+...+w47.u7 vw50.u0+...+w53.u3
- // vw54.u4+...+w57.u7
- vrow6q2 = vcombine_s32(vpadd_s32(vget_low_s32(vrow6q2), vget_high_s32(vrow6q2)),
- vpadd_s32(vget_low_s32(vrow7q2), vget_high_s32(vrow7q2)));
- // vrow0q2 = vw60.u0+...+w63.u3 vw64.u4+...+w67.u7 vw70.u0+...+w73.u3
- // vw74.u4+...+w77.u7
-
- vrow0q2 = vcombine_s32(vpadd_s32(vget_low_s32(vrow0q2), vget_high_s32(vrow0q2)),
- vpadd_s32(vget_low_s32(vrow2q2), vget_high_s32(vrow2q2)));
- // vrow0q2 = vw00.u0+...+w07.u7 vw10.u0+...+w17.u7 vw20.u0+...+w27.u7
- // vw30.u0+...+w37.u7
- vrow4q2 = vcombine_s32(vpadd_s32(vget_low_s32(vrow4q2), vget_high_s32(vrow4q2)),
- vpadd_s32(vget_low_s32(vrow6q2), vget_high_s32(vrow6q2)));
- // vrow0q2 = vw40.u0+...+w47.u7 vw50.u0+...+w57.u7 vw60.u0+...+w67.u7
- // vw70.u0+...+w77.u7
-
- result0123 = vaddq_s32(result0123, vrow0q2);
- result4567 = vaddq_s32(result4567, vrow4q2);
- u += 8;
- wi += 64;
+ for (int j = 0; j < num_in; j += kNumInputsPerGroup) {
+ int8x16_t vu = vld1q_s8(u);
+ int8x8_t vu_low = vget_low_s8(vu);
+ int8x8_t vu_high = vget_high_s8(vu);
+ result0 = AddProducts(result0, vld1q_s8(wi), vu_low, vu_high);
+ result1 = AddProducts(result1, vld1q_s8(wi + 16 * 1), vu_low, vu_high);
+ result2 = AddProducts(result2, vld1q_s8(wi + 16 * 2), vu_low, vu_high);
+ result3 = AddProducts(result3, vld1q_s8(wi + 16 * 3), vu_low, vu_high);
+ result4 = AddProducts(result4, vld1q_s8(wi + 16 * 4), vu_low, vu_high);
+ result5 = AddProducts(result5, vld1q_s8(wi + 16 * 5), vu_low, vu_high);
+ result6 = AddProducts(result6, vld1q_s8(wi + 16 * 6), vu_low, vu_high);
+ result7 = AddProducts(result7, vld1q_s8(wi + 16 * 7), vu_low, vu_high);
+ u += kNumInputsPerGroup;
+ wi += kNumInputsPerGroup * kNumOutputsPerRegister;
}
- {
- int8x8_t bias = vld1_s8(wi); // vw0 = b0 b1 b2 b3 b4 b5 b6 b7
- int16x8_t scaled_bias = vmull_s8(bias, bias_scale);
- result0123 = vaddw_s16(result0123, vget_low_s16(scaled_bias));
- result4567 = vaddw_s16(result4567, vget_high_s16(scaled_bias));
- *v++ = vget_lane_s32(vget_low_s32(result0123), 0) * *scales++;
- if (num_out > 1)
- *v++ = vget_lane_s32(vget_low_s32(result0123), 1) * *scales++;
- if (num_out > 2)
- *v++ = vget_lane_s32(vget_high_s32(result0123), 0) * *scales++;
- if (num_out > 3)
- *v++ = vget_lane_s32(vget_high_s32(result0123), 1) * *scales++;
- if (num_out > 4)
- *v++ = vget_lane_s32(vget_low_s32(result4567), 0) * *scales++;
- if (num_out > 5)
- *v++ = vget_lane_s32(vget_low_s32(result4567), 1) * *scales++;
- if (num_out > 6)
- *v++ = vget_lane_s32(vget_high_s32(result4567), 0) * *scales++;
- if (num_out > 7)
- *v = vget_lane_s32(vget_high_s32(result4567), 1) * *scales;
+ WriteResults(HorizontalSums(result0, result1, result2, result3),
+ HorizontalSums(result4, result5, result6, result7), wi, scales,
+ v, num_out);
+}
+
+# if defined(HAVE_NEON_DOTPROD)
+
+// Same as PartialMatrixDotVector8, but uses the dot product instruction SDOT,
+// which multiplies 16 pairs of values and adds the products to four partial
+// sums at once.
+TARGET_DOTPROD static void PartialMatrixDotVector8DotProd(
+ const int8_t *__restrict wi, const TFloat *__restrict scales,
+ const int8_t *__restrict u, int num_in, TFloat *__restrict v, int num_out) {
+ // Initialize all the results to 0.
+ int32x4_t result0 = vdupq_n_s32(0);
+ int32x4_t result1 = result0;
+ int32x4_t result2 = result0;
+ int32x4_t result3 = result0;
+ int32x4_t result4 = result0;
+ int32x4_t result5 = result0;
+ int32x4_t result6 = result0;
+ int32x4_t result7 = result0;
+ // Iterate over the input (u), one registerful at a time.
+ for (int j = 0; j < num_in; j += kNumInputsPerGroup) {
+ int8x16_t vu = vld1q_s8(u);
+ result0 = vdotq_s32(result0, vld1q_s8(wi), vu);
+ result1 = vdotq_s32(result1, vld1q_s8(wi + 16 * 1), vu);
+ result2 = vdotq_s32(result2, vld1q_s8(wi + 16 * 2), vu);
+ result3 = vdotq_s32(result3, vld1q_s8(wi + 16 * 3), vu);
+ result4 = vdotq_s32(result4, vld1q_s8(wi + 16 * 4), vu);
+ result5 = vdotq_s32(result5, vld1q_s8(wi + 16 * 5), vu);
+ result6 = vdotq_s32(result6, vld1q_s8(wi + 16 * 6), vu);
+ result7 = vdotq_s32(result7, vld1q_s8(wi + 16 * 7), vu);
+ u += kNumInputsPerGroup;
+ wi += kNumInputsPerGroup * kNumOutputsPerRegister;
}
+ WriteResults(HorizontalSums(result0, result1, result2, result3),
+ HorizontalSums(result4, result5, result6, result7), wi, scales,
+ v, num_out);
}
-static void matrixDotVector(int dim1, int dim2, const int8_t *wi, const TFloat *scales,
- const int8_t *u, TFloat *v) {
+# endif // HAVE_NEON_DOTPROD
+
+// Computes matrix.vector v = Wu with the given function for the parts.
+static inline void MatrixDotVectorPartial(PartialFunction partial_func,
+ int dim1, int dim2, const int8_t *wi,
+ const TFloat *scales, const int8_t *u,
+ TFloat *v) {
const int num_out = dim1;
const int num_in = dim2 - 1;
// Each call to a partial_func_ produces group_size outputs, except the
@@ -177,14 +210,19 @@ static void matrixDotVector(int dim1, int dim2, const int8_t *wi, const TFloat *
int w_step = (rounded_num_in + 1) * group_size;
for (; output + group_size <= num_out; output += group_size) {
- PartialMatrixDotVector8(wi, scales, u, rounded_num_in, v, kNumOutputsPerRegister);
+ partial_func(wi, scales, u, rounded_num_in, v, kNumOutputsPerRegister);
wi += w_step;
scales += group_size;
v += group_size;
}
if (output < num_out)
- PartialMatrixDotVector8(wi, scales, u, rounded_num_in, v,
- num_out & (kNumOutputsPerRegister - 1));
+ partial_func(wi, scales, u, rounded_num_in, v,
+ num_out & (kNumOutputsPerRegister - 1));
+}
+
+static void matrixDotVector(int dim1, int dim2, const int8_t *wi,
+ const TFloat *scales, const int8_t *u, TFloat *v) {
+ MatrixDotVectorPartial(PartialMatrixDotVector8, dim1, dim2, wi, scales, u, v);
}
const IntSimdMatrix IntSimdMatrix::intSimdMatrixNEON = {
@@ -200,6 +238,34 @@ const IntSimdMatrix IntSimdMatrix::intSimdMatrixNEON = {
kNumInputsPerGroup
};
+# if defined(HAVE_NEON_DOTPROD)
+static void matrixDotVectorDotProd(int dim1, int dim2, const int8_t *wi,
+ const TFloat *scales, const int8_t *u,
+ TFloat *v) {
+ MatrixDotVectorPartial(PartialMatrixDotVector8DotProd, dim1, dim2, wi, scales,
+ u, v);
+}
+# endif
+
+// The weights are in the same order as for intSimdMatrixNEON.
+const IntSimdMatrix IntSimdMatrix::intSimdMatrixNEONDotProd = {
+ // Function.
+# if defined(HAVE_NEON_DOTPROD)
+ matrixDotVectorDotProd,
+# else
+ // The compiler cannot build the code with SDOT.
+ matrixDotVector,
+# endif
+ // Number of 32 bit outputs held in each register.
+ kNumOutputsPerRegister,
+ // Maximum number of registers that we will use to hold outputs.
+ kMaxOutputRegisters,
+ // Number of 8 bit inputs in the inputs register.
+ kNumInputsPerRegister,
+ // Number of inputs in each weight group.
+ kNumInputsPerGroup
+};
+
} // namespace tesseract.
#endif /* __ARM_NEON */
diff --git a/src/arch/simddetect.cpp b/src/arch/simddetect.cpp
index 33c41e4a..5d520035 100644
--- a/src/arch/simddetect.cpp
+++ b/src/arch/simddetect.cpp
@@ -66,6 +66,15 @@
# endif
#endif
+#if defined(__aarch64__) && defined(__linux__) && \
+ !defined(__ARM_FEATURE_DOTPROD)
+// Needed for the detection of the NEON dot product instructions at runtime.
+# include <sys/auxv.h>
+# if !defined(HWCAP_ASIMDDP)
+# define HWCAP_ASIMDDP (1 << 20)
+# endif
+#endif
+
#if defined(HAVE_RVV)
# if defined(HAVE_GETAUXVAL) || defined(HAVE_ELF_AUX_INFO)
# include <sys/auxv.h>
@@ -124,9 +133,13 @@ static const SIMDDetect &detector_init = SIMDDetect::GetDetector();
#if defined(__aarch64__)
// ARMv8 always has NEON.
bool SIMDDetect::neon_available_ = true;
+// If true, then the NEON dot product instructions have been detected.
+bool SIMDDetect::neon_dotprod_available_;
#elif defined(HAVE_NEON)
// If true, then Neon has been detected.
bool SIMDDetect::neon_available_;
+// The NEON dot product instructions are only used for ARMv8.
+bool SIMDDetect::neon_dotprod_available_;
#elif defined(HAVE_RVV)
bool SIMDDetect::rvv_available_;
#else
@@ -304,6 +317,15 @@ SIMDDetect::SIMDDetect() {
# endif
#endif
+#if defined(__aarch64__)
+# if defined(__ARM_FEATURE_DOTPROD)
+ // The code was built for a CPU with the NEON dot product instructions.
+ neon_dotprod_available_ = true;
+# elif defined(__linux__)
+ neon_dotprod_available_ = (getauxval(AT_HWCAP) & HWCAP_ASIMDDP) != 0;
+# endif
+#endif
+
#if defined(HAVE_RVV)
# if defined(HAVE_GETAUXVAL)
const unsigned long hwcap = getauxval(AT_HWCAP);
@@ -343,7 +365,9 @@ SIMDDetect::SIMDDetect() {
#if defined(HAVE_NEON) || defined(__aarch64__)
} else if (neon_available_) {
// NEON detected.
- SetDotProduct(DotProductNEON, &IntSimdMatrix::intSimdMatrixNEON);
+ SetDotProduct(DotProductNEON, neon_dotprod_available_
+ ? &IntSimdMatrix::intSimdMatrixNEONDotProd
+ : &IntSimdMatrix::intSimdMatrixNEON);
#endif
#if defined(HAVE_RVV)
} else if (rvv_available_) {
@@ -400,7 +424,9 @@ void SIMDDetect::Update() {
#if defined(HAVE_NEON) || defined(__aarch64__)
} else if (dotproduct == "neon" && neon_available_) {
// NEON selected by config variable.
- SetDotProduct(DotProductNEON, &IntSimdMatrix::intSimdMatrixNEON);
+ SetDotProduct(DotProductNEON, neon_dotprod_available_
+ ? &IntSimdMatrix::intSimdMatrixNEONDotProd
+ : &IntSimdMatrix::intSimdMatrixNEON);
dotproduct_method = "neon";
#endif
} else if (dotproduct == "std::inner_product") {
diff --git a/src/arch/simddetect.h b/src/arch/simddetect.h
index 8e903607..1ed3cdcc 100644
--- a/src/arch/simddetect.h
+++ b/src/arch/simddetect.h
@@ -81,6 +81,11 @@ public:
static inline bool IsNEONAvailable() {
return GetDetector().neon_available_;
}
+ // Returns true if the NEON dot product instructions (an optional feature
+ // of ARMv8.2) are available on this system.
+ static inline bool IsNEONDotProdAvailable() {
+ return GetDetector().neon_dotprod_available_;
+ }
// Returns true if RVV is available on this system.
static inline bool IsRVVAvailable() {
return GetDetector().rvv_available_;
@@ -111,6 +116,8 @@ private:
static TESS_API bool sse_available_;
// If true, then NEON has been detected.
static TESS_API bool neon_available_;
+ // If true, then the NEON dot product instructions have been detected.
+ static TESS_API bool neon_dotprod_available_;
// If true, then RVV has been detected.
static TESS_API bool rvv_available_;
};
diff --git a/unittest/intsimdmatrix_test.cc b/unittest/intsimdmatrix_test.cc
index c3e18099..b7bc1010 100644
--- a/unittest/intsimdmatrix_test.cc
+++ b/unittest/intsimdmatrix_test.cc
@@ -17,6 +17,8 @@
#include "intsimdmatrix.h"
#include <gtest/gtest.h>
#include <gtest/internal/gtest-port.h>
+#include <algorithm>
+#include <cstdint>
#include <memory>
#include <vector>
#include "include_gunit.h"
@@ -97,6 +99,53 @@ protected:
#endif
}
+ // Tests the extreme values of weights and inputs for some sizes and
+ // compares the results against the generic version. The products of these
+ // values are the largest possible ones, so this detects implementations
+ // which overflow in intermediate results.
+ void ExpectEqualResultsForExtremeValues(const IntSimdMatrix &matrix) {
+ const int8_t kExtremes[] = {INT8_MIN, INT8_MAX};
+ for (int8_t weight : kExtremes) {
+ for (int8_t input : kExtremes) {
+ for (int num_out : {1, 7, 8, 9, 33}) {
+ for (int num_in : {1, 15, 16, 17, 100}) {
+ GENERIC_2D_ARRAY<int8_t> w(num_out, num_in + 1, weight);
+ std::vector<int8_t> u(matrix.RoundInputs(num_in), 0);
+ std::fill(u.begin(), u.begin() + num_in, input);
+ std::vector<TFloat> scales(num_out, 1);
+ int ro = num_out;
+ if (IntSimdMatrix::intSimdMatrix) {
+ ro = IntSimdMatrix::intSimdMatrix->RoundOutputs(ro);
+ }
+ std::vector<TFloat> base_result(num_out);
+ IntSimdMatrix::MatrixDotVector(w, scales, u.data(),
+ base_result.data());
+ std::vector<TFloat> test_result(ro);
+ std::vector<int8_t> shaped_wi;
+ int32_t rounded_num_out;
+ matrix.Init(w, shaped_wi, rounded_num_out);
+ scales.resize(rounded_num_out);
+ if (matrix.matrixDotVectorFunction) {
+ matrix.matrixDotVectorFunction(w.dim1(), w.dim2(), &shaped_wi[0],
+ &scales[0], &u[0],
+ &test_result[0]);
+ } else {
+ IntSimdMatrix::MatrixDotVector(w, scales, u.data(),
+ test_result.data());
+ }
+ for (int i = 0; i < num_out; ++i) {
+ EXPECT_EQ(base_result[i], test_result[i])
+ << "weight=" << static_cast<int>(weight)
+ << ", input=" << static_cast<int>(input)
+ << ", num_out=" << num_out << ", num_in=" << num_in
+ << ", i=" << i;
+ }
+ }
+ }
+ }
+ }
+ }
+
TRand random_;
};
@@ -134,4 +183,51 @@ TEST_F(IntSimdMatrixTest, AVX2) {
#endif
}
+// Tests that the NEON implementation gets the same result as the vanilla.
+TEST_F(IntSimdMatrixTest, NEON) {
+#if defined(__aarch64__)
+ ExpectEqualResults(IntSimdMatrix::intSimdMatrixNEON);
+#else
+ GTEST_LOG_(INFO) << "NEON unsupported! Not tested!";
+ GTEST_SKIP();
+#endif
+}
+
+TEST_F(IntSimdMatrixTest, NEONExtremeValues) {
+#if defined(__aarch64__)
+ ExpectEqualResultsForExtremeValues(IntSimdMatrix::intSimdMatrixNEON);
+#else
+ GTEST_LOG_(INFO) << "NEON unsupported! Not tested!";
+ GTEST_SKIP();
+#endif
+}
+
+// Tests that the NEON implementation with the dot product instructions gets
+// the same result as the vanilla.
+TEST_F(IntSimdMatrixTest, NEONDotProd) {
+#if defined(__aarch64__)
+ if (!SIMDDetect::IsNEONDotProdAvailable()) {
+ GTEST_LOG_(INFO) << "No NEON dot product instructions found! Not tested!";
+ GTEST_SKIP();
+ }
+ ExpectEqualResults(IntSimdMatrix::intSimdMatrixNEONDotProd);
+#else
+ GTEST_LOG_(INFO) << "NEON unsupported! Not tested!";
+ GTEST_SKIP();
+#endif
+}
+
+TEST_F(IntSimdMatrixTest, NEONDotProdExtremeValues) {
+#if defined(__aarch64__)
+ if (!SIMDDetect::IsNEONDotProdAvailable()) {
+ GTEST_LOG_(INFO) << "No NEON dot product instructions found! Not tested!";
+ GTEST_SKIP();
+ }
+ ExpectEqualResultsForExtremeValues(IntSimdMatrix::intSimdMatrixNEONDotProd);
+#else
+ GTEST_LOG_(INFO) << "NEON unsupported! Not tested!";
+ GTEST_SKIP();
+#endif
+}
+
} // namespace tesseract