As of the crawl at 2026-10-26 21:30 UTC+3, built at 21:49.
The board: master and the release branches.
1053 failed jobs; 328 of them say something about CI:
328 failed jobs in 3 weeks back from the last crawl. An arrow compares a number with the column before it: ↗ or ↘ when it moved by more than a fifth, ≈ when it did not.
| cause | Oct 5-12 | Oct 12-19 | Oct 19-26 |
|---|---|---|---|
| module tests failed | 3 | ↗ 19 | ↗ 51 |
| ctest timeout | 6 | ↗ 23 | ≈ 19 |
| test failed | 15 | ↘ 9 | ↗ 17 |
| VK S3 unreachable | 2 | ↗ 104 | ↘ 13 |
| entrypoint tag missing | 0 | ≈ 0 | ↗ 8 |
| distro mirror | 5 | ↗ 29 | ↘ 4 |
| pecl.php.net unreachable | 0 | ≈ 0 | ↗ 2 |
| github.com unreachable | 0 | ↗ 1 | ≈ 1 |
| branch not found | 0 | ≈ 0 | ↗ 1 |
| build error | 0 | ↗ 19 | ↘ 0 |
| coveralls.io unreachable | 9 | ↘ 0 | ≈ 0 |
| unexplained | 1 | ↗ 3 | ↘ 0 |
| git transfer dropped | 0 | ↗ 2 | ↘ 0 |
| killed | 1 | ≈ 1 | ↘ 0 |
| python traceback | 0 | ↗ 2 | ↘ 0 |
| name resolution failed | 0 | ↗ 1 | ↘ 0 |
| cause | Oct 5-12 | Oct 12-19 | Oct 19-26 |
|---|---|---|---|
| module tests failed | 0.6 | ↗ 2.3 | ↗ 7.3 |
| ctest timeout | 1.1 | ↗ 2.8 | ≈ 2.7 |
| test failed | 2.8 | ↘ 1.1 | ↗ 2.4 |
| VK S3 unreachable | 0.4 | ↗ 12.5 | ↘ 1.9 |
| entrypoint tag missing | 0.0 | ≈ 0.0 | ↗ 1.1 |
| distro mirror | 0.9 | ↗ 3.5 | ↘ 0.6 |
| pecl.php.net unreachable | 0.0 | ≈ 0.0 | ↗ 0.3 |
| github.com unreachable | 0.0 | ↗ 0.1 | ≈ 0.1 |
| branch not found | 0.0 | ≈ 0.0 | ↗ 0.1 |
| build error | 0.0 | ↗ 2.3 | ↘ 0.0 |
| coveralls.io unreachable | 1.7 | ↘ 0.0 | ≈ 0.0 |
| unexplained | 0.2 | ↗ 0.4 | ↘ 0.0 |
| git transfer dropped | 0.0 | ↗ 0.2 | ↘ 0.0 |
| killed | 0.2 | ↘ 0.1 | ↘ 0.0 |
| python traceback | 0.0 | ↗ 0.2 | ↘ 0.0 |
| name resolution failed | 0.0 | ↗ 0.1 | ↘ 0.0 |
| span | failed | of jobs run | per 1000 | |
|---|---|---|---|---|
| Oct 5-12 | (10-05 21:30:54, 10-12 21:30:54] | 39 | 5451 | 7.2 |
| Oct 12-19 | (10-12 21:30:54, 10-19 21:30:54] | 174 | 8296 | 21.0 |
| Oct 19-26 | (10-19 21:30:54, 10-26 21:30:54] | 115 | 7014 | 16.4 |
115 of them in the last 7 days.
| cause | Mon 19 | Tue 20 | Wed 21 | Thu 22 | Fri 23 | Sat 24 | Sun 25 |
|---|---|---|---|---|---|---|---|
| ctest timeout | 6 | ↘ 2 | ↗ 6 | ↘ 3 | ↘ 0 | ≈ 0 | ↗ 2 |
| module tests failed | 3 | ↗ 26 | ↘ 20 | ↘ 2 | ↘ 0 | ≈ 0 | ≈ 0 |
| test failed | 2 | ↗ 5 | ↘ 2 | ↗ 8 | ↘ 0 | ≈ 0 | ≈ 0 |
| VK S3 unreachable | 1 | ↗ 9 | ↘ 2 | ↘ 1 | ↘ 0 | ≈ 0 | ≈ 0 |
| entrypoint tag missing | 0 | ≈ 0 | ↗ 8 | ↘ 0 | ≈ 0 | ≈ 0 | ≈ 0 |
| distro mirror | 1 | ↗ 3 | ↘ 0 | ≈ 0 | ≈ 0 | ≈ 0 | ≈ 0 |
| pecl.php.net unreachable | 0 | ↗ 2 | ↘ 0 | ≈ 0 | ≈ 0 | ≈ 0 | ≈ 0 |
| branch not found | 0 | ↗ 1 | ↘ 0 | ≈ 0 | ≈ 0 | ≈ 0 | ≈ 0 |
| github.com unreachable | 0 | ↗ 1 | ↘ 0 | ≈ 0 | ≈ 0 | ≈ 0 | ≈ 0 |
| cause | Mon 19 | Tue 20 | Wed 21 | Thu 22 | Fri 23 | Sat 24 | Sun 25 |
|---|---|---|---|---|---|---|---|
| ctest timeout | 2.6 | ↘ 1.3 | ↗ 5.9 | ↘ 2.3 | ↘ 0.0 | ≈ 0.0 | ↗ 3.2 |
| module tests failed | 1.3 | ↗ 16.6 | ≈ 19.6 | ↘ 1.6 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 |
| test failed | 0.9 | ↗ 3.2 | ↘ 2.0 | ↗ 6.2 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 |
| VK S3 unreachable | 0.4 | ↗ 5.7 | ↘ 2.0 | ↘ 0.8 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 |
| entrypoint tag missing | 0.0 | ≈ 0.0 | ↗ 7.8 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 |
| distro mirror | 0.4 | ↗ 1.9 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 |
| pecl.php.net unreachable | 0.0 | ↗ 1.3 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 |
| branch not found | 0.0 | ↗ 0.6 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 |
| github.com unreachable | 0.0 | ↗ 0.6 | ↘ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 | ≈ 0.0 |
| span | failed | of jobs run | per 1000 | |
|---|---|---|---|---|
| Mon 19 | (10-19 21:30:54, 10-20 21:30:54] | 13 | 2325 | 5.6 |
| Tue 20 | (10-20 21:30:54, 10-21 21:30:54] | 48 | 1568 | 30.6 |
| Wed 21 | (10-21 21:30:54, 10-22 21:30:54] | 38 | 1021 | 37.2 |
| Thu 22 | (10-22 21:30:54, 10-23 21:30:54] | 14 | 1285 | 10.9 |
| Fri 23 | (10-23 21:30:54, 10-24 21:30:54] | 0 | 150 | 0.0 |
| Sat 24 | (10-24 21:30:54, 10-25 21:30:54] | 0 | 32 | 0.0 |
| Sun 25 | (10-25 21:30:54, 10-26 21:30:54] | 2 | 633 | 3.2 |
A failed job's log is matched against the signatures below. Each one that matches is a cause, so a job can count under two; a job that matches none is unexplained.
Tests failing in the integration workflow, the modules' own tests run against tarantool, count as module tests failed, apart from tarantool's own tests (the last entry below).
# One cause, several faces: the disk fills, apt's lists are truncated and
# apt reports a signature it cannot verify; rpm and dnf ask for more space;
# yum finds its download directory full.
('runner disk', re.compile(
r'(At least one invalid signature was encountered|No space left on device'
r'|At least \d+ ?[KMG]i?B more space needed on the \S+ filesystem'
r'|needs \d+ ?[KMG]i?B more space on the \S+ filesystem'
r'|Insufficient space in download directory)', re.I)),# A test that failed at least once, which a retry may have undone, or the
# final verdict of test-run or ctest, which nothing undoes. A module's
# integration job runs the module's own, older test-run: no worker
# number before a result, and '===== N tests failed:' as the verdict --
# 5 vshard failures read as unexplained until 2026-09-24; the line is in
# none of 3,237 passing logs. Other modules run other runners: crud's
# luatest closes with 'Failed tests:', php-queue's PHPUnit with
# 'FAILURES!' -- 50 failures read as unexplained until 2026-10-26, 42 of
# them php-queue's stats tests against the new stats.extra of
# tarantool/queue@f789e07c87e7 (10-20 to 10-22, fixed by
# tarantool-php/queue#27); neither line is in 300 passing logs. In the
# integration workflow all of these are the modules' tests (classify).
('test failed', re.compile(
r'^\[\d+\]\s+\S+.*\[ fail \]'
r'|^Failed tasks:'
r'|^Failed tests:$'
r'|^(?:FAILURES|ERRORS)!$'
r'|^===== \d+ tests? failed:'
r'|^\s+\d+ - .+ \((?:Failed|SEGFAULT|Subprocess aborted|Child aborted'
r'|OTHER_FAULT|ILLEGAL|INTERRUPT|NUMERICAL|BAD_COMMAND|Exception'
r'|Not Run)\)(?:\s.*)?$', re.M)),# GitHub's artifact service as well, which the packaging tests download
# their test libraries from: one job of 2026-10-21, ECONNRESET.
('github.com unreachable', re.compile(
r'(Failed to connect to (?:codeload\.)?github\.com'
r'|Could not resolve host: (?:codeload\.)?github\.com'
r'|Failed to download archive .https://codeload\.github\.com'
r'|Unable to download artifact\(s\): Failed to ListArtifacts: Unable to'
r' make request)', re.I)),# A git transfer that started and then broke off. Kept apart from
# github.com unreachable until it is clear they are the same path failing:
# here the connection was made. The TLS lines are git built with GnuTLS
# (the packpack images) or OpenSSL reporting a reset or a mangled record
# mid-stream: 12 jobs read as unexplained until 2026-09-28, 11 on vds*.
# curl's own words for the same, "Failure when receiving data from the
# peer" and "Empty reply from server": 2 jobs of 2026-10-14, until
# 2026-10-26.
('git transfer dropped', re.compile(
r'RPC failed; curl \d+|unexpected disconnect while reading sideband packet'
r'|fatal: early EOF|invalid index-pack output'
r"|fatal: unable to access '[^']*': (?:GnuTLS recv error"
r'|OpenSSL SSL_(?:read|connect)|Recv failure'
r'|Failure when receiving data from the peer|Empty reply from server)'
r'|the remote end hung up upon initial contact')),# A submodule commit GitHub no longer has. `git fetch` recurses into
# submodules on demand, so the environment action's `git fetch --tags -f`
# in a workspace without the history -- fetch-depth 1 on a runner back
# from a stop -- asks for every commit a submodule ever pointed at, and
# luajit bd34ea57 (2015) is gone. Once per workspace; see todo.d/0021.
('submodule commit gone', re.compile(
r'upload-pack: not our ref [0-9a-f]{40}')),# ctest's closing summary, not its per-test '***Timeout' line: ctest runs
# with --repeat until-pass:3, so a timeout that passed on a repeat leaves
# that line in a green log (9 of 126 green ctest jobs). A newer ctest
# prints the test's labels after the verdict (3.31 in the FreeBSD VM, 4.x
# on macOS: "(Timeout) app-luatest regression"); 3.17, the
# static build's, does not.
('ctest timeout', re.compile(r'^\s+\d+ - .+ \(Timeout\)(?:\s.*)?$',
re.M)),# yum's "No more mirrors to try" is the CentOS 7 build failing on a
# package. apt's warning that it could not connect counts only with the
# failure it leads to, a package missing from the index that stayed
# behind: alone, it is in 7 jobs of 2026-10-07 that went on and failed
# on a test or on luacheck. One job each, unexplained until 2026-10-26.
('distro mirror', re.compile(
r'(E: Failed to fetch|Cannot (?:download|prepare internal) mirrorlist'
r'|Errors during downloading metadata'
r'|temporary error \(try again later\)'
r'|\[Errno 256\] No more mirrors to try'
r"|W: Failed to fetch \S+ +Connection failed \[IP:[\s\S]*?"
r"E: (?:Package '[^']+' has no installation candidate"
r'|Unable to locate package))', re.I)),('luarocks', re.compile(
r'(Error: Failed downloading.{0,80}(?:luarocks|rocks\.tarantool)'
r'|Could not resolve host: (?:www\.)?luarocks\.org'
r'|Failed to connect to (?:www\.)?luarocks\.org)', re.I)),# coveralls-lcov's connect to coveralls.io (Cloudflare) outlived Ruby's
# 60 s open_timeout. The gem retries a 5xx, not an exception, so one
# dropped connect fails the coverage job. 3 jobs on 10-02 and 10-04,
# unexplained until 2026-10-05; see todo.d/0022.
('coveralls.io unreachable', re.compile(
r'\(Net::OpenTimeout\)(?:\n\s+from [^\n]*)*?\n\s+from \S*/coveralls-lcov-')),VK_S3 = (r'(?:hb\.bizmrg\.com|[\w.-]*hb\.vkcs\.cloud|[\w.-]*hb\.vkcloud-storage\.ru'
r'|95\.163\.53\.117)')
# Everything CI fetches from VK goes through one address, 95.163.53.117:
# download.tarantool.org and rocks.tarantool.org redirect to hb.bizmrg.com,
# and the packpack, distrib and multivac buckets are *.hb.vkcs.cloud and
# *.hb.vkcloud-storage.ru names for it. Its front end dropped and stalled
# connections on 2026-10-14 15:17-16:54 UTC and from 10-15 13:58 UTC; 57
# failed jobs read as unexplained until then. Only what names the address
# counts: apt merely warns about a repository it could not reach, and the
# job then fails on a package or a key from it, which can have other
# causes. See todo.d/0023. curl's "Connection reset by peer in connection
# to <host>" and apt's "Err:" line, whose reason names the address on the
# line after it, count too: one job each, until 2026-10-26.
('VK S3 unreachable', re.compile(
r'(?:Could not connect to|Unable to connect to|Failed to connect to'
r'|connect to|in connection to|ETIMEDOUT'
r'|Timeout was reached for https://)\s*' + VK_S3
+ r'|' + VK_S3 + r'\S*: \[Errno \d+\] Timeout on'
r'|Failed to fetch https://(?:download\.tarantool\.org|' + VK_S3
+ r')\S* +Could not wait for server fd'
r'|Could not wait for server fd[^\n]*\[IP: ' + VK_S3
+ r'|Failed downloading https?://rocks\.tarantool\.org')),('checkpatch style', re.compile(r'has style problems, please review')),('luacheck', re.compile(
r'^Total: (?:[1-9]\d* warnings? / \d+ errors?'
r'|\d+ warnings? / [1-9]\d* errors?) in \d+ files', re.M)),# The compiler, the linker, CMake or ninja said so. Not make's own
# "*** [target] Error": a test, luacheck or packaging target fails through
# make too, and that line made every ctest failure a build error.
('build error', re.compile(
r'CMake Error'
r'|^\S+:\d+(?::\d+)?: (?:fatal )?error: '
r'|error: .*\[-W'
r'|undefined reference to |collect2: error: ld returned'
r'|ld(?:\.lld|\.gold)?: error:|linker command failed'
r'|No rule to make target'
r'|^FAILED: |^ninja: build stopped', re.M)),# abuild rejects the version of a 2.11 entrypoint commit; see
# ~/w/alpine-pkg-version.
('apk version', re.compile(r'ERROR: \S+: \S+ is not a valid version')),('python traceback', re.compile(r'^Traceback \(most recent call last\)', re.M)),('OOM', re.compile(r'Out of memory: Killed|oom-kill', re.I)),# A step that ended on SIGKILL, 128 + 9, with no word of why: the OOM
# killer of the runner's container leaves none in the job's log, and
# neither does anything else that kills. 2 jobs on ghacts-shared runners,
# 10-07 and 10-15, until 2026-10-26; in none of 300 passing logs.
('killed', re.compile(r'##\[error\]Process completed with exit code 137\.')),('docker pull', re.compile(r'(Docker pull failed|toomanyrequests|manifest unknown)', re.I)),# tools/check-entrypoint-tag.sh: once a release branch is cut, master's
# next commit needs the next entrypoint tag, and until it is pushed every
# run of master fails here. 8 jobs on 2026-10-22, after release/3.9 --
# two runs of four attempts each -- unexplained until 2026-10-26.
('entrypoint tag missing', re.compile(
r'Missing tag for commit [0-9a-f]{40} after branching')),# The PHP connectors' images install extensions from PECL, whose REST
# interface gave nothing back within 20-30 s: 2 jobs, 10-20 and 10-21.
('pecl.php.net unreachable', re.compile(
r'No releases available for package "pecl\.php\.net/'
r'|does not have REST info xml available')),# A getaddrinfo failure, whoever asked: rubygems.org from gem install on
# 2026-10-15, unexplained until 2026-10-26.
('name resolution failed', re.compile(
r'Temporary failure in name resolution|Temporary failure resolving')),# actions/checkout asked for a branch the repository does not have:
# tarantool-ee's release/3.9, which came later than tarantool's.
('branch not found', re.compile(
r"A branch or tag with the name '[^']+' could not be found")),# The integration workflow runs each module's own tests against tarantool.
# A test failing there is counted apart from tarantool's own tests, which
# are the ones the flaky ranking names: it can be tarantool breaking the
# module, or the module itself -- php-queue's tests against a new queue
# were 42 failures in October 2026 (decided with the user, 2026-10-26).
MODULES = 'integration'
def classify(text, workflow):
"""The causes a failed job's log shows: every signature it matches, a
test failure of the modules' workflow under a name of its own."""
signs = [name for name, rx in SIGNS if rx.search(text)]
if workflow == MODULES:
signs = ['module tests failed' if s == 'test failed' else s
for s in signs]
return signs or ['unexplained']