Why the jobs failed

As of the crawl at 2026-10-26 21:30 UTC+3, built at 21:49.

The board: master and the release branches.

1053 failed jobs; 328 of them say something about CI:

By week

328 failed jobs in 3 weeks back from the last crawl. An arrow compares a number with the column before it: ↗ or ↘ when it moved by more than a fifth, ≈ when it did not.

Failed jobs. How many failed jobs show each cause; a job re-run and failing again counts each time.
causeOct 5-12Oct 12-19Oct 19-26
module tests failed3↗ 19↗ 51
ctest timeout6↗ 23≈ 19
test failed15↘ 9↗ 17
VK S3 unreachable2↗ 104↘ 13
entrypoint tag missing0≈ 0↗ 8
distro mirror5↗ 29↘ 4
pecl.php.net unreachable0≈ 0↗ 2
github.com unreachable0↗ 1≈ 1
branch not found0≈ 0↗ 1
build error0↗ 19↘ 0
coveralls.io unreachable9↘ 0≈ 0
unexplained1↗ 3↘ 0
git transfer dropped0↗ 2↘ 0
killed1≈ 1↘ 0
python traceback0↗ 2↘ 0
name resolution failed0↗ 1↘ 0
Per 1000 jobs run. Failed jobs for every thousand jobs that ran, so that a busy week does not look worse only for being busy.
causeOct 5-12Oct 12-19Oct 19-26
module tests failed0.6↗ 2.3↗ 7.3
ctest timeout1.1↗ 2.8≈ 2.7
test failed2.8↘ 1.1↗ 2.4
VK S3 unreachable0.4↗ 12.5↘ 1.9
entrypoint tag missing0.0≈ 0.0↗ 1.1
distro mirror0.9↗ 3.5↘ 0.6
pecl.php.net unreachable0.0≈ 0.0↗ 0.3
github.com unreachable0.0↗ 0.1≈ 0.1
branch not found0.0≈ 0.0↗ 0.1
build error0.0↗ 2.3↘ 0.0
coveralls.io unreachable1.7↘ 0.0≈ 0.0
unexplained0.2↗ 0.4↘ 0.0
git transfer dropped0.0↗ 0.2↘ 0.0
killed0.2↘ 0.1↘ 0.0
python traceback0.0↗ 0.2↘ 0.0
name resolution failed0.0↗ 0.1↘ 0.0
The columns. What each covers, to the second in UTC+3, and how much of CI ran in it.
spanfailedof jobs runper 1000
Oct 5-12(10-05 21:30:54, 10-12 21:30:54]3954517.2
Oct 12-19(10-12 21:30:54, 10-19 21:30:54]174829621.0
Oct 19-26(10-19 21:30:54, 10-26 21:30:54]115701416.4

By day

115 of them in the last 7 days.

Failed jobs. How many failed jobs show each cause; a job re-run and failing again counts each time.
causeMon 19Tue 20Wed 21Thu 22Fri 23Sat 24Sun 25
ctest timeout6↘ 2↗ 6↘ 3↘ 0≈ 0↗ 2
module tests failed3↗ 26↘ 20↘ 2↘ 0≈ 0≈ 0
test failed2↗ 5↘ 2↗ 8↘ 0≈ 0≈ 0
VK S3 unreachable1↗ 9↘ 2↘ 1↘ 0≈ 0≈ 0
entrypoint tag missing0≈ 0↗ 8↘ 0≈ 0≈ 0≈ 0
distro mirror1↗ 3↘ 0≈ 0≈ 0≈ 0≈ 0
pecl.php.net unreachable0↗ 2↘ 0≈ 0≈ 0≈ 0≈ 0
branch not found0↗ 1↘ 0≈ 0≈ 0≈ 0≈ 0
github.com unreachable0↗ 1↘ 0≈ 0≈ 0≈ 0≈ 0
Per 1000 jobs run. Failed jobs for every thousand jobs that ran, so that a busy week does not look worse only for being busy.
causeMon 19Tue 20Wed 21Thu 22Fri 23Sat 24Sun 25
ctest timeout2.6↘ 1.3↗ 5.9↘ 2.3↘ 0.0≈ 0.0↗ 3.2
module tests failed1.3↗ 16.6≈ 19.6↘ 1.6↘ 0.0≈ 0.0≈ 0.0
test failed0.9↗ 3.2↘ 2.0↗ 6.2↘ 0.0≈ 0.0≈ 0.0
VK S3 unreachable0.4↗ 5.7↘ 2.0↘ 0.8↘ 0.0≈ 0.0≈ 0.0
entrypoint tag missing0.0≈ 0.0↗ 7.8↘ 0.0≈ 0.0≈ 0.0≈ 0.0
distro mirror0.4↗ 1.9↘ 0.0≈ 0.0≈ 0.0≈ 0.0≈ 0.0
pecl.php.net unreachable0.0↗ 1.3↘ 0.0≈ 0.0≈ 0.0≈ 0.0≈ 0.0
branch not found0.0↗ 0.6↘ 0.0≈ 0.0≈ 0.0≈ 0.0≈ 0.0
github.com unreachable0.0↗ 0.6↘ 0.0≈ 0.0≈ 0.0≈ 0.0≈ 0.0
The columns. What each covers, to the second in UTC+3, and how much of CI ran in it.
spanfailedof jobs runper 1000
Mon 19(10-19 21:30:54, 10-20 21:30:54]1323255.6
Tue 20(10-20 21:30:54, 10-21 21:30:54]48156830.6
Wed 21(10-21 21:30:54, 10-22 21:30:54]38102137.2
Thu 22(10-22 21:30:54, 10-23 21:30:54]14128510.9
Fri 23(10-23 21:30:54, 10-24 21:30:54]01500.0
Sat 24(10-24 21:30:54, 10-25 21:30:54]0320.0
Sun 25(10-25 21:30:54, 10-26 21:30:54]26333.2

How a failure gets its cause

A failed job's log is matched against the signatures below. Each one that matches is a cause, so a job can count under two; a job that matches none is unexplained.

Tests failing in the integration workflow, the modules' own tests run against tarantool, count as module tests failed, apart from tarantool's own tests (the last entry below).

runner disk
# One cause, several faces: the disk fills, apt's lists are truncated and
# apt reports a signature it cannot verify; rpm and dnf ask for more space;
# yum finds its download directory full.
('runner disk', re.compile(
    r'(At least one invalid signature was encountered|No space left on device'
    r'|At least \d+ ?[KMG]i?B more space needed on the \S+ filesystem'
    r'|needs \d+ ?[KMG]i?B more space on the \S+ filesystem'
    r'|Insufficient space in download directory)', re.I)),
test failed
# A test that failed at least once, which a retry may have undone, or the
# final verdict of test-run or ctest, which nothing undoes. A module's
# integration job runs the module's own, older test-run: no worker
# number before a result, and '===== N tests failed:' as the verdict --
# 5 vshard failures read as unexplained until 2026-09-24; the line is in
# none of 3,237 passing logs. Other modules run other runners: crud's
# luatest closes with 'Failed tests:', php-queue's PHPUnit with
# 'FAILURES!' -- 50 failures read as unexplained until 2026-10-26, 42 of
# them php-queue's stats tests against the new stats.extra of
# tarantool/queue@f789e07c87e7 (10-20 to 10-22, fixed by
# tarantool-php/queue#27); neither line is in 300 passing logs. In the
# integration workflow all of these are the modules' tests (classify).
('test failed', re.compile(
    r'^\[\d+\]\s+\S+.*\[ fail \]'
    r'|^Failed tasks:'
    r'|^Failed tests:$'
    r'|^(?:FAILURES|ERRORS)!$'
    r'|^===== \d+ tests? failed:'
    r'|^\s+\d+ - .+ \((?:Failed|SEGFAULT|Subprocess aborted|Child aborted'
    r'|OTHER_FAULT|ILLEGAL|INTERRUPT|NUMERICAL|BAD_COMMAND|Exception'
    r'|Not Run)\)(?:\s.*)?$', re.M)),
github.com unreachable
# GitHub's artifact service as well, which the packaging tests download
# their test libraries from: one job of 2026-10-21, ECONNRESET.
('github.com unreachable', re.compile(
    r'(Failed to connect to (?:codeload\.)?github\.com'
    r'|Could not resolve host: (?:codeload\.)?github\.com'
    r'|Failed to download archive .https://codeload\.github\.com'
    r'|Unable to download artifact\(s\): Failed to ListArtifacts: Unable to'
    r' make request)', re.I)),
git transfer dropped
# A git transfer that started and then broke off. Kept apart from
# github.com unreachable until it is clear they are the same path failing:
# here the connection was made. The TLS lines are git built with GnuTLS
# (the packpack images) or OpenSSL reporting a reset or a mangled record
# mid-stream: 12 jobs read as unexplained until 2026-09-28, 11 on vds*.
# curl's own words for the same, "Failure when receiving data from the
# peer" and "Empty reply from server": 2 jobs of 2026-10-14, until
# 2026-10-26.
('git transfer dropped', re.compile(
    r'RPC failed; curl \d+|unexpected disconnect while reading sideband packet'
    r'|fatal: early EOF|invalid index-pack output'
    r"|fatal: unable to access '[^']*': (?:GnuTLS recv error"
    r'|OpenSSL SSL_(?:read|connect)|Recv failure'
    r'|Failure when receiving data from the peer|Empty reply from server)'
    r'|the remote end hung up upon initial contact')),
submodule commit gone
# A submodule commit GitHub no longer has. `git fetch` recurses into
# submodules on demand, so the environment action's `git fetch --tags -f`
# in a workspace without the history -- fetch-depth 1 on a runner back
# from a stop -- asks for every commit a submodule ever pointed at, and
# luajit bd34ea57 (2015) is gone. Once per workspace; see todo.d/0021.
('submodule commit gone', re.compile(
    r'upload-pack: not our ref [0-9a-f]{40}')),
ctest timeout
# ctest's closing summary, not its per-test '***Timeout' line: ctest runs
# with --repeat until-pass:3, so a timeout that passed on a repeat leaves
# that line in a green log (9 of 126 green ctest jobs). A newer ctest
# prints the test's labels after the verdict (3.31 in the FreeBSD VM, 4.x
# on macOS: "(Timeout)          app-luatest regression"); 3.17, the
# static build's, does not.
('ctest timeout', re.compile(r'^\s+\d+ - .+ \(Timeout\)(?:\s.*)?$',
                             re.M)),
distro mirror
# yum's "No more mirrors to try" is the CentOS 7 build failing on a
# package. apt's warning that it could not connect counts only with the
# failure it leads to, a package missing from the index that stayed
# behind: alone, it is in 7 jobs of 2026-10-07 that went on and failed
# on a test or on luacheck. One job each, unexplained until 2026-10-26.
('distro mirror', re.compile(
    r'(E: Failed to fetch|Cannot (?:download|prepare internal) mirrorlist'
    r'|Errors during downloading metadata'
    r'|temporary error \(try again later\)'
    r'|\[Errno 256\] No more mirrors to try'
    r"|W: Failed to fetch \S+ +Connection failed \[IP:[\s\S]*?"
    r"E: (?:Package '[^']+' has no installation candidate"
    r'|Unable to locate package))', re.I)),
luarocks
('luarocks', re.compile(
    r'(Error: Failed downloading.{0,80}(?:luarocks|rocks\.tarantool)'
    r'|Could not resolve host: (?:www\.)?luarocks\.org'
    r'|Failed to connect to (?:www\.)?luarocks\.org)', re.I)),
coveralls.io unreachable
# coveralls-lcov's connect to coveralls.io (Cloudflare) outlived Ruby's
# 60 s open_timeout. The gem retries a 5xx, not an exception, so one
# dropped connect fails the coverage job. 3 jobs on 10-02 and 10-04,
# unexplained until 2026-10-05; see todo.d/0022.
('coveralls.io unreachable', re.compile(
    r'\(Net::OpenTimeout\)(?:\n\s+from [^\n]*)*?\n\s+from \S*/coveralls-lcov-')),
VK S3 unreachable
VK_S3 = (r'(?:hb\.bizmrg\.com|[\w.-]*hb\.vkcs\.cloud|[\w.-]*hb\.vkcloud-storage\.ru'
         r'|95\.163\.53\.117)')

# Everything CI fetches from VK goes through one address, 95.163.53.117:
# download.tarantool.org and rocks.tarantool.org redirect to hb.bizmrg.com,
# and the packpack, distrib and multivac buckets are *.hb.vkcs.cloud and
# *.hb.vkcloud-storage.ru names for it. Its front end dropped and stalled
# connections on 2026-10-14 15:17-16:54 UTC and from 10-15 13:58 UTC; 57
# failed jobs read as unexplained until then. Only what names the address
# counts: apt merely warns about a repository it could not reach, and the
# job then fails on a package or a key from it, which can have other
# causes. See todo.d/0023. curl's "Connection reset by peer in connection
# to <host>" and apt's "Err:" line, whose reason names the address on the
# line after it, count too: one job each, until 2026-10-26.
('VK S3 unreachable', re.compile(
    r'(?:Could not connect to|Unable to connect to|Failed to connect to'
    r'|connect to|in connection to|ETIMEDOUT'
    r'|Timeout was reached for https://)\s*' + VK_S3
    + r'|' + VK_S3 + r'\S*: \[Errno \d+\] Timeout on'
    r'|Failed to fetch https://(?:download\.tarantool\.org|' + VK_S3
    + r')\S* +Could not wait for server fd'
    r'|Could not wait for server fd[^\n]*\[IP: ' + VK_S3
    + r'|Failed downloading https?://rocks\.tarantool\.org')),
checkpatch style
('checkpatch style', re.compile(r'has style problems, please review')),
luacheck
('luacheck', re.compile(
    r'^Total: (?:[1-9]\d* warnings? / \d+ errors?'
    r'|\d+ warnings? / [1-9]\d* errors?) in \d+ files', re.M)),
build error
# The compiler, the linker, CMake or ninja said so. Not make's own
# "*** [target] Error": a test, luacheck or packaging target fails through
# make too, and that line made every ctest failure a build error.
('build error', re.compile(
    r'CMake Error'
    r'|^\S+:\d+(?::\d+)?: (?:fatal )?error: '
    r'|error: .*\[-W'
    r'|undefined reference to |collect2: error: ld returned'
    r'|ld(?:\.lld|\.gold)?: error:|linker command failed'
    r'|No rule to make target'
    r'|^FAILED: |^ninja: build stopped', re.M)),
apk version
# abuild rejects the version of a 2.11 entrypoint commit; see
# ~/w/alpine-pkg-version.
('apk version', re.compile(r'ERROR: \S+: \S+ is not a valid version')),
python traceback
('python traceback', re.compile(r'^Traceback \(most recent call last\)', re.M)),
OOM
('OOM', re.compile(r'Out of memory: Killed|oom-kill', re.I)),
killed
# A step that ended on SIGKILL, 128 + 9, with no word of why: the OOM
# killer of the runner's container leaves none in the job's log, and
# neither does anything else that kills. 2 jobs on ghacts-shared runners,
# 10-07 and 10-15, until 2026-10-26; in none of 300 passing logs.
('killed', re.compile(r'##\[error\]Process completed with exit code 137\.')),
docker pull
('docker pull', re.compile(r'(Docker pull failed|toomanyrequests|manifest unknown)', re.I)),
entrypoint tag missing
# tools/check-entrypoint-tag.sh: once a release branch is cut, master's
# next commit needs the next entrypoint tag, and until it is pushed every
# run of master fails here. 8 jobs on 2026-10-22, after release/3.9 --
# two runs of four attempts each -- unexplained until 2026-10-26.
('entrypoint tag missing', re.compile(
    r'Missing tag for commit [0-9a-f]{40} after branching')),
pecl.php.net unreachable
# The PHP connectors' images install extensions from PECL, whose REST
# interface gave nothing back within 20-30 s: 2 jobs, 10-20 and 10-21.
('pecl.php.net unreachable', re.compile(
    r'No releases available for package "pecl\.php\.net/'
    r'|does not have REST info xml available')),
name resolution failed
# A getaddrinfo failure, whoever asked: rubygems.org from gem install on
# 2026-10-15, unexplained until 2026-10-26.
('name resolution failed', re.compile(
    r'Temporary failure in name resolution|Temporary failure resolving')),
branch not found
# actions/checkout asked for a branch the repository does not have:
# tarantool-ee's release/3.9, which came later than tarantool's.
('branch not found', re.compile(
    r"A branch or tag with the name '[^']+' could not be found")),
module tests failed
# The integration workflow runs each module's own tests against tarantool.
# A test failing there is counted apart from tarantool's own tests, which
# are the ones the flaky ranking names: it can be tarantool breaking the
# module, or the module itself -- php-queue's tests against a new queue
# were 42 failures in October 2026 (decided with the user, 2026-10-26).
MODULES = 'integration'


def classify(text, workflow):
    """The causes a failed job's log shows: every signature it matches, a
    test failure of the modules' workflow under a name of its own."""
    signs = [name for name, rx in SIGNS if rx.search(text)]
    if workflow == MODULES:
        signs = ['module tests failed' if s == 'test failed' else s
                 for s in signs]
    return signs or ['unexplained']