Skip to content

prevent downgrade of certifi#1

Open
Ni-Langguth wants to merge 1 commit intomainfrom
keep_current_certifi
Open

prevent downgrade of certifi#1
Ni-Langguth wants to merge 1 commit intomainfrom
keep_current_certifi

Conversation

@Ni-Langguth
Copy link
Copy Markdown

@Ni-Langguth Ni-Langguth commented Jan 20, 2026

Building the metapackage LCG, containing the Python package as a requirement, led to an SSL validation error when uploading symlinks to s3 on some of our platforms (all of them on x86_64 and el9 with these compilers: clang16-dbg, clang16-opt, clang19-opt, gcc13-dbg, gcc13-opt, gcc14fp-opt), more detail below under "Typical build failure".
Other build platforms uploaded the symlink successfully (all of them on x86_64 with OS and compilers: el9-clang19-dgb, el9-gcc14-dbg, el9-gcc14-opt, el9-gcc15-dbg, el9-gcc15-opt, el10-gcc15-opt), more detail below under "Typical success".
In the python recipe of common.bits, SSL_CERT_FILE is exported here. This happened just before the last upload of symlinks in each of our build logs:

2026-01-20@11:32:34:DEBUG:LCG:LCG:0: +++ export SSL_CERT_FILE=/builds/sailer/sw/x86_64-el9-clang16-dbg/Python/v3.9.16-1/lib/python3.9/site-packages/certifi/cacert.pem
2026-01-20@11:32:34:DEBUG:LCG:LCG:0: +++ SSL_CERT_FILE=/builds/sailer/sw/x86_64-el9-clang16-dbg/Python/v3.9.16-1/lib/python3.9/site-packages/certifi/cacert.pem

Certifi was downgraded to version 2022.12.17 when executing this line of the python recipe.

2026-01-20@11:32:13:DEBUG:LCG:Python:0: Collecting certifi==2022.12.7
2026-01-20@11:32:13:DEBUG:LCG:Python:0:   Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
2026-01-20@11:32:13:DEBUG:LCG:Python:0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 KB 6.9 MB/s eta 0:00:00
2026-01-20@11:32:13:DEBUG:LCG:Python:0: Installing collected packages: certifi
2026-01-20@11:32:13:DEBUG:LCG:Python:0:   Attempting uninstall: certifi
2026-01-20@11:32:13:DEBUG:LCG:Python:0:     Found existing installation: certifi 2026.1.4
2026-01-20@11:32:13:DEBUG:LCG:Python:0:     Uninstalling certifi-2026.1.4:
2026-01-20@11:32:13:DEBUG:LCG:Python:0:       Successfully uninstalled certifi-2026.1.4
2026-01-20@11:32:13:DEBUG:LCG:Python:0: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

After applying the commit in this PR, all our builds ran without SSL validation errors. The SSL_CERT_FILE variable was set to a local installation of certifi, because this satisfied the version requirement. Should certifi just be installed under INSTALLROOT instead of relying on the local path of python?

2026-01-20@14:18:36:DEBUG:LCG:Python:0: Requirement already satisfied: certifi>=2022.12.7 in /home/sftnight/.local/lib/python3.9/site-packages (2026.1.4)
...
2026-01-20@14:20:42:DEBUG:LCG:LCG:0: +++ export SSL_CERT_FILE=/home/sftnight/.local/lib/python3.9/site-packages/certifi/cacert.pem
2026-01-20@14:20:42:DEBUG:LCG:LCG:0: +++ SSL_CERT_FILE=/home/sftnight/.local/lib/python3.9/site-packages/certifi/cacert.pem

#Typical build failure:

2026-01-20@11:32:34:DEBUG:LCG:LCG:0: Comparing dist symlinks against S3 from TARS/x86_64-el9-clang16-dbg/dist/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:32:34:DEBUG:LCG:LCG:0: Comparing dist symlinks against S3 from TARS/x86_64-el9-clang16-dbg/dist-direct/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:32:34:DEBUG:LCG:LCG:0: Comparing dist symlinks against S3 from TARS/x86_64-el9-clang16-dbg/dist-runtime/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:32:34:DEBUG:LCG:LCG:0: Uploading tarball and symlinks for LCG v1.0-2026-01-
[log_fail.tar.gz](https://github.com/user-attachments/files/24740173/log_fail.tar.gz)
[log_success.tar.gz](https://github.com/user-attachments/files/24740174/log_success.tar.gz)
20-1 (3f8d29eda25dbeba619e486a76a87a255977efac) to S3
2026-01-20@11:32:38:ERROR:LCG:LCG:0: Failed to upload symlink TARS/x86_64-el9-clang16-dbg/dist/LCG/LCG-v1.0-2026-01-20-1/FreeType-v2.10.1-1.x86_64-el9-clang16-dbg.tar.gz: SSL validation failed for https://s3.cern.ch/bucketofpieces/TARS/x86_64-el9-clang16-dbg/dist/LCG/LCG-v1.0-2026-01-20-1/FreeType-v2.10.1-1.x86_64-el9-clang16-dbg.tar.gz [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket
    context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/botocore/httpsession.py", line 465, in send
    urllib_response = conn.urlopen(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 509, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 700, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1015, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 404, in ssl_wrap_socket
    raise SSLError(e)
urllib3.exceptions.SSLError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sftnight/.local/bin/bitsBuild", line 133, in <module>
    doMain(args, parser)
  File "/home/sftnight/.local/bin/bitsBuild", line 82, in doMain
    doBuild(args, parser)
  File "/builds/sailer/bits/bits_helpers/build.py", line 1417, in doBuild
    runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scriptDir, workDir, syncHelper)
  File "/builds/sailer/bits/bits_helpers/build.py", line 663, in runBuildCommand
    doFinalSync(spec, specs, args, syncHelper)
  File "/builds/sailer/bits/bits_helpers/build.py", line 677, in doFinalSync
    syncHelper.upload_symlinks_and_tarball(spec)
  File "/builds/sailer/bits/bits_helpers/sync.py", line 732, in upload_symlinks_and_tarball
    future.result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/builds/sailer/bits/bits_helpers/sync.py", line 713, in _upload_single_symlink
    self.s3.put_object(Bucket=self.writeStore,
  File "/usr/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.9/site-packages/botocore/client.py", line 963, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/lib/python3.9/site-packages/botocore/client.py", line 986, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/lib/python3.9/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/lib/python3.9/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/usr/lib/python3.9/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/lib/python3.9/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python3.9/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/usr/lib/python3.9/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/usr/lib/python3.9/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/lib/python3.9/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/usr/lib/python3.9/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/usr/lib/python3.9/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/usr/lib/python3.9/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/usr/lib/python3.9/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/usr/lib/python3.9/site-packages/botocore/httpsession.py", line 492, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://s3.cern.ch/bucketofpieces/TARS/x86_64-el9-clang16-dbg/dist/LCG/LCG-v1.0-2026-01-20-1/FreeType-v2.10.1-1.x86_64-el9-clang16-dbg.tar.gz [Errno 2] No such file or directory

#Typical build success:

2026-01-20@11:34:46:DEBUG:LCG:LCG:0: Comparing dist symlinks against S3 from TARS/x86_64-el9-gcc14-dbg/dist/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:34:46:DEBUG:LCG:LCG:0: Comparing dist symlinks against S3 from TARS/x86_64-el9-gcc14-dbg/dist-direct/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:34:46:DEBUG:LCG:LCG:0: Comparing dist symlinks against S3 from TARS/x86_64-el9-gcc14-dbg/dist-runtime/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:34:46:DEBUG:LCG:LCG:0: Uploading tarball and symlinks for LCG v1.0-2026-01-20-1 (3f8d29eda25dbeba619e486a76a87a255977efac) to S3
2026-01-20@11:34:57:DEBUG:LCG:LCG:0: Uploaded 12 dist symlinks to S3 from TARS/x86_64-el9-gcc14-dbg/dist/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:34:57:DEBUG:LCG:LCG:0: Uploaded 4 dist symlinks to S3 from TARS/x86_64-el9-gcc14-dbg/dist-direct/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:34:57:DEBUG:LCG:LCG:0: Uploaded 7 dist symlinks to S3 from TARS/x86_64-el9-gcc14-dbg/dist-runtime/LCG/LCG-v1.0-2026-01-20-1
2026-01-20@11:34:57:DEBUG:LCG:LCG:0: Uploaded 23 dist symlinks in 9.47 seconds
Build of LCG successfully completed on `runner-mbsnm5pr-project-220187-concurrent-7-108hvv4o'.

m-fol pushed a commit to m-fol/common.bits that referenced this pull request Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant