At SecureDrop project we have Python wheels built for Python 3.7 on Buster in a reproducible way. We use the same wheels inside of the Debian packages. The whole process has checks to verify the sha256sums based on gpg signatures, and at the end we point pip to a local directory to find all the dependencies.

Now, I am working to update the scripts so that we can build wheels for Ubuntu Focal, and in future we can use those wheels in the SecureDrop server side packages. While working on this, in the CI I suddenly noticed that all wheels started failing on the reproducibility test on Buster. I did not make any change other than the final directory path, so was wondering what is going on. Diffoscope helped to find out the difference:

│ -6 files, 34110 bytes uncompressed, 9395 bytes compressed:  72.5%
│ +6 files, 34132 bytes uncompressed, 9404 bytes compressed:  72.4%
├── six-1.11.0.dist-info/METADATA
│ @@ -9,14 +9,15 @@
│  Platform: UNKNOWN
│  Classifier: Programming Language :: Python :: 2
│  Classifier: Programming Language :: Python :: 3
│  Classifier: Intended Audience :: Developers
│  Classifier: License :: OSI Approved :: MIT License
│  Classifier: Topic :: Software Development :: Libraries
│  Classifier: Topic :: Utilities
│ +License-File: LICENSE
│  
│  .. image:: http://img.shields.io/pypi/v/six.svg
│     :target: https://pypi.python.org/pypi/six
│  
│  .. image:: https://travis-ci.org/benjaminp/six.svg?branch=master
│      :target: https://travis-ci.org/benjaminp/six
├── six-1.11.0.dist-info/RECORD
│ @@ -1,6 +1,6 @@
│  six.py,sha256=A08MPb-Gi9FfInI3IW7HimXFmEH2T2IPzHgDvdhZPRA,30888
│  six-1.11.0.dist-info/LICENSE,sha256=Y0eGguhOjJj0xGMImV8fUhpohpduJUIYJ9KivgNYEyg,1066
│ -six-1.11.0.dist-info/METADATA,sha256=vfvF0GW2vCjz99oMyLbw15XSkmo1IxC-G_339_ED4h8,1607
│ +six-1.11.0.dist-info/METADATA,sha256=Beq9GTDD6nYVwLYrN3oOcts0HSPHotfRWQ_Zn8_9a7g,1629
│  six-1.11.0.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│  six-1.11.0.dist-info/top_level.txt,sha256=_iVH_iYEtEXnD8nYGQYpYFUvkUW9sEO1GYbkeKSAais,4
│  six-1.11.0.dist-info/RECORD,,

It seems somehow the packages were picking up metadata about the LICENSE file. After looking into the environment, I found virtualenv is pulling latest setuptools into the virtualenv. Thus breaking the reproducibility. It was a quick fix. Yes, we do pin setuptools and pip in our wheels repository.

Next, the extension based wheels started failing, and I was going totally crazy to find out why there is some ABI change. I still don't know the correct reason, but noticed that the circleci/python:3.7-buster container image (and the upstream Python container) is using different flags to build the extensions than Debian Buster on vm/bare metal. For example, below on the top we have the command line from the container and then inside of the normal Debian Buster.

creating build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cprocessors.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cresultproxy' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cresultproxy.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cutils' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cutils.cpython-37m-x86_64-linux-gnu.so



creating build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cprocessors.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cresultproxy' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cresultproxy.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cutils' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cutils.cpython-37m-x86_64-linux-gnu.so
installing to build/bdist.linux-x86_64/wheel

If you want to see the difference in the flags, you can try out the following command:

python3 -m sysconfig

Look for PY_CFLAGS and related flags in the command output. Reproducibility depends on the environment, and we should not expect the latest container image (with the latest Python 3.7.x release) creating the same thing like in Debian Buster, but the failure started recently, we did not notice this a few months. Also, this means in future we should enable reproducibility tests on nightlies in CI.