[2024-10-02 14:30:46,563][ INFO][PID:4112329] Marking build as starting [2024-10-02 14:30:46,660][ INFO][PID:4112329] Trying to allocate VM [2024-10-02 14:31:34,802][ INFO][PID:4112329] Allocated host ResallocHost, ticket_id=5224246, hostname=3.84.0.178 [2024-10-02 14:31:34,803][ INFO][PID:4112329] Allocating ssh connection to builder [2024-10-02 14:31:34,804][ INFO][PID:4112329] Checking that builder machine is OK [2024-10-02 14:31:35,008][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.0.178 '/usr/bin/test -f /etc/mock/epel-8-x86_64.cfg' [2024-10-02 14:31:35,224][ INFO][PID:4112329] Filling build.info file with builder info [2024-10-02 14:31:35,226][ INFO][PID:4112329] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "lantw44", "project_name": "tcc", "project_dirname": "tcc", "submitter": "lantw44", "ended_on": null, "started_on": 1727879495.2257547, "submitted_on": null, "status": 3, "chroot": "epel-8-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8102616-epel-8-x86_64", "build_id": 8102616, "package_name": "tcc-self", "package_version": "0.9.28-0.7.20240928git3ead10d", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lantw44/tcc/tcc-self", "git_hash": "7f7b48718b3617230a08d630e02892c4bc662ddd", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "lantw44/tcc--lantw44", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "background": false, "bootstrap": "on", "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/lantw44/tcc/epel-8-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/lantw44/tcc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lantw44/tcc", "result_dir": "08102616-tcc-self", "built_packages": "", "id": 8102616, "mockchain_macros": { "copr_username": "lantw44", "copr_projectname": "tcc", "vendor": "Fedora Project COPR (lantw44/tcc)" } } ] } [2024-10-02 14:31:35,290][ INFO][PID:4112329] Sending fedora-messaging bus message in build.start [2024-10-02 14:31:35,998][ INFO][PID:4112329] Sending fedora-messaging bus message in chroot.start [2024-10-02 14:31:36,023][ INFO][PID:4112329] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8102616-epel-8-x86_64 --chroot epel-8-x86_64 --detached [2024-10-02 14:31:36,390][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 1 [2024-10-02 14:31:36,392][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.0.178 copr-rpmbuild-log [2024-10-02 14:31:41,446][ INFO][PID:4112329] Status: False [2024-10-02 14:33:52,192][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:34:02,193][WARNING][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-10-02 14:34:12,194][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 2 [2024-10-02 14:34:12,195][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.0.178 copr-rpmbuild-log [2024-10-02 14:35:12,264][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:35:22,267][WARNING][PID:4112329] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-10-02 14:35:32,268][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 3 [2024-10-02 14:35:32,269][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.0.178 copr-rpmbuild-log [2024-10-02 14:36:32,338][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:36:42,346][WARNING][PID:4112329] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-10-02 14:36:52,348][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 4 [2024-10-02 14:36:52,353][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.84.0.178 copr-rpmbuild-log [2024-10-02 14:37:52,431][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:38:02,444][ ERROR][PID:4112329] Giving up for unstable SSH, failures: 1727879642.193267, 1727879722.267269, 1727879802.3462038, 1727879882.4443643 [2024-10-02 14:38:02,449][ ERROR][PID:4112329] Re-try request for task on 'ResallocHost, ticket_id=5224246, hostname=3.84.0.178': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-10-02 14:38:02,451][ INFO][PID:4112329] Releasing VM back to pool [2024-10-02 14:38:02,548][ INFO][PID:4112329] Retry #1 (on other host) [2024-10-02 14:38:02,570][ INFO][PID:4112329] Trying to allocate VM [2024-10-02 14:43:20,912][ INFO][PID:4112329] Allocated host ResallocHost, ticket_id=5224529, hostname=54.167.16.145 [2024-10-02 14:43:20,913][ INFO][PID:4112329] Allocating ssh connection to builder [2024-10-02 14:43:20,914][ INFO][PID:4112329] Checking that builder machine is OK [2024-10-02 14:43:21,250][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.167.16.145 '/usr/bin/test -f /etc/mock/epel-8-x86_64.cfg' [2024-10-02 14:43:21,466][ INFO][PID:4112329] Filling build.info file with builder info [2024-10-02 14:43:21,468][ INFO][PID:4112329] Not re-notifying FE and msg buses for the new host. [2024-10-02 14:43:21,468][ INFO][PID:4112329] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8102616-epel-8-x86_64 --chroot epel-8-x86_64 --detached [2024-10-02 14:43:21,832][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 1 [2024-10-02 14:43:21,835][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.167.16.145 copr-rpmbuild-log [2024-10-02 14:43:26,869][ INFO][PID:4112329] Status: False [2024-10-02 14:44:25,744][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:44:35,748][WARNING][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-10-02 14:44:45,751][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 2 [2024-10-02 14:44:45,756][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.167.16.145 copr-rpmbuild-log [2024-10-02 14:45:45,833][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:45:55,837][WARNING][PID:4112329] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-10-02 14:46:05,838][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 3 [2024-10-02 14:46:05,840][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.167.16.145 copr-rpmbuild-log [2024-10-02 14:47:05,916][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:47:15,920][WARNING][PID:4112329] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2024-10-02 14:47:25,921][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 4 [2024-10-02 14:47:25,923][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.167.16.145 copr-rpmbuild-log [2024-10-02 14:48:25,995][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2024-10-02 14:48:35,996][ ERROR][PID:4112329] Giving up for unstable SSH, failures: 1727880275.7486937, 1727880355.8371964, 1727880435.9202776, 1727880515.9965386 [2024-10-02 14:48:35,998][ ERROR][PID:4112329] Re-try request for task on 'ResallocHost, ticket_id=5224529, hostname=54.167.16.145': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2024-10-02 14:48:35,998][ INFO][PID:4112329] Releasing VM back to pool [2024-10-02 14:48:36,014][ INFO][PID:4112329] Retry #2 (on other host) [2024-10-02 14:48:36,016][ INFO][PID:4112329] Trying to allocate VM [2024-10-02 14:49:04,109][ INFO][PID:4112329] Allocated host ResallocHost, ticket_id=5224757, hostname=54.167.16.145 [2024-10-02 14:49:04,110][ INFO][PID:4112329] Allocating ssh connection to builder [2024-10-02 14:49:04,111][ INFO][PID:4112329] Checking that builder machine is OK [2024-10-02 14:50:04,182][ ERROR][PID:4112329] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 54.167.16.145 port 22: Connection timed out [2024-10-02 14:50:14,183][ ERROR][PID:4112329] Re-try request for task on 'ResallocHost, ticket_id=5224757, hostname=54.167.16.145': Unable to finish after 1 SSH attempts [2024-10-02 14:50:14,184][ INFO][PID:4112329] Releasing VM back to pool [2024-10-02 14:50:14,202][ INFO][PID:4112329] Retry #3 (on other host) [2024-10-02 14:50:14,204][ INFO][PID:4112329] Trying to allocate VM [2024-10-02 14:55:32,420][ INFO][PID:4112329] Allocated host ResallocHost, ticket_id=5224791, hostname=44.201.16.161 [2024-10-02 14:55:32,421][ INFO][PID:4112329] Allocating ssh connection to builder [2024-10-02 14:55:32,422][ INFO][PID:4112329] Checking that builder machine is OK [2024-10-02 14:55:32,779][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.16.161 '/usr/bin/test -f /etc/mock/epel-8-x86_64.cfg' [2024-10-02 14:55:32,994][ INFO][PID:4112329] Filling build.info file with builder info [2024-10-02 14:55:33,000][ INFO][PID:4112329] Not re-notifying FE and msg buses for the new host. [2024-10-02 14:55:33,000][ INFO][PID:4112329] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8102616-epel-8-x86_64 --chroot epel-8-x86_64 --detached [2024-10-02 14:55:33,396][ INFO][PID:4112329] Downloading the builder-live.log file, attempt 1 [2024-10-02 14:55:33,398][ INFO][PID:4112329] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.16.161 copr-rpmbuild-log [2024-10-02 14:55:38,408][ INFO][PID:4112329] Status: False [2024-10-02 14:58:10,451][ INFO][PID:4112329] Downloading results from builder [2024-10-02 14:58:10,452][ INFO][PID:4112329] rsyncing of mockbuilder@44.201.16.161:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self started [2024-10-02 14:58:10,453][ INFO][PID:4112329] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.201.16.161:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/ &> /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/build-08102616.rsync.log [2024-10-02 14:58:10,769][ INFO][PID:4112329] rsyncing finished. [2024-10-02 14:58:10,770][ INFO][PID:4112329] Releasing VM back to pool [2024-10-02 14:58:10,787][ INFO][PID:4112329] Searching for 'success' file in resultdir [2024-10-02 14:58:10,788][ INFO][PID:4112329] Going to sign pkgs from source: 8102616-epel-8-x86_64 in chroot: /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64 [2024-10-02 14:58:10,789][ INFO][PID:4112329] Calling '/bin/sign -u lantw44#tcc@copr.fedorahosted.org -p' (attempt #1) [2024-10-02 14:58:11,095][ INFO][PID:4112329] Calling '/bin/sign -4 -h sha256 -u lantw44#tcc@copr.fedorahosted.org -r /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/tcc-self-0.9.28-0.7.20240928git3ead10d.el8.src.rpm' (attempt #1) [2024-10-02 14:58:11,954][ INFO][PID:4112329] signed rpm: /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/tcc-self-0.9.28-0.7.20240928git3ead10d.el8.src.rpm [2024-10-02 14:58:11,955][ INFO][PID:4112329] Calling '/bin/sign -4 -h sha256 -u lantw44#tcc@copr.fedorahosted.org -r /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/tcc-self-0.9.28-0.7.20240928git3ead10d.el8.x86_64.rpm' (attempt #1) [2024-10-02 14:58:12,798][ INFO][PID:4112329] signed rpm: /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/tcc-self-0.9.28-0.7.20240928git3ead10d.el8.x86_64.rpm [2024-10-02 14:58:12,800][ INFO][PID:4112329] Sign done [2024-10-02 14:58:12,801][ INFO][PID:4112329] Incremental createrepo run, adding 08102616-tcc-self into https://download.copr.fedorainfracloud.org/results/lantw44/tcc/epel-8-x86_64, (auto-create-repo=True) [2024-10-02 14:58:12,807][ INFO][PID:4112329] Running command 'copr-repo --batched /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64 --add 08102616-tcc-self --no-appstream-metadata' as PID 141771 [2024-10-02 14:58:14,203][ INFO][PID:4112329] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64 --add 08102616-tcc-self --no-appstream-metadata) [2024-10-02 14:58:14,206][ INFO][PID:4112329] Getting build details [2024-10-02 14:58:14,210][ INFO][PID:4112329] Listing built binary packages in /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self [2024-10-02 14:58:14,215][ INFO][PID:4112329] Built packages: ['tcc-self 0.9.28'] [2024-10-02 14:58:14,216][ INFO][PID:4112329] build details: {'built_packages': 'tcc-self 0.9.28'} [2024-10-02 14:58:14,216][ INFO][PID:4112329] Retrieving pubkey [2024-10-02 14:58:14,217][ INFO][PID:4112329] Calling '/bin/sign -u lantw44#tcc@copr.fedorahosted.org -p' (attempt #1) [2024-10-02 14:58:14,409][ INFO][PID:4112329] Added pubkey for user lantw44 project tcc into: /var/lib/copr/public_html/results/lantw44/tcc/pubkey.gpg [2024-10-02 14:58:14,410][ INFO][PID:4112329] Finished build: id=8102616 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/lantw44/tcc chroot=epel-8-x86_64 [2024-10-02 14:58:14,413][ INFO][PID:4112329] Worker succeeded build, took 161.4136028289795 [2024-10-02 14:58:14,415][ INFO][PID:4112329] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "lantw44", "project_name": "tcc", "project_dirname": "tcc", "submitter": "lantw44", "ended_on": 1727881094.4137626, "started_on": 1727880933.0001597, "submitted_on": null, "status": 1, "chroot": "epel-8-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8102616-epel-8-x86_64", "build_id": 8102616, "package_name": "tcc-self", "package_version": "0.9.28-0.7.20240928git3ead10d", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lantw44/tcc/tcc-self", "git_hash": "7f7b48718b3617230a08d630e02892c4bc662ddd", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "lantw44/tcc--lantw44", "results": { "packages": [ { "name": "tcc-self", "epoch": null, "version": "0.9.28", "release": "0.7.20240928git3ead10d.el8", "arch": "x86_64" }, { "name": "tcc-self", "epoch": null, "version": "0.9.28", "release": "0.7.20240928git3ead10d.el8", "arch": "src" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "background": false, "bootstrap": "on", "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/lantw44/tcc/epel-8-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/lantw44/tcc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lantw44/tcc", "result_dir": "08102616-tcc-self", "built_packages": "tcc-self 0.9.28", "id": 8102616, "mockchain_macros": { "copr_username": "lantw44", "copr_projectname": "tcc", "vendor": "Fedora Project COPR (lantw44/tcc)" } } ] } [2024-10-02 14:58:14,535][ INFO][PID:4112329] Sending fedora-messaging bus message in build.end [2024-10-02 14:58:14,563][ INFO][PID:4112329] Compressing /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/builder-live.log by gzip [2024-10-02 14:58:14,566][ INFO][PID:4112329] Running command 'gzip /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/builder-live.log' as PID 142102 [2024-10-02 14:58:14,577][ INFO][PID:4112329] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/builder-live.log) [2024-10-02 14:58:14,578][ INFO][PID:4112329] Compressing /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/backend.log by gzip [2024-10-02 14:58:14,579][ INFO][PID:4112329] Running command 'gzip /var/lib/copr/public_html/results/lantw44/tcc/epel-8-x86_64/08102616-tcc-self/backend.log' as PID 142106