Random 'Connection reset' error in build pipelines using Maven on Kubernetes runners

We're seeing a random 'connection reset' error on our CI Pipelines building Java apps with Maven. The error occurs randomly, if you rerun a job that has the error it will eventually run to completion with no errors.

When the error occurs, re-running the job may see the same error but downloading a different dependency, or the job may complete with no errors.

Example of error:

Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact com.google.cloud:google-cloud-bigquerystorage-bom:pom:3.2.0 from/to central (https://repo.maven.apache.org/maven2): Connection reset

Example of recent jobs with the error:

• https://gitlab.developers.cam.ac.uk/uis/devops/hr/wr/wrs-webapp/-/jobs/2125107
• https://gitlab.developers.cam.ac.uk/uis/devops/hr/tes/tes-webapp/-/jobs/2126659
• https://gitlab.developers.cam.ac.uk/uis/devops/hr/tes/tes-webapp/-/jobs/2126652
• https://gitlab.developers.cam.ac.uk/uis/devops/hr/tes/tes-webapp/-/jobs/2125282

On some pipelines the build job output appears to get truncated but fails randomly without showing the 'Connection Reset' error in the log, but we're assuming these are failing for the same reason too as they are also random, and if you re-run they will complete eventually:

• https://gitlab.developers.cam.ac.uk/uis/devops/hr/tes/tes-webapp/-/jobs/2126501

From research, this question describes the same symptoms we are seeing, and the linked article describes a potential issue with invalid packets running Docker on Kubernetes causing the issue, and a suggestion for a iptables config change to drop the invalid packages that avoids the issue

added Operations issuetypeBug teamHolberton workflowBacklog labels

removed workflowBacklog label

removed Operations label

removed teamHolberton label

added teamCloud label

changed title from Connection reset' error in build pipelines to Random 'Connection reset' error in build pipelines using Maven on Kubernetes runners

changed the description

assigned to @du228

added workflowIn Progress label

changed iteration to Cloud Team Sprints Feb 26, 2025 - Mar 11, 2025

mentioned in merge request !120 (merged)

mentioned in merge request !121 (merged)

I spent couple of days in experiments, troubleshooting and searching for similar issue on the internet.

The main issue seems to be caused by maven's transport component.

https://maven.apache.org/wagon/wagon-providers/wagon-http/

https://maven.apache.org/guides/mini/guide-resolver-transport.html

People on the Internet experiences the same issues randomly

https://stackoverflow.com/questions/65349648/why-does-my-maven-dependency-fetching-fail-with-a-connection-reset-when-running

https://stackoverflow.com/questions/55899091/maven-retry-dependency-download-if-failed

And all answers/recommendations are about setting http* or wagon* options. This seems to be "fixed" issue

https://issues.apache.org/jira/browse/MRESOLVER-383

https://issues.apache.org/jira/browse/WAGON-486

But is seems it's not. I found a good article about it - https://questdb.com/blog/maven-troubleshooting-open-source/

And it sounds like it was fixed in maven 3.9.1 But we use 3.9.9 and we still see this issue (proof: https://gitlab.developers.cam.ac.uk/uis/devops/hr/wr/wrs-webapp/-/jobs/2153454)

Adding -Dmaven.resolver.transport=wagon to maven options list in CI templates !121 (merged) "helped", in terms of "connection reset"

But still, ~50% of jobs are red but for another reason

https://gitlab.developers.cam.ac.uk/uis/devops/hr/wr/wrs-webapp/-/jobs/2156528

I am sort of stuck here, tbh.

Part of the guides suggesting to "tune" IPTABLES chains but most probably it's not something we could easily do in GKE managed cluster.

@rjw57 just for vis, as this is affecting dev experience.

Worth to mention, that we see this issue on 3.9 and on 3.8 as well. (TES uses 3.8 both in dockerimage and in CI https://gitlab.developers.cam.ac.uk/uis/devops/hr/tes/tes-webapp/-/blob/main/.gitlab-ci.yml?ref_type=heads#L12) so it's not version specific issue. It

connection reset error on 3.8.8 - https://gitlab.developers.cam.ac.uk/uis/devops/hr/tes/tes-webapp/-/jobs/2153937

connection reset error on 3.9.9 - https://gitlab.developers.cam.ac.uk/uis/devops/hr/wr/wrs-webapp/-/jobs/2153454

Re. the failure for another reason:

This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced

We occasionally see this locally if a dependency download has failed for some reason and has marked the attempt as failed in your local ~/.m2/ cache, but that shouldn't occur in the pipeline if we're downloading all deps fresh every time and not caching anything - we can look into that seperately.

Thanks for the research and notes on this. Re. the fix in Maven 3.9.1 - I don't think we're consistently using 3.9.x or above when we're building within the Docker images though, some of the steps are 3.8? So we could bump them up to > 3.9.1?

we see this issue on 3.9 as well

I was looking at the changes on the fix, is there also a config property that can bump the retry attempts up to something higher than the default of 3? MIght be worth trying?

I also noticed the last comments on https://issues.apache.org/jira/browse/MRESOLVER-383 :

I haven't faced this with newer version. Let's close this for now

... so is it worth bumping up to the latest Maven version we can and see if that also avoids the issue?

Taking a look at the job you said is still getting the error on 3.9.9 ...

Sorry, I'm probably just confirming what you're already looked at, but yes even that maven:verify job is using 3.9.9, so bumping up to latest is not a fix either

docker run -it maven:3.9-eclipse-temurin-8 bash 
root@e3cb6ed8a83e:/# mvn -v
Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /usr/share/maven
Java version: 1.8.0_432, vendor: Temurin, runtime: /opt/java/openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.10.14-linuxkit", arch: "aarch64", family: "unix"

yes, that's what I meant. So 3.9.9 and we still see this "connection reset".

Yes, so, at the very least we can say (intermediate results):

• issue is not version specific;
• lates version of 3.9 (3.9.9) also affected;
• changing transport from new (maven-resolver-transport-http) to legacy (maven-resolver-transport-wagon) as multiple articles suggested didn't help;
• setting -Dhttp.keepAlive=false and -Dmaven.wagon.http.pool=false options as another part of articles and stackeoverflows suggested didn't help as well;

mentioned in issue #108

removed iteration Cloud Team Sprints Feb 26, 2025 - Mar 11, 2025

changed iteration to Cloud Team Sprints Mar 12, 2025 - Mar 25, 2025

Few days ago I opened a support ticket on GCP

https://console.cloud.google.com/support/cases/detail/v2/57817785?hl=en&inv=1&invt=Abr0mw&project=gitlab-runner-prod-22257483

I got the answer, but nothing useful, just generic recommendations :( .

This error occurs when a peer across the network unexpectedly closes the TCP connection established by the application. To address this issue, here are several steps you can try:

• If you are performing background tasks with CPU throttling, consider using instance-based billing settings to avoid unnecessary connection interruptions.
• Ensure that your application stays within the outbound request timeouts. If any connections remain idle beyond the configured timeout threshold, the gateway may close the connection. This could lead to connection resets.
• By default, the TCP socket keep-alive option is disabled for Cloud Run. There is no direct way to configure this at the service level. However, you can enable the keep-alive option by providing the necessary socket settings when opening a new TCP connection, based on the client library you are using in your application.
• Outbound connections[2] may be reset due to infrastructure updates. If your application relies on long-lived connections, we recommend implementing a mechanism to re-establish connections to avoid the reuse of dead connections.
• If you're using an HTTP proxy to route your Cloud Run services or jobs egress traffic, and the proxy enforces maximum connection duration, the proxy might silently drop long-running TCP connections such as the ones established using connection pooling. This causes HTTP clients to fail when reusing an already closed connection. If you intend to route egress traffic through an HTTP proxy, you must implement connection validation, retries, and exponential backoff. For connection pools, configure maximum values for connection age, idle connections, and connection idle timeout.

Basically, chatgpt type of answer

mentioned in merge request !122 (merged)

added uis/devops/hr#36 as parent epic

removed iteration Cloud Team Sprints Mar 12, 2025 - Mar 25, 2025

changed iteration to Cloud Team Sprints Mar 26, 2025 - Apr 8, 2025

removed iteration Cloud Team Sprints Mar 26, 2025 - Apr 8, 2025

changed iteration to Cloud Team Sprints Apr 9, 2025 - Apr 22, 2025