[HTTPCLIENT-2381] Respect system properties by default

[rschmitt]

The practical effect of this change is that default JVM configuration for key stores (for mutual TLS) and proxy selection will be respected by default. This change has two goals.

First, applications and libraries built on the client will work with proxies much more reliably, since their authors no longer need to opt in to a non-default setting in order for things like proxies to be configurable in the usual manner (i.e. without code changes).

Second, this change, along with the change in 5.6 to make BUILTIN the default HostnameVerificationPolicy, makes the client's configuration philosophy fully consistent across all supported features. To wit:

1. HttpComponents itself is always configured programmatically and does not directly read system properties to obtain config values.
2. For a given feature, the default behavior is to delegate to the JDK implementation. This implementation may support out-of-band configuration via system properties, static methods, the java.security file, system-wide OS configuration, etc.
3. This delegation behavior can be overridden by a programmatic config option, either to directly customize the JDK-supplied implementation or to supply an alternate implementation.

This table shows what this philosophy looks like concretely across various features:

Feature	Default JDK behavior	Example JDK config property	Override strategy
Trust store	Load from OS	javax.net.ssl.trustStore	Set a TrustManager[]
Key store	Load nothing	javax.net.ssl.keyStore	Set a KeyManager[]
Hostname verification	Run HostnameChecker from sun.security.util	None	Set a HostnameVerifier and HostnameVerificationPolicy
Proxy config	Use system properties or load from OS	java.net.useSystemProxies	Set a ProxySelector or HttpRoutePlanner
Client cipher suites	Send all supported cipher suites	java.security.properties	Set an SSLContext
IP family selection	Prefer IPv6	java.net.preferIPv4Stack	Set a DetachedSocketFactory or DnsResolver, call setUnixDomainSocket, etc
DNS resolution	Use built-in resolver	networkaddress.cache.ttl	Set a DnsResolver

See also the previous discussion at:

#773

[rschmitt]

When we upgrade httpcore.version, there will be a handful of compiler warnings to clean up, but other than that this change builds successfully against both 5.4 and the latest commit.

[jonenst]

Thanks !
I have tested with the latest code from master, as a user I was indeed in control of the proxyHost/proxyPort through java system properties when the apache httpclient was created with HttpClients.createDefault(). For basicauth as you planned, it only works if you either:

• add manually as a developper the Authenticator.setDefault somewhere (note: this method is still an improvement, it gives locus of control to the final application developper; previously you could have been blocked because a lib developper used HttpClients.createDefault())
• or (should work?, untested and complicated?) use -javaagents to inject the code the same code as a user (haven't tested it though) => locus of control to the user if the user controls the java launcher command line (even indirectly through env variables...) but pretty technical, may be forbidden, and doesn't work with graalvm native image
• or run as a user an unauthenticated proxy that proxies upstream with your auth (e.g. squid proxy cache_peer login=my_username:my_password)
• ... are there other better ways ?

Maybe all this (and more options ? can you make it more secure (tied to user, not all tcp of localhost) ? socks (never used it)?) should go into some guide, otherwise I'm pretty sure people are going to be pretty confused (especially with the removal of the support for -DproxyUser -DproxyPass)

Thanks again for taking the time to work on this,
Cheers