Oracle OHS 14.1.2 Upgrade: Cache Broke Authentication

During a recent Oracle Fusion Middleware upgrade from 12.2.1.4 to 14.1.2, I encountered an interesting issue with Oracle HTTP Server (OHS) and WebGate. The upgraded components were running, but WebGate initialization failed and the application was inaccessible.

The root cause was not an incorrect OAM registration or a missing configuration file. It was a stale ObAccessClient.xml in the OHS/WebGate runtime cache.

WebGate Initialization Failure After Upgrade

After upgrading OHS and WebGate to 14.1.2, the application returned HTTP 500 errors. The WebGate logs reported:

Exception thrown during WebGate initialization

The Access Server has returned a fatal error with no detailed information.

Additional errors included:

OAP over REST libcurl operation failed

These messages initially suggested an OAM connectivity or agent configuration issue.

However, the OAM endpoint was reachable:

curl -sv \
  http://<oam-host>:14100/iam/access/binding/api/v10/oap \
  -o /dev/null

Response:

HTTP/1.1 405 Method Not Allowed

This confirmed HTTP connectivity to the endpoint, although it did not validate the complete OAP authentication exchange.

The next step was to verify whether WebGate was using the expected configuration.

Root Cause Analysis: Active Configuration vs. Runtime Cache

WebGate uses ObAccessClient.xml and wallet files for its OAM agent configuration. I identified three relevant locations:

# OAM-generated configuration
$IDM_DOMAIN/output/<agent_name>/

# Active WebGate configuration
$ADF_DOMAIN/config/fmwconfig/components/OHS/instances/ohs1/webgate/config/

# WebGate runtime cache
$ADF_DOMAIN/servers/ohs1/cache/

First, I compared the SHA-256 checksums of the OAM-generated files against the active OHS configuration.

File	OAM-generated	OHS active	Result
ObAccessClient.xml	b110e7f0...3f07	b110e7f0...3f07	MATCH
cwallet.sso	d3c1aacb...5ba4	d3c1aacb...5ba4	MATCH
wallet/cwallet.sso	5650892e...35a5	5650892e...35a5	MATCH

All three files were identical between the OAM-generated configuration and the active OHS configuration.

This was important: the deployed WebGate configuration and wallets were consistent with OAM.

I then compared the active ObAccessClient.xml with the cached version.

sha256sum \
  $ADF_DOMAIN/servers/ohs1/cache/ObAccessClient.xml \
  $ADF_DOMAIN/config/fmwconfig/components/OHS/instances/ohs1/webgate/config/ObAccessClient.xml

Actual output:

a1a5932a32b1c178461ab4cb373a14ff88446a6ad0cd9d2cdade90e5aecd4800  cache/ObAccessClient.xml
b110e7f0c8b153c1d17b7b79d4fc8273623fdb42dabafe0054d7c13e025d3f07  webgate/config/ObAccessClient.xml

The checksums are different. The cache directory also contained:

-rw-r----- 7618 Oct 20 16:26 ObAccessClient.xml
-rw-r-----    0 May 25  2021 ObAccessClient.xml.lck
-rw-r-----    0 May 25  2021 polltracking.lck

While different checksums do not automatically indicate corruption, the cached configuration was inconsistent with the verified active configuration. This became my primary suspect.

Resolution: Removing the Stale WebGate Cache

I stopped OHS using the domain management script:

$ADF_DOMAIN/bin/stopComponent.sh ohs1

Output:

Successfully Connected to Node Manager.
Killing server ohs1 ...
Successfully killed server ohs1

I confirmed that no OHS HTTP processes remained:

ps -ef | grep '[h]ttpd'

No processes were returned. Next, I preserved the existing cached configuration:

cd $ADF_DOMAIN/servers/ohs1/cache/
cp -p ObAccessClient.xml \
  ObAccessClient.xml.pre14c_20261023

I removed only the cached configuration and its associated lock file:

rm -f ObAccessClient.xml ObAccessClient.xml.lck

The active WebGate configuration and wallets were left untouched. I then restarted OHS:

$ADF_DOMAIN/bin/startComponent.sh ohs1

Output:

Successfully Connected to Node Manager.
Starting server ohs1 ...
Successfully started server ohs1 ...

An interesting observation occurred immediately after the restart. The initial directory listing showed that ObAccessClient.xml had not yet been recreated:

ObAccessClient.xml.pre14c_20261023
polltracking.lck

A subsequent check showed that WebGate had automatically regenerated its runtime cache:

-rw-r----- 7618 Oct 23 10:39 ObAccessClient.xml
-rw-r-----    0 Oct 23 10:39 ObAccessClient.xml.lck
-rw-r----- 7618 Oct 20 16:26 ObAccessClient.xml.pre14c_20261023

The newly generated files had fresh timestamps, while the backup preserved the original cached configuration.

Following this operation, WebGate initialized successfully and the protected application became accessible again through the expected authentication flow.

No modification to the active configuration, wallet replacement, agent re-registration or WebGate reinstallation was necessary.

Conclusion

This case demonstrates an important troubleshooting consideration after upgrading Oracle HTTP Server and WebGate.

Even when the active ObAccessClient.xml and wallet files match the configuration generated by OAM, an outdated WebGate runtime cache can still interfere with initialization.

For future Oracle FMW upgrades, comparing the active and cached WebGate configurations is a valuable diagnostic step when authentication fails despite apparently correct OAM integration.

Related blogs.