TCP/IP Connection Dropping

[mcisho]

On 2026-06-18 a user named Venjoa posted on the Hercules-390 group:

z/OS TCP/IP Connection Dropping on Hercules Emulator (OSA/HSCH Loop)

We are experiencing an intermittent connectivity drop between our CentOS Linux host system and the z/OS guest(3.1) running inside the Hercules emulator. The connection initializes correctly but drops after several hours, falling into an unrecoverable 5-minute retry loop.
 
18:01:12 HHC03997I 0:1501 OSA: tun0: using MAC address 02-AA-BB-CC-DD-EE
18:01:12 HHC03997I 0:1501 OSA: tun0: using IP address 192.168.100.1
18:01:12 HHC03997I 0:1501 OSA: tun0: using subnet mask 255.255.255.0
18:01:12 HHC03997I 0:1501 OSA: tun0: using MTU 1500
18:01:12 HHC03997I 0:1501 OSA: tun0: using drive MAC address 8e:6d:17:15:57:73
18:01:12 HHC03997I 0:1501 OSA: tun0: using drive IP address fe80::8e6d:17ff:fe15:5773
18:01:12 HHC03805I 0:1501 OSA: tun0: Register guest IP address 192.168.100.150
18:01:19 HHC03997I 0:1501 OSA: tun0: not using MAC address 02-AA-BB-CC-DD-EE
18:01:19 HHC03997I 0:1501 OSA: tun0: not using IP address 192.168.100.1
18:01:19 HHC03997I 0:1501 OSA: tun0: not using subnet mask 255.255.255.0
18:01:19 HHC03997I 0:1501 OSA: tun0: not using MTU 1500
18:01:19 HHC00905I 0:1502 OSA: HSCH recognized for data device
18:01:19 HHC00905I 0:1502 OSA: HSCH completed for data device
18:01:21 HHC00905I 0:1502 OSA: HSCH recognized for data device
18:01:21 HHC00905I 0:1502 OSA: HSCH completed for data device
18:01:21 HHC03800I 0:1501 OSA: Adapter mode set to Layer 3
18:06:46 HHC00905I 0:1502 OSA: HSCH recognized for data device
18:06:46 HHC00905I 0:1502 OSA: HSCH completed for data device
18:07:16 HHC00905I 0:1502 OSA: HSCH recognized for data device
18:07:16 HHC00905I 0:1502 OSA: HSCH completed for data device
18:07:16 HHC03800I 0:1501 OSA: Adapter mode set to Layer 3
18:12:32 HHC00905I 0:1502 OSA: HSCH recognized for data device
18:12:32 HHC00905I 0:1502 OSA: HSCH completed for data device
18:13:02 HHC00905I 0:1502 OSA: HSCH recognized for data device
18:13:02 HHC00905I 0:1502 OSA: HSCH completed for data device
18:13:02 HHC03800I 0:1501 OSA: Adapter mode set to Layer 3
 
Hercules Log Says:
 
1) Initial startup and Guest IP registration (192.168.100.150) complete cleanly.
2) After an extended period of runtime(in this instance 16hrs), the connection is broken by z/OS sending a Halt Subchannel (HSCH) command to the OSA data device (01502).
3) Hercules attempts an automatic recovery every 5 minutes, but the z/OS stack immediately rejects the subchannel reconnection within seconds.
4) I did the IPL and working fine and not sure how long will it run 
 
I have attached the log here.
Has anyone encountered this specific HSCH data device disconnect loop on a Hercules setup before? I am looking for a helping hand to fix the issue.
Thanks,
Venjoa

I was able to reproduce the problem between two Hercules' running on two Linux hosts. The problem appeared to be traffic rate related, the greater the packet rate between the Hercules' the less time before the problem occurred. There was never any indication of a problem on the failing Hercules, the guest would simply send a ULP_DISABLE request to the QETH before issuing the message:

EZZ4338I ERROR REPORTED ON INTERFACE OSAAFACE4 - CODE 80100040 DIAGNOSTIC CODE 03

The code and the diagnostic code were always the same. After issuing the message the guest would attempt to recover. Sometimes after two or three attempts the recovery was successful, sometimes after many attempts the recovery wasn't successful (or at least it wasn't before I lost patience!).

Referring to the Data link control (DLC) status codes the code isn't very helpful:

X'80'   Permanent error
Explanation: Request rejected due to failure of either a system or network function.

X'10'   LLC or DNI layer local error
Explanation: A request was processed and an error was found by the local VTAM.

X'0040' INOP-deact SAP
Explanation: SAP becomes inoperative.

I suspect, though I have no evidence in support, that there is an error in a buffer passed from QETH to the guest. The SNA sense code 8010 is described:

Segmented RU length error: An RU was found to exceed a maximum length, or required buffer allocation
that might cause future buffer depletion.

The use of 8010 in the code may be a coincidence, or not. Investigation continues.

[venjoa]

I really appreciate your help, Mcisho, in creating the new issue. Due to my busy weekday schedule, I wasn't able to create it myself.
Thank you so much for taking the time to help. I truly appreciate it.

Ian suggested keeping the "qeth debug on 1500 updown" command in Hercules. I captured the logs after enabling it and have attached them here for your reference.

MF_31_log.log

The main point highlighted in the logs is the output generated by the "qeth debug on 1500 updown" command. It may be helpful for further debugging by the core team.

02:26:25 HHC03997I 0:1501 OSA: tun0: not using MAC address 02-AA-BB-CC-DD-EE
02:26:25 HHC03997I 0:1501 OSA: tun0: not using IP address 192.168.100.1
02:26:25 HHC03997I 0:1501 OSA: tun0: not using subnet mask 255.255.255.0
02:26:25 HHC03997I 0:1501 OSA: tun0: not using MTU 1500
02:26:25 HHC00905I 0:1502 OSA: HSCH recognized for Data device
02:26:25 HHC03991D 0:1502 OSA: Halt data device: Active true (1)
02:26:25 HHC03991D 0:1502 OSA: Halting data device
02:26:25 HHC03991D 0:1502 OSA: Activate Queues: QDSIG_HALT received
02:26:25 HHC03991D 0:1502 OSA: Activate Queues: Exit
02:26:25 HHC03991D 0:1502 OSA: Data device halted
02:26:25 HHC00905I 0:1502 OSA: HSCH completed for Data device
02:26:27 HHC00905I 0:1502 OSA: HSCH recognized for Data device
02:26:27 HHC03991D 0:1502 OSA: Halt data device: Active false (0)
02:26:27 HHC00905I 0:1502 OSA: HSCH completed for Data device
02:26:27 HHC03995D 0:1502 OSA: SID:
02:26:27 FF173101 173201
02:29:48 HHC01603I exit

[mcisho]

For a long time QETH was described as experimental, a description I believe should still be in place as QETH still has so many unknowns. QETH works very well while it does work, and it usually does work, but when things go wrong we are blind. I still have absolutely no idea what triggers the failure in this case, but I believe I can see design decisions old and new are holding hands to prevent recovery from working.

I can now trigger the failure at will, using Hercules on a Windows virtual machine. Starting an EE connection to or from the Hercules guest running on the Windows vm, the failure occurs after the exchange of a handful of packets, during the EE connectivity check phase of connection establishment. Presumably my environment is contributing to the failure, as I'm sure others have successfully used EE with Hercules guests on a Windows host.

I don't anticipate a quick resolution of this problem. If this QETH problem is causing you problems, perhaps you should switch from a QETH to a PTP or a CTCI interface.

[venjoa]

Thank you for the update and for looking into this issue. I appreciate the detailed explanation. I'll switch to a CTCI interface for now as you suggested. I understand that diagnosing issues like this can be challenging, especially when the root cause isn't yet clear. Thanks again for your efforts.

One question: is CTCI expected to be free from this type of halt/recovery issue affecting QETH or could similar failures occur there as well?

[mcisho]

CTCI is much simpler and, having existed for many years, is much better tested. I would not expect similar failures.

[mcisho]

A few days ago I mentioned that I can now trigger the failure at will, by starting an EE connection to or from a Hercules guest running on a Windows virtual machine. In this particular case the failure is caused by packets not being sent due to a timing hole in the QETH code.

When a QETH is used for EE traffic, EE makes use of the output queues implemented by QETH to action the high, medium and low traffic priorities provided by EE. In this case EE is using queue 0 for high priority, queue 2 for medium priority and queue 3 for low priority.

EE needs to send some high priority traffic so it puts the outbound packets in queue 0 buffers and issues a SIGA write queue 0. QETH springs into life and starts processing the queue 0 buffers, sending the traffic to the data-devices tun interface. While QETH is processing queue 0, EE finds it needs to send some medium priority traffic, so it puts the outbound packets in queue 2 buffers and issues a SIGA write queue 2; EE then finds it needs to send some low priority traffic, so it puts the outbound packets in queue 3 buffers and issues a SIGA write queue 3.

QETH's SIGA write queue processing (and SIGA write multiple queue processing) updates various data-device qdio-related variables; in particular the variable qdio.o_qmask is replaced by the mask supplied by the SIGA.

Addendum: QETH's SIGA write queue processing also modifies variables that are used in controlling the loop that processes queue buffers, so buffers is the queue could be ignored, or maybe even processed twice(?).

QETH now finishes processing queue 0, and checks whether any of its other output queues have traffic waiting to be sent. At this point variable qdio.o_qmask only indicates that queue 3 has traffic waiting, because the queue 0 indication was replaced by the queue 2 indication, which was in turn was replaced by the queue 3 indication. As a result QETH starts processing the queue 3 buffers, completely ignoring the outbound packets in the queue 2 buffers.

Because the outbound packets in the queue 2 buffers were not sent there are no inbound packets in response, which makes EE believe the QETH has a serious problem, and results in the EZZ4338I message and recovery. Which seems a bit extreme as UDP packets could easily get lost traversing the network.

@venjoa Were/are you using EE, or any other IP traffic prioritising?

[mcisho]

Commit e3f54c1 has fixed the EE connection problem on my Windows virtual machine.

@venjoa Can you please try the latest Hercules develop branch on your system, to see whether it fixes your problem?