Skip to content

Alibaba NLB disabled zone, PrivateLink getting blackholed

On Alibaba Cloud PrivateLink, take the NLB out of one zone and the endpoint in the other account can keep sending traffic into that dead zone. The endpoint's DNS record for the zone is not pulled, and the traffic blackholes.

Two objects, and where each lives:

  • NLB and endpoint service, same account. The endpoint service is built on the NLB and follows its zones. Say the NLB runs in zones B and C.
  • Endpoint, a different account. It connects to the endpoint service across zones B and C, and gets one ENI, one IP, per zone. Its default domain resolves to both:
$ nslookup <endpoint-domain>
ep-xxxx.epsrv-xxxx.<region>.privatelink.aliyuncs.com
Address: 10.0.11.11   # zone B endpoint ENI
Address: 10.0.12.6    # zone C endpoint ENI

Now zone B is taken out of the NLB on the endpoint-service account. In the case I saw, a network ACL blocked the zone. The NLB stops answering in zone B.

The endpoint account sees no change. PrivateLink runs managed failover on the endpoint by probing each zone's endpoint ENI IP and pulling that zone's DNS record when the IP goes bad (Alibaba docs, below). But the endpoint ENI in zone B is the endpoint account's own interface, and it is still up. So the zone B record stays. The endpoint still reads Connected. Clients keep resolving both IPs, about half pick zone B, reach a healthy ENI, cross the endpoint service, and die at an NLB that is gone. A blackhole the endpoint account cannot see from the endpoint.

PrivateLink blackhole when an NLB zone is removed The client resolves the endpoint to one IP per zone. Each endpoint ENI connects through the endpoint service to the NLB. Zone C is healthy end to end. On zone B the endpoint ENI and endpoint service are still up, so the zone's DNS record is kept, but the NLB in zone B was removed, so traffic dies before the NLB. ENDPOINT ACCOUNT ENDPOINT-SERVICE ACCOUNT Client 2 zone IPs Endpoint ENI zone C Endpoint service zone C NLB zone C Endpoint ENI zone B (up) Endpoint service zone B (up) NLB zone B removed blackhole

Reconciling the doc with what I saw, the failover signal looks to be the endpoint ENI, not the path from it to the NLB. A backend that disappears behind a still-healthy ENI does not trip the probe.

Avoiding it

The endpoint's own zone list is the only thing that stops the traffic, so drain both sides together, not just the NLB:

  • Take the zone off the endpoint. On the endpoint account, remove the zone from the endpoint (RemoveZoneFromVpcEndpoint, or the Zone and ENI tab). That deletes the zone's ENI and its DNS record, so nothing resolves it.
  • Do not disable a zone with a network ACL. A network ACL kills the path to the NLB but leaves the endpoint service and the endpoint ENI up, which is exactly what keeps the stale DNS record. Remove the zone at the NLB or endpoint-service layer instead, then confirm the endpoint drops it.
  • Health-check the backend where you control it. You cannot attach an active health check to the endpoint itself, its only health signal is the fixed ENI-IP probe. Put the check on the client instead (passive outlier detection, or active checks against the real backend) so it ejects the dead zone IP that PrivateLink keeps advertising.

The zones are coupled going up and have to be drained together going down.

Related: Alibaba PrivateLink zones cascade from the NLB, the endpoint zones ⊆ endpoint-service zones ⊆ NLB zones rule.

Docs: Create and manage endpoints. PrivateLink probes each zone's endpoint ENI IP in real time and removes that zone's DNS record when it detects an anomaly, restoring it on recovery.