Troubleshooting an App-ID Policy Denial

I recently dealed with an incident where an internal application communicating with a payment API server established a connection but the server failed to process each checkout request. The situation appeared to be a problem with the network but it was not a simple connectivity error.

The data moves as follows

• Source: 10.41.18.25 in the production environment
• Destination: 198.51.100.84 on the public internet
• Protocol: TCP/443
• Expected source NAT: 203.0.113.26
• Expected route: the default route that faces the internet

(The addresses above are for documentation and serve as examples.)

The traffic log indicates that the firewall allows traffic and the TCP handshake is successful. I initially thought that the firewall was not the cause of the failure - but the App-ID feature can change how it identifies the application during the data stream after it analyzes a sufficient amount of the payload - This change in identification causes the system to search for a matching security policy again. Because of this, "allow" during the setup of the connection and "allow" at the conclusion of the session are different.

It is assumed that <payment-app-id> represents the specific App-ID that the firewall identifies.

Checking the Live Session

First step is finding the connection:

show session all filter source 10.41.18.25 destination 198.51.100.84 destination-port 443 protocol 6

ID       Application  State     Flag
74126    ssl          DISCARD   NS

The session is present in the table and the NS flag indicates that source NAT is active - but the state is DISCARD, which is the primary issue.

As this information does not explain the cause, let's viewed the session details

show session id 74126

The network-path fields shows:

total byte count(c2s): 2841
total byte count(s2c): 1764
nat-rule: app-egress-snat
ingress interface: ethernet1/6
egress interface: ethernet1/3

The data packets moved in both directions - and the NAT rule and egress interface are consistent with the configuration. It is possible that the payment service is unresponsive but those results indicate that the routing and the return path function correctly.

The fields that describe the discard are located lower in the output

The session does not use the initial ssl classification because the firewall identified the traffic as <payment-app-id> and blocked it during a second policy evaluation.

application: <payment-app-id>
state: DISCARD
rule: outbound-web-initial
tracker stage firewall: appid policy lookup deny
end-reason: policy-deny

Testing the Rulebase

The issue is no longer "is TCP/443 allowed" but "is this specific application allowed."

test security-policy-match application <payment-app-id> source 10.41.18.25 destination 198.51.100.84 destination-port 443 protocol 6

"restricted-payment-apps" {

  application <payment-app-id>;

  action deny;

}

The connection has now been reclassified, App-ID picks up the change, and a deny rule has been applied.

Packet Capture

For individual packets analysis, I configured a specific filter within the Monitor → Packet Capture menu. The settings must include the source address, the destination address, the TCP protocol and port 443. Using the receive and transmit stages I can verify that the device forwarded the packet. To determine if the device discarded subsequent packets during the processing of policies, I examine the firewall and drop stages.

debug dataplane packet diag set capture on

The user sends a single controlled test request and then deactivates the function right away.

debug dataplane packet diag set capture off

It is necessary to configure the filters before creating the test traffic. An existing connection does not always receive a capture that an administrator applies after the connection starts.

Implementing the Fix

The most common error involves allowing TCP/443 for every application.

It is more secure to create a security rule with a specific scope

• The source zone is app prod
• The source address is 10.41.18.25
• The destination zone is the internet
• The destination address is 198.51.100.84
• The application is <payment-app-id>
• The rule includes only the application dependencies that the software requires
• The service setting is accurate

To ensure accuracy, compare application dependencies and service recommendations with the current App-ID documentation and the installed content version, as those values change.

By applying this change, the policy test selected the correct rule

"payment-api-egress" {

application <payment-app-id>;

action allow;

}

The test confirms that the policy identifies the intended result - but the test does not confirm that the checkout process functions. This requires a live session to verify the connection.

Verifying With a Fresh Session

Let's remove the expired session so that the firewall evaluates the subsequent request independently.

clear session id 74126

On the command line, I verify the session tuple and the timestamp before I execute the command. The firewall reuses session IDs frequently, which creates a risk of clearing an incorrect entry. I avoid the command to clear all sessions because it disrupts unrelated traffic.

A new attempt to complete a purchase generated session 75291.

show session id 75291

state - ACTIVE

application: <payment-app-id>

rule - payment-api-egress

nat-rule - app-egress-snat

total byte count(c2s): 18472

total byte count(s2c): 23109

The session uses the intended application, security rule & NAT configuration. Both byte counters increase steadily. There are no indicators of traffic rejection present in this entry. The final verification confirms that the purchase reached completion.

Three things worth remembering:

• An initial permit action is not a guarantee that the firewall will sustain the session until completion.
• Test the security policy against the specific App-ID that the firewall identifies instead of relying on the port number.
• Apply the minor adjustment required and then confirm the result to observe an entirely new session.