Securing the Gate with Security Profiles Series: Application Control

1. Introduction and Concepts

Application Control is a FortiGate security feature that identifies and governs network traffic based on the application generating it, rather than relying on port and protocol alone. It uses the FortiGuard Application Control signature database together with deep inspection of packet payloads to recognize thousands of applications, including those that deliberately use evasive techniques such as port hopping, encryption, and tunneling inside common ports like TCP 80 and 443.

Traditional firewall rules assume that a service is bound to a known port. Modern applications break this assumption. A messaging app, a file-sharing service, and a video conferencing tool may all traverse TCP 443. Application Control inspects the traffic flow, matches it against signatures, and lets you allow, monitor, block, or shape it independently of the port in use.

1.1 How Application Identification Works

FortiGate identifies applications through several complementary mechanisms that are applied in sequence as a session develops:

– Pattern and signature matching against the FortiGuard application database, which inspects payload bytes for known application fingerprints.
– Protocol decoders that understand the structure of common protocols (HTTP, DNS, TLS, SSH, SIP) and extract identifying metadata.
– Behavioral and heuristic analysis for traffic that does not match a precise signature but exhibits recognizable patterns.
– Cloud-assisted identification through FortiGuard for newer or rapidly changing cloud applications when the relevant service is licensed.

Because identification often requires several packets, the firewall may permit the first few packets of a flow before a verdict is reached. Once the application is identified, the configured action is enforced for the remainder of the session. This is normal and expected behavior, and it is the reason a small amount of traffic from a blocked application can appear in logs before the block takes effect.

1.2 Categories, Applications, and Signatures

The application database is organized into a three-level hierarchy. Understanding this hierarchy is essential to building maintainable profiles.

Level	Description	Example
Category	A broad functional grouping of applications. Actions can be applied to an entire category at once.	Video/Audio, Social.Media, P2P, Proxy
Application	A specific named application within a category.	YouTube, BitTorrent, Telegram
Signature	An individual pattern that detects a particular behavior or sub-function of an application.	YouTube_Video.Play, Facebook_Chat

Note: Acting at the category level keeps profiles concise and resilient as new applications are added to FortiGuard, because new members of a category automatically inherit the category action. Override individual applications only where you need behavior that differs from their category.

1.3 Where Application Control Fits in the Inspection Path

Application Control runs as part of the FortiGate security profile inspection chain. Traffic first matches a firewall policy. If that policy references an Application Control profile, the traffic is then evaluated by the application engine. Application Control is frequently paired with SSL/SSH inspection, because the majority of modern traffic is encrypted and cannot be classified beyond the TLS handshake without decryption.

Important: Without deep SSL inspection, FortiGate can identify many applications only from unencrypted handshake metadata such as the TLS Server Name Indication (SNI). Granular control of sub-functions within an encrypted application usually requires a deep-inspection SSL profile. Plan certificate distribution before relying on this.

2. Prerequisites and Planning

2.1 Licensing and Database

Application Control depends on an up-to-date FortiGuard application signature database. Confirm the following before deployment:

– A valid FortiGuard subscription that includes Application Control signature updates. Without it, the database becomes stale and identification accuracy degrades over time.
– Scheduled automatic updates so that new and revised signatures are pulled regularly.
– Connectivity from the FortiGate to FortiGuard servers, or to a FortiManager acting as a local FortiGuard distribution server in isolated networks.

Verify the installed database version from the CLI:

get system fortiguard-service status
diagnose autoupdate versions

2.2 Inspection Mode

FortiOS supports two inspection modes, set per VDOM or globally, that affect how security profiles including Application Control behave:

Mode	Behavior	When to use
Flow-based	Inspects packets as they pass through without full buffering. Lower latency and higher throughput. Default in recent FortiOS releases.	Most deployments, especially where performance is a priority.
Proxy-based	Buffers and reconstructs the full transaction before inspection. Enables some advanced features at the cost of latency.	Where features that require full reconstruction are needed.

Check and set the inspection mode:

config system settings
   set inspection-mode flow
end

2.3 Policy Mode: NGFW Profile-based vs Policy-based

When operating in flow mode, FortiGate offers two NGFW models that determine how applications are referenced in policy:

Profile-based: Application Control is configured as a reusable security profile and attached to firewall policies. This is the classic model and is recommended for most environments because profiles are portable and auditable.
Policy-based: Applications and URL categories are selected directly as match criteria inside the firewall policy itself, alongside source, destination, and service. This allows applications to be part of the matching logic rather than a post-match profile.

config system settings
set ngfw-mode profile-based
end

Caution: Switching NGFW mode is disruptive. It can invalidate existing policy references and should only be done during a maintenance window with a tested configuration backup in hand.

2.4 Pre-Deployment Checklist

1. Take a full configuration backup and record the current firmware and database versions.
2. Confirm FortiGuard licensing and successful signature updates.
3. Decide inspection mode and NGFW mode and confirm they match your design.
4. Inventory the applications your organization sanctions, tolerates, and prohibits.
5. Plan an SSL inspection strategy and certificate deployment if granular control is required.
6. Identify a pilot scope (a test VLAN or a small user group) for monitor-first rollout.

3. Building an Application Control Profile

This section walks through creating a profile from first principles, beginning with a monitor-only posture and progressively tightening it. The monitor-first approach is the single most important practice for a low-risk deployment: it lets you observe real traffic and identify business-critical applications before anything is blocked.

3.1 Creating a Baseline Monitor Profile (GUI)

1. Navigate to Security Profiles > Application Control.
2. Select Create New and give the profile a descriptive name, for example AppCtrl-Monitor-Baseline.
3. Set the action for all categories to Monitor. This logs application traffic without blocking it.
4. Enable logging so that identified applications appear in the application log and FortiView.
5. Leave Quarantine and traffic shaping unset for the baseline.
6. Save the profile and attach it to a firewall policy covering your pilot scope.

Best practice: Run the monitor baseline for one to two full business weeks. This captures weekly cycles such as payroll runs, reporting jobs, and weekend maintenance that a shorter window would miss.

3.2 Creating the Same Profile (CLI)

The equivalent profile in the CLI, setting all categories to monitor and enabling logging, is shown below. Category numbers are stable identifiers used internally by FortiOS.

config application list
    edit "AppCtrl-Monitor-Baseline"
      set comment "Baseline monitor-only profile"
      set other-application-action pass
      set unknown-application-action pass
      set app-replacemsg enable
config entries
     edit 1
       set action pass
       set log enable
     next
   end
  next
end

In this baseline, a single catch-all entry passes and logs everything. As you move toward enforcement you will add more specific entries above the catch-all, because entries are evaluated top to bottom and the first match wins.

3.3 Understanding Profile Actions

Action	Effect on matching traffic
Allow / Pass	Permits the application and, when logging is enabled, records it. Used for sanctioned applications.
Monitor	Permits the application but always logs it. Used during the observation phase and for visibility into tolerated apps.
Block	Resets or drops the session for the matched application and logs the event. Used for prohibited applications.
Quarantine	Blocks the application and additionally quarantines the source endpoint for a configured duration.
Traffic shaping	Permits the application but applies a bandwidth guarantee or limit through an associated shaper.

3.4 Category and Application Overrides

Override entries let a profile treat a specific application or signature differently from its parent category. A common pattern is to block an entire risky category while explicitly allowing one sanctioned member. Because the first matching entry wins, the more specific allow entry must sit above the broad category block.

config application list
    edit "AppCtrl-Corp"
    config entries
       edit 1
          set application 15832
          set action pass
          set log enable
       next
       edit 2
          set category 23
          set action block
          set log enable
       next
     end
   next
end

In this example, entry 1 explicitly allows a single application by its signature ID while entry 2 blocks the rest of its category. To find the numeric identifiers for an application or category, search the FortiGuard application encyclopedia or list them from the CLI.

diagnose application-control list category
diagnose application-control list application | grep -i <name>

4. SSL Inspection for Encrypted Applications

Because most application traffic is encrypted, the depth of your SSL inspection directly determines how granular your application control can be. There are two relevant SSL inspection profiles.

SSL profile	What FortiGate can see	Application Control capability
Certificate inspection	TLS handshake metadata only: SNI, certificate subject, issuer. Payload stays encrypted.	Identifies many applications by hostname. Cannot distinguish in-app sub-functions.
Deep inspection (full SSL)	The decrypted payload, by acting as a trusted man-in-the-middle using a re-signing CA.	Full granularity, including sub-functions such as chat versus file transfer within one app.

Deep inspection requires the FortiGate re-signing CA certificate to be trusted by every client, otherwise users see certificate warnings. Distribute the CA through your endpoint management platform before enabling deep inspection broadly.

Privacy and exemptions: Decrypting traffic has privacy, legal, and operational implications. Always exempt categories that break under inspection or carry sensitive data, such as health, finance, and government services, using SSL exemption lists. Some applications also use certificate pinning and will fail under deep inspection unless exempted.

4.1 Attaching SSL Inspection to a Policy

The Application Control profile and the SSL inspection profile are attached to the same firewall policy. The SSL profile prepares the traffic so that the application engine can classify it.

config firewall policy
    edit 12
       set name "Corp-Outbound-AppCtrl"
       set srcintf "internal"
       set dstintf "wan1"
       set srcaddr "Corp-LAN"
       set dstaddr "all"
       set action accept
       set schedule "always"
       set service "ALL"
       set utm-status enable
       set application-list "AppCtrl-Corp"
       set ssl-ssh-profile "deep-inspection"
       set nat enable #If you are using policy-NAT
       set logtraffic all
    next
end

5. Applying Application Control to Firewall Policies

5.1 Profile-based Attachment

In profile-based NGFW mode, set utm-status to enable on the policy and reference the application list. Ensure logtraffic is set to all during rollout so that allowed and monitored applications are recorded, not only blocked ones.

5.2 Policy-based Application Matching

In policy-based NGFW mode, applications become part of the match criteria of the security policy. The application and the SSL profile are referenced directly, and a central SSL inspection profile applies.

config firewall security-policy
     edit 5
      set name "Block-P2P"
     set srcintf "internal"
     set dstintf "wan1"
     set srcaddr "all"
     set dstaddr "all"
     set application 788 789 790
     set action deny
     set logtraffic all
   next
end

5.3 Policy Ordering

Firewall policies are evaluated top to bottom. An Application Control profile only ever sees traffic that the policy it is attached to actually matches. A broad allow policy placed above a more specific control policy will shadow it, so the control policy never receives traffic. Always position more specific policies above broader ones, and use the policy hit counters to confirm that traffic lands where you expect.

6. Advanced Configuration

6.1 Traffic Shaping per Application

Rather than blocking a tolerated but bandwidth-heavy application outright, you can constrain it. Associate a traffic shaper with an application entry to guarantee or cap its bandwidth, which is useful for streaming and software-update traffic that should not starve business applications.

config firewall shaper traffic-shaper
   edit "limit-streaming"
      set maximum-bandwidth 5000
      set per-policy enable
    next
end

6.2 Blocking Application Sub-functions

Many applications expose distinct behaviors that map to separate signatures. With deep SSL inspection in place, you can permit an application while blocking specific actions within it, for example allowing a collaboration suite but blocking its file-transfer signature. Identify the sub-function signatures in the FortiGuard encyclopedia and add them as higher-priority block entries above the application allow entry.

6.3 Handling Unknown and Other Applications

Two catch-all settings govern traffic that is not matched by a specific entry:

unknown-application-action: applies to traffic the engine cannot classify at all. Keep this at pass during early rollout to avoid breaking legitimate but unrecognized traffic, then tighten cautiously.
other-application-action: applies to identified applications that have no explicit entry in the profile. Setting this to block creates an allow-list posture where only explicitly permitted applications pass.

Warning: Moving to an allow-list posture by blocking other-application-action is powerful but high-risk. Validate it extensively in monitor mode first, because it will block any application you have not anticipated, including legitimate internal tools.

6.4 Deep Application Inspection and Replacement Messages

Enable application replacement messages so that blocked web applications return an informative page to the user rather than a silent connection reset. This reduces help-desk tickets by telling users why access was denied and whom to contact. Enable deep-app-inspection where supported to improve classification of layered application traffic.

7. Validation and Testing

After attaching a profile, validate that traffic is being identified and the configured actions are enforced. Use a combination of GUI views and CLI diagnostics.

7.1 GUI Verification

– FortiView > Applications shows real-time and historical application usage by user, source, and bandwidth.
– Log & Report > Application Control lists individual application events with the matched action.
– The policy list hit counters confirm that traffic is matching the intended policy.

7.2 CLI Diagnostics

Confirm the engine is running and inspect live classification with the following commands:

diagnose test application ipsmonitor 1
get ips session
diagnose application-control status
diagnose debug application appctrl -1
diagnose debug enable

Remember to disable debugging once finished, because verbose debug output consumes resources on a production device:

diagnose debug disable
diagnose debug reset

7.3 Functional Testing

1. From a client in the pilot scope, generate traffic for a known-allowed application and confirm it appears in logs with the allow action.
2. Generate traffic for a known-blocked application and confirm the session is blocked and logged.
3. Verify that an SSL-inspected application is classified at the expected granularity and that no certificate warnings appear on managed clients.
4. Confirm that exempted sensitive categories are not being decrypted.

8. Troubleshooting

Symptom	Likely cause	Resolution
Application not identified or shown as unknown	Encrypted traffic without deep inspection, or stale signature database.	Enable deep SSL inspection for the flow and update the FortiGuard database.
Block not enforced; first packets pass	Identification requires several packets before a verdict.	Expected behavior. The session is blocked once classified. Confirm with logs.
Certificate warnings on clients	Re-signing CA not trusted by endpoints under deep inspection.	Distribute and trust the FortiGate CA via endpoint management.
Legitimate app broken after enabling control	App uses certificate pinning or matched a broad block entry.	Add an SSL exemption or a higher-priority allow entry for the app.
Profile changes have no effect	Policy shadowing by a broader policy above, or profile not attached.	Check policy order and hit counters; confirm utm-status and the profile reference.
High CPU after enabling deep inspection	Decryption is resource intensive across all flows.	Scope deep inspection to required policies and exempt high-volume trusted traffic.

When a specific application behaves unexpectedly, trace a single session end to end. Filter the application debug to the source address of your test client to keep the output readable, reproduce the traffic, and read the classification verdict from the debug stream.

9. Operational Best Practices

– Always begin with monitor mode and observe real traffic for one to two weeks before enforcing blocks.
– Prefer category-level actions and reserve application and signature overrides for genuine exceptions, to keep profiles maintainable as the database grows.
– Order profile entries from most specific to least specific, because the first match wins.
– Keep the FortiGuard database on an automatic update schedule and monitor for failed updates.
– Scope deep SSL inspection deliberately and maintain a documented exemption list for sensitive and pinned traffic.
– Use descriptive, convention-based names for profiles and policies so intent is clear during audits.
– Document the business justification for each block and allow override so future administrators understand the rationale.
– Review application logs and FortiView regularly to catch new applications and shifting usage patterns.
– Test every change in a pilot scope and keep a tested configuration backup before each significant change.
– Coordinate enforcement changes with stakeholders and provide a clear path for users to request exceptions.