Skip to content

[Fortinet Fortiproxy] URL Field Parsing Errors #19603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
robester0403 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[Fortinet Fortiproxy] URL Field Parsing Errors #19603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d4d18a0
fix: reconstruct full URL before uri_parts to prevent pipeline failur…
robester0403 Jun 17, 2026
923bd46
Chore: Update pr link url
robester0403 Jun 17, 2026
ad01b33
fix: corrected pr link
robester0403 Jun 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/fortinet_fortiproxy/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.4.2"
changes:
- description: Fix URI parsing failures when url field contains only a path or query string.
type: bugfix
link: https://github.com/elastic/integrations/pull/19603
- version: "1.4.1"
changes:
- description: Remove top level note from docs
Expand Down
4 changes: 3 additions & 1 deletion packages/fortinet_fortiproxy/data_stream/log/_dev/test/pipeline/test-example.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,6 @@ date=2017-11-15 time=11:44:16 tz="+0200" logid="0000000013" type="traffic" subty
<189>date=2024-05-09 time=06:20:04 devname="TEST-PXY01" devid="FPXTESTPXY01" eventtime=1715260803895122957 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="root" srcip=10.0.0.3 srcport=41460 srcintf="port2" srcintfrole="lan" dstcountry="United States" srccountry="Reserved" dstip=67.43.156.171 dstport=443 dstintf="port1" dstintfrole="undefined" sessionid=1781818021 service="HTTPS" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="27b09930-033d-51ef-0c72-6c1221a8d893" policyname="test-proxy" trandisp="snat" transip=10.0.128.2 transport=7242 clientip=10.0.0.3 duration=12536 wanin=3665 rcvdbyte=3665 wanout=667 lanin=755 sentbyte=755 lanout=3737 appcat="unscanned" utmaction="allow" countssl=1
<189>date=2024-05-09 time=06:21:14 devname="TEST-PXY01" devid="FPXTESTPXY01" eventtime=1715260873739449705 tz="-0700" logid="0100040704" type="event" subtype="system" level="notice" vd="root" logdesc="System performance statistics" action="perf-stats" cpu=0 mem=29 totalsession=38 disk=0 bandwidth="20/20" setuprate=1 disklograte=0 fazlograte=0 freediskstorage=0 sysuptime=166235 waninfo="N/A" msg="Performance statistics: average CPU: 0, memory: 29, concurrent sessions: 38, setup-rate: 1"
<189>date=2024-05-09 time=06:19:39 devname="TEST-PXY01" devid="FPXTESTPXY01" eventtime=1715260778798356673 tz="-0700" logid="0000000010" type="traffic" subtype="forward" level="notice" vd="root" srcip=10.0.0.3 srcport=47886 srcintf="port2" srcintfrole="lan" dstcountry="United States" srccountry="Reserved" dstip=67.43.156.10 dstport=443 dstintf="port1" dstintfrole="undefined" sessionid=1781818019 service="HTTPS" proxyapptype="web-proxy" proto=6 action="accept" policyid=1 policytype="proxy-policy" poluuid="27b09930-033d-51ef-0c72-6c1221a8d893" policyname="test-proxy" trandisp="snat" transip=10.0.128.2 transport=53184 clientip=10.0.0.3 duration=8089 wanin=125800732 rcvdbyte=125800732 wanout=632 lanin=798 sentbyte=798 lanout=125824455 appcat="unscanned" utmaction="allow"
<189>logver=704080649 timestamp=1760084266 devname="TEST-PXY01" devid="FPXTESTPXY01" vd="root" date=2025-10-10 time=08:17:46 eventtime=1760077067153677744 tz="+0200" logid="0000000010" type="event" subtype="user" level="notice" logdesc="Explicit proxy authentication failed" srcip=10.0.0.175 dstip=10.0.0.199 authid="999-WGS-AUTH-DEFAULT" user=""http" authproto="HTTP(10.0.0.175)" action="NTLM-auth" status="failure" url="http://10.0.0.199/" reason="Authentication failed" msg="User "http failed in authentication"
<189>logver=704080649 timestamp=1760084266 devname="TEST-PXY01" devid="FPXTESTPXY01" vd="root" date=2025-10-10 time=08:17:46 eventtime=1760077067153677744 tz="+0200" logid="0000000010" type="event" subtype="user" level="notice" logdesc="Explicit proxy authentication failed" srcip=10.0.0.175 dstip=10.0.0.199 authid="999-WGS-AUTH-DEFAULT" user=""http" authproto="HTTP(10.0.0.175)" action="NTLM-auth" status="failure" url="http://10.0.0.199/" reason="Authentication failed" msg="User "http failed in authentication"
<190>date=2025-09-01 time=12:37:12 devname="testdevice" devid="FPX412345678" eventtime=1756730232283394873 tz="+0000" logid="1059028704" type="utm" subtype="app-ctrl" eventtype="signature" level="information" vd="root" appid=15893 srcip=10.10.10.1 srccountry="Reserved" dstip=10.0.0.100 dstcountry="Reserved" srcport=64986 dstport=443 srcintf="port2" srcintfrole="lan" dstintf="port1" dstintfrole="wan" proto=6 service="HTTPS" direction="outbound" policyid=5 poluuid="f742ad16-b894-51ef-1128-ff401f6ee4ef" policytype="policy" sessionid=1516513751 applist="app_Default" action="pass" appcat="Web.Client" app="HTTP.BROWSER" hostname="fonts.googleapis.com" incidentserialno=628964575 url="/css?family=Roboto:300,400,500,700|Google+Sans:400,500,700|Google+Sans+Text:400,500,700&lang=de" msg="Web.Client: HTTP.BROWSER" apprisk="medium" rawdataid="1/1" rawdata="Method=GET|User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36|Referer=https://something.com/"
<189>date=2024-06-07 time=09:33:55 devname="TEST-PXY01" devid="FPXTESTPXY01" eventtime=1717770835912000000 tz="-0600" logid="0010000099" type="traffic" subtype="http-transaction" level="notice" vd="root" srcip=10.10.0.2 dstip=10.0.0.101 clientip=10.10.0.2 scheme="https" srcport=57784 dstport=443 hostname="example.com" url="/search?q=test" prefetch=0 policyid=1 sessionid=370197374 transid=139874927 reqlength=120 resplength=0 rcvdbyte=0 sentbyte=120 resptype="normal" httpmethod="GET" agent="curl/7.68.0" statuscode="200" rawdata="Time=10ms|Header-Host=example.com" reqtime=1717770835 resptime=1717770835 respfinishtime=1717770835 duration=10 appcat="Web.Client"
216 changes: 216 additions & 0 deletions ...ges/fortinet_fortiproxy/data_stream/log/_dev/test/pipeline/test-example.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,7 @@
},
"url": {
"domain": "google.com",
"full": "https://google.com/",
"original": "https://google.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -916,6 +917,7 @@
},
"url": {
"domain": "steampowered.com",
"full": "https://steampowered.com/",
"original": "https://steampowered.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -1050,6 +1052,7 @@
},
"url": {
"domain": "github.com",
"full": "https://github.com/",
"original": "https://github.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -1195,6 +1198,7 @@
},
"url": {
"domain": "google.com",
"full": "https://google.com/",
"original": "https://google.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -1321,6 +1325,7 @@
},
"url": {
"domain": "google.com",
"full": "https://google.com/",
"original": "https://google.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -1447,6 +1452,7 @@
},
"url": {
"domain": "google.com",
"full": "https://google.com/",
"original": "https://google.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -1573,6 +1579,7 @@
},
"url": {
"domain": "adobe.com",
"full": "https://adobe.com/",
"original": "https://adobe.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -1699,6 +1706,7 @@
},
"url": {
"domain": "www.adobe.com",
"full": "https://www.adobe.com/",
"original": "https://www.adobe.com/",
"path": "/",
"scheme": "https"
Expand Down Expand Up @@ -2886,10 +2894,218 @@
},
"url": {
"domain": "10.0.0.199",
"full": "http://10.0.0.199/",
"original": "http://10.0.0.199/",
"path": "/",
"scheme": "http"
}
},
{
"@timestamp": "2025-09-01T12:37:12.000Z",
"client": {
"ip": "10.10.10.1",
"port": 64986
},
"destination": {
"ip": "10.0.0.100",
"port": 443
},
"ecs": {
"version": "8.17.0"
},
"event": {
"action": "pass",
"category": [
"network"
],
"code": "1059028704",
"kind": "event",
"original": "<190>date=2025-09-01 time=12:37:12 devname=\"testdevice\" devid=\"FPX412345678\" eventtime=1756730232283394873 tz=\"+0000\" logid=\"1059028704\" type=\"utm\" subtype=\"app-ctrl\" eventtype=\"signature\" level=\"information\" vd=\"root\" appid=15893 srcip=10.10.10.1 srccountry=\"Reserved\" dstip=10.0.0.100 dstcountry=\"Reserved\" srcport=64986 dstport=443 srcintf=\"port2\" srcintfrole=\"lan\" dstintf=\"port1\" dstintfrole=\"wan\" proto=6 service=\"HTTPS\" direction=\"outbound\" policyid=5 poluuid=\"f742ad16-b894-51ef-1128-ff401f6ee4ef\" policytype=\"policy\" sessionid=1516513751 applist=\"app_Default\" action=\"pass\" appcat=\"Web.Client\" app=\"HTTP.BROWSER\" hostname=\"fonts.googleapis.com\" incidentserialno=628964575 url=\"/css?family=Roboto:300,400,500,700|Google+Sans:400,500,700|Google+Sans+Text:400,500,700&lang=de\" msg=\"Web.Client: HTTP.BROWSER\" apprisk=\"medium\" rawdataid=\"1/1\" rawdata=\"Method=GET|User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36|Referer=https://something.com/\"",
"start": "2025-09-01T12:37:12.283Z",
"timezone": "+0000"
},
"fortinet": {
"proxy": {
"app": "HTTP.BROWSER",
"appid": "15893",
"applist": "app_Default",
"apprisk": "medium",
"dstintfrole": "wan",
"eventtype": "signature",
"incidentserialno": 628964575,
"rawdata": "Method=GET|User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36|Referer=https://something.com/",
"rawdataid": "1/1",
"sessionid": "1516513751",
"srcintfrole": "lan",
"subtype": "app-ctrl",
"type": "utm",
"url": "/css?family=Roboto:300,400,500,700|Google+Sans:400,500,700|Google+Sans+Text:400,500,700&lang=de",
"vd": "root"
}
},
"log": {
"level": "information",
"syslog": {
"facility": {
"code": 23
},
"priority": 190,
"severity": {
"code": 6
}
}
},
"message": "Web.Client: HTTP.BROWSER",
"network": {
"direction": "outbound",
"iana_number": "6",
"protocol": "https",
"transport": "tcp"
},
"observer": {
"egress": {
"interface": {
"name": "port1"
}
},
"ingress": {
"interface": {
"name": "port2"
}
},
"name": "testdevice",
"product": "FortiProxy",
"serial_number": "FPX412345678",
"type": "proxy",
"vendor": "Fortinet"
},
"rule": {
"category": "Web-Client",
"id": "5",
"ruleset": "policy",
"uuid": "f742ad16-b894-51ef-1128-ff401f6ee4ef"
},
"server": {
"ip": "10.0.0.100",
"port": 443
},
"source": {
"ip": "10.10.10.1",
"port": 64986
},
"url": {
"domain": "fonts.googleapis.com",
"full": "https://fonts.googleapis.com/css?family=Roboto:300,400,500,700|Google+Sans:400,500,700|Google+Sans+Text:400,500,700&lang=de",
"original": "/css?family=Roboto:300,400,500,700|Google+Sans:400,500,700|Google+Sans+Text:400,500,700&lang=de",
"path": "/css",
"query": "family=Roboto:300,400,500,700|Google+Sans:400,500,700|Google+Sans+Text:400,500,700&lang=de",
"scheme": "https"
}
},
{
"@timestamp": "2024-06-07T15:33:55.000Z",
"client": {
"bytes": 120,
"ip": "10.10.0.2",
"port": 57784
},
"destination": {
"bytes": 0,
"ip": "10.0.0.101",
"port": 443
},
"ecs": {
"version": "8.17.0"
},
"event": {
"category": [
"network"
],
"code": "0010000099",
"duration": 10000000000,
"kind": "event",
"original": "<189>date=2024-06-07 time=09:33:55 devname=\"TEST-PXY01\" devid=\"FPXTESTPXY01\" eventtime=1717770835912000000 tz=\"-0600\" logid=\"0010000099\" type=\"traffic\" subtype=\"http-transaction\" level=\"notice\" vd=\"root\" srcip=10.10.0.2 dstip=10.0.0.101 clientip=10.10.0.2 scheme=\"https\" srcport=57784 dstport=443 hostname=\"example.com\" url=\"/search?q=test\" prefetch=0 policyid=1 sessionid=370197374 transid=139874927 reqlength=120 resplength=0 rcvdbyte=0 sentbyte=120 resptype=\"normal\" httpmethod=\"GET\" agent=\"curl/7.68.0\" statuscode=\"200\" rawdata=\"Time=10ms|Header-Host=example.com\" reqtime=1717770835 resptime=1717770835 respfinishtime=1717770835 duration=10 appcat=\"Web.Client\"",
"start": "2024-06-07T14:33:55.912Z",
"timezone": "-0600"
},
"fortinet": {
"proxy": {
"prefetch": 0,
"rawdata": "Time=10ms|Header-Host=example.com",
"reqtime": 1717770835,
"respfinishtime": 1717770835,
"resptime": 1717770835,
"resptype": "normal",
"sessionid": "370197374",
"subtype": "http-transaction",
"transid": "139874927",
"type": "traffic",
"url": "/search?q=test",
"vd": "root"
}
},
"http": {
"request": {
"bytes": 120,
"method": "GET"
},
"response": {
"bytes": 0,
"status_code": 200
}
},
"log": {
"level": "notice",
"syslog": {
"facility": {
"code": 23
},
"priority": 189,
"severity": {
"code": 5
}
}
},
"network": {
"bytes": 120
},
"observer": {
"name": "TEST-PXY01",
"product": "FortiProxy",
"serial_number": "FPXTESTPXY01",
"type": "proxy",
"vendor": "Fortinet"
},
"rule": {
"category": "Web-Client",
"id": "1"
},
"server": {
"bytes": 0,
"ip": "10.0.0.101",
"port": 443
},
"source": {
"bytes": 120,
"ip": "10.10.0.2",
"port": 57784
},
"url": {
"domain": "example.com",
"full": "https://example.com/search?q=test",
"original": "/search?q=test",
"path": "/search",
"query": "q=test",
"scheme": "https"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "curl",
"original": "curl/7.68.0",
"version": "7.68.0"
}
}
]
}
50 changes: 47 additions & 3 deletions packages/fortinet_fortiproxy/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -576,11 +576,55 @@ processors:
field: client.ip
if: ctx._fields_.clientip != null

- script:

@ilyannn ilyannn Jun 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the Painless script is necessary as you can also use the set processors to reconstruct the URL:

- set: temp.scheme <- url.scheme / network.protocol
- set: temp.url <- fields.url        # if already http(s)://
- set: temp.url <- {{scheme}}://{{url.domain}}{{fields.url}}
...

tag: script_normalize_url
lang: painless
source: >-
String raw = ctx._fields_?.url;
if (raw == null) { return; }
raw = raw.trim();
if (raw.length() == 0) { return; }
if (ctx._temp_ == null) { ctx._temp_ = [:]; }
String lc = raw.toLowerCase();
if (lc.startsWith('http://') || lc.startsWith('https://')) {
ctx._temp_.url = raw;
return;
}
String scheme;
if (ctx.url?.scheme != null) { scheme = ctx.url.scheme.toString().toLowerCase(); }
else if (['http','https'].contains(ctx.network?.protocol)) { scheme = ctx.network.protocol; }
else if (ctx.server?.port instanceof Number && ctx.server.port == 443) { scheme = 'https'; }

@ilyannn ilyannn Jun 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wary of special-casing 443 as HTTPS may very well be using a different port. Note there are no instances of ctx.server.port == 443 string in the current integrations codebase.

Perhaps the scheme will in fact be always present in these cases so we don't need to guess?

else { scheme = 'http'; }
String host = ctx.url?.domain;
if (raw.startsWith('//')) {
ctx._temp_.url = scheme + ':' + raw;
} else if (raw.startsWith('/')) {
ctx._temp_.url = (host != null && host.length() > 0) ? (scheme + '://' + host + raw) : raw;
} else {
ctx._temp_.url = scheme + '://' + raw;
}
- uri_parts:
tag: process_url
field: _fields_.url
keep_original: true
ignore_missing: true
field: _temp_.url
target_field: url
keep_original: false
ignore_missing: true
on_failure:
- set:
tag: set_url_original_on_fail
field: url.original
copy_from: _fields_.url
- set:
tag: set_url_original
field: url.original
copy_from: _fields_.url
ignore_empty_value: true
- set:
tag: set_url_full
field: url.full
copy_from: _temp_.url
ignore_empty_value: true
if: ctx.url?.domain != null

# ------------------------------------------------------------------------------
# Cleanup.
Expand Down
2 changes: 1 addition & 1 deletion packages/fortinet_fortiproxy/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.1.3
name: fortinet_fortiproxy
title: "Fortinet FortiProxy"
version: "1.4.1"
version: "1.4.2"
description: "Collect logs from Fortinet FortiProxy with Elastic Agent."
type: integration
categories:
Expand Down
Loading