Cache Key Manipulation Plugin¶
Description¶
This plugin allows some common cache key manipulations based on various HTTP request components. It can
- sort query parameters to prevent query parameter reordering being a cache miss
- ignore specific query parameters from the cache key by name or regular expression
- ignore all query parameters from the cache key
- only use specific query parameters in the cache key by name or regular expression
- include headers or cookies by name
- capture values from the
User-Agent
header. - classify request using
User-Agent
and a list of regular expressions - capture and replace strings from the URI and include them in the cache key
- do more - please find more examples below.
Detailed examples and troubleshooting¶
| hierarchical part query
HTTP request | ┌────────────────────────────────┴─────────────────────────────────────────┐┌────┴─────┐
components | URI host and port HTTP headers and cookies URI path URI query
| ┌────────┴────────┐┌────────────────┴─────────────────────────┐┌─────┴─────┐┌────┴─────┐
Sample 1 | /www.example.com/80/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&c=3
Sample 2 | /nice_custom_prefix/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&c=3
| └────────┬────────┘└───┬──┘└─────┬────┘└────┬─────┘└─────┬────┘└─────┬─────┘└────┬─────┘
Cache Key | host:port or UA-class UA-captures headers cookies path query
components | custom prefix replacement
The following is an example of how the above sample keys were generated (Sample 1
and Sample 2
).
Traffic server configuration
$ cat etc/trafficserver/remap.config
map http://www.example.com http://www.origin.com \
@plugin=cachekey.so \
@pparam=--ua-whitelist=popular:popular_agents.config \
@pparam=--ua-capture=(Mozilla\/[^\s]*).* \
@pparam=--include-headers=H1,H2 \
@pparam=--include-cookies=C1,C2 \
@pparam=--include-params=a,b,c \
@pparam=--sort-params=true
$ cat etc/trafficserver/popular_agents.config
^Mozilla.*
^Twitter.*
^Facebo.*
$ cat etc/trafficserver/plugin.config
xdebug.so
HTTP request
$ curl 'http://www.example.com/path/to/data?c=3&a=1&b=2&x=1&y=2&z=3' \
-v -x 127.0.0.1:8080 -o /dev/null -s \
-H "H1: v1" \
-H "H2: v2" \
-H "Cookie: C1=v1; C2=v2" \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A' \
-H 'X-Debug: X-Cache-Key'
* About to connect() to proxy 127.0.0.1 port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET http://www.example.com/path/to/data?c=3&a=1&b=2&x=1&y=2&z=3 HTTP/1.1
> Host: www.example.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> H1: v1
> H2: v2
> Cookie: C1=v1; C2=v2
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A
> X-Debug: X-Cache-Key
>
< HTTP/1.1 200 OK
< Server: ATS/6.1.0
< Date: Thu, 19 Nov 2015 23:17:58 GMT
< Content-type: application/json
< Age: 0
< Transfer-Encoding: chunked
< Proxy-Connection: keep-alive
< X-Cache-Key: /www.example.com/80/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&c=3
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0
The response header X-Cache-Key
header contains the cache key:
/www.example.com/80/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&c=3
The xdebug.so
plugin and X-Debug
request header are used just to demonstrate basic cache key troubleshooting.
If we add --static-prefix=nice_custom_prefix
to the remap rule then the cache key would look like the following:
/nice_custom_prefix/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&c=3
Usage examples¶
URI query parameters¶
Ignore the query string (all query parameters)¶
The following added to the remap rule will ignore the query, removing it from the cache key.
@plugin=cachekey.so @pparam=--remove-all-params=true
Cache key normalization by sorting the query parameters¶
The following will normalize the cache key by sorting the query parameters.
@plugin=cachekey.so @pparam=--sort-params=true
If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use a=1&b=2&c=1&k=1&u=1&x=1&y=1
Ignore (exclude) certain query parameters¶
The following will make sure query parameters a and b will not be used when constructing the cache key.
@plugin=cachekey.so @pparam=--exclude-params=a,b
If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&x=1&k=1&u=1&y=1
Ignore (exclude) certain query parameters from the cache key by using regular expression (PCRE)¶
The following will make sure query parameters a
and b
will not be used when constructing the cache key.
@plugin=cachekey.so @pparam=--exclude-match-params=(a|b)
If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&x=1&k=1&u=1&y=1
Include only certain query parameters¶
The following will make sure only query parameters a and c will be used when constructing the cache key and the rest will be ignored.
@plugin=cachekey.so @pparam=--include-params=a,c
If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&a=1
Include only certain query parameters by using regular expression (PCRE)¶
The following will make sure only query parameters a
and c
will be used when constructing the cache key and the rest will be ignored.
@plugin=cachekey.so @pparam=--include-match-params=(a|c)
If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&a=1
White-list + black-list certain parameters using multiple parameters in the same remap rule.¶
If the plugin is used with the following plugin parameters in the remap rule:
@plugin=cachekey.so \
@pparam=--exclude-params=x \
@pparam=--exclude-params=y \
@pparam=--exclude-params=z \
@pparam=--include-params=y,c \
@pparam=--include-params=x,b
and if the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&b=1
White-list + black-list certain parameters using multiple parameters in the same remap rule and regular expressions (PCRE).¶
If the plugin is used with the following plugin parameters in the remap rule:
@plugin=cachekey.so \
@pparam=--exclude-match-params=x \
@pparam=--exclude-match-params=y \
@pparam=--exclude-match-params=z \
@pparam=--include-match-params=(y|c) \
@pparam=--include-match-params=(x|b)
and if the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&b=1
Mixing –include-params, –exclude-params, –include-match-param and –exclude-match-param¶
If the plugin is used with the following plugin parameters in the remap rule:
@plugin=cachekey.so \
@pparam=--exclude-params=x \
@pparam=--exclude-match-params=y \
@pparam=--exclude-match-params=z \
@pparam=--include-params=y,c \
@pparam=--include-match-params=(x|b)
and if the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1
the cache key will use c=1&b=1
HTTP Headers¶
Include certain headers in the cache key¶
The following headers HeaderA
and HeaderB
will be used when constructing the cache key and the rest will be ignored.
@plugin=cachekey.so @pparam=--include-headers=HeaderA,HeaderB
HTTP Cookies¶
Include certain cookies in the cache key¶
The following headers CookieA
and CookieB
will be used when constructing the cache key and the rest will be ignored.
@plugin=cachekey.so @pparam=--include-headers=CookieA,CookieB
Prefix (host, port, capture and replace from URI)¶
Replacing host:port with a static cache key prefix¶
If the plugin is used with the following plugin parameter in the remap rule.
@plugin=cachekey.so @pparam=--static-prefix=static_prefix
the cache key will be prefixed with /static_prefix
instead of host:port
when --static-prefix
is not used.
Capturing from the host:port and adding it to the prefix section¶
If the plugin is used with the following plugin parameter in the remap rule.
@plugin=cachekey.so \
@pparam=--capture-prefix=(test_prefix).*:([^\s\/$]*)
the cache key will be prefixed with /test_prefix/80
instead of test_prefix_371.example.com:80
when --capture-prefix
is not used.
Capturing from the entire URI and adding it to the prefix section¶
If the plugin is used with the following plugin parameter in the remap rule.
@plugin=cachekey.so \
@pparam=--capture-prefix-uri=/(test_prefix).*:.*(object).*$/$1_$2/
and if the request URI is the following
http://test_prefix_123.example.com/path/to/object?a=1&b=2&c=3
the the cache key will be prefixed with /test_prefix_object
instead of test_prefix_123.example.com:80
when --capture-prefix-uri
is not used.
Combining prefix plugin parameters, i.e. –static-prefix and –capture-prefix¶
If the plugin is used with the following plugin parameters in the remap rule.
@plugin=cachekey.so \
@pparam=--capture-prefix=(test_prefix).*:([^\s\/$]*) \
@pparam=--static-prefix=static_prefix
the cache key will be prefixed with /static_prefix/test_prefix/80
instead of test_prefix_371.example.com:80
when either --capture-prefix
nor --static-prefix
are used.
Path, capture and replace from the path or entire URI¶
Capture and replace groups from path for the “Path” section¶
If the plugin is used with the following plugin parameter in the remap rule.
@plugin=cachekey.so \
@pparam=--capture-path=/.*(object).*/const_path_$1/
and the request URI is the following
http://test_path_123.example.com/path/to/object?a=1&b=2&c=3
then the cache key will have /const_path_object
in the path section of the cache key instead of /path/to/object
when either --capture-path
nor --capture-path-uri
are used.
Capture and replace groups from whole URI for the “Path” section¶
If the plugin is used with the following plugin parameter in the remap rule.
@plugin=cachekey.so \
@pparam=--capture-path-uri=/(test_path).*(object).*/$1_$2/
and the request URI is the following
http://test_path_123.example.com/path/to/object?a=1&b=2&c=3
the the cache key will have /test_path_object
in the path section of the cache key instead of /path/to/object
when either --capture-path
nor --capture-path-uri
are used.
Combining path plugin parameters –capture-path and –capture-path-uri¶
If the plugin is used with the following plugin parameters in the remap rule.
@plugin=cachekey.so \
@pparam=--capture-path=/.*(object).*/const_path_$1/ \
@pparam=--capture-path-uri=/(test_path).*(object).*/$1_$2/
and the request URI is the following
http://test_path_123.example.com/path/to/object?a=1&b=2&c=3
the the cache key will have /test_path_object/const_path_object
in the path section of the cache key instead of /path/to/object
when either --capture-path
nor --capture-path-uri
are used.
User-Agent capturing, replacement and classification¶
Let us say we have a request with User-Agent
header:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3)
AppleWebKit/537.75.14 (KHTML, like Gecko)
Version/7.0.3 Safari/7046A194A
Capture PCRE groups from User-Agent header¶
If the plugin is used with the following plugin parameter:
@plugin=cachekey.so \
@pparam=--ua-capture=(Mozilla\/[^\s]*).*(AppleWebKit\/[^\s]*)
then Mozilla/5.0
and AppleWebKit/537.75.14
will be used when constructing the key.
Capture and replace groups from User-Agent header¶
If the plugin is used with the following plugin parameter:
@plugin=cachekey.so \
@pparam=--ua-capture=/(Mozilla\/[^\s]*).*(AppleWebKit\/[^\s]*)/$1_$2/
then Mozilla/5.0_AppleWebKit/537.75.14
will be used when constructing the key.
User-Agent white-list classifier¶
If the plugin is used with the following plugin parameter:
@plugin=cachekey.so \
@pparam=--ua-whitelist=browser:browser_agents.config
and if browser_agents.config
contains:
^Mozilla.*
^Twitter.*
^Facebo.*
then browser
will be used when constructing the key.
User-Agent black-list classifier¶
If the plugin is used with the following plugin parameter:
@plugin=cachekey.so \
@pparam=--ua-blacklist=browser:tool_agents.config
and if tool_agents.config
contains:
^PHP.*
^Python.*
^curl.*
then browser
will be used when constructing the key.