my opera meets varnish, dec 2009
DESCRIPTION
Slide for a talk I presented internally at Opera in December 2009 about the deployment of varnish in our production environment at my.opera.com, the social network community.TRANSCRIPT
![Page 2: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/2.jpg)
What is Varnish?
varnish
backends● Caching reverse proxy, like Squid● Delegates memory mgmt to OS cache● Mainly developed at Linpro in Oslo
![Page 3: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/3.jpg)
Two typical Varnish setups
varnish
backends
frontends
incomingrequests
![Page 4: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/4.jpg)
• man vcl• VCL is compiled to C code• Injected into the running instance, without restart• Must define a backend or a director• VCL gives you several hooks:vcl_recv()vcl_hash()vcl_fetch()vcl_hit()vcl_miss()vcl_deliver()
VCL - Varnish Config Language
![Page 5: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/5.jpg)
• In production beginning of October 2009• 1 old recycled machine, 2 Gb of disk allocated• Started serving avatars
1M+ requests per day before Unitehttp://my.opera.com/<username>/avatar.pl
• Soon after, added Desktop Team RSS (very popular!)• then user pictures, hundreds of thousands req/day• then Unite/ASD API requests
- friends of a user- groups of a user
• In total, 13,25% of all My Opera requests are «varnished»• Around 7,2M req/day
Varnish deployment in My Opera
![Page 6: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/6.jpg)
• Still using Debian Etch?First Varnish instance was running v1.x from Etch.several years old, not good
• Experienced VIPs– ”Very Interesting Problems”– User X getting User Y's session– Random users getting admin powers. Nightmare!
• Theory: Varnish was caching response bodies that containedSet-Cookie: opera_session=<session_id>
Varnish deployment in My OperaProblems /1
![Page 7: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/7.jpg)
• There wasn't any obvious configuration problem.Same config worked with 2.0.x from Backports.
• v2.0.{4,5} is highly recommended!
Varnish deployment in My OperaProblems /2
![Page 8: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/8.jpg)
• We tried caching the frontpage of My Opera, but had to revert the change due to too many different custom layouts for Opera Mobile, Mini, IE, Firefox, etc...
• Maybe using clever vcl_hash() tricks we can achieve that too.
Varnish deployment in My OperaProblems /3
![Page 9: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/9.jpg)
My Opera configuration
![Page 10: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/10.jpg)
• Backendsingle backend machine, or load-balanced virtual server
• Director– simple round-robin or random weighted “balancing” logic– has basic connection retries mechanism– has basic backend health check
• If you already have an LVS, define a single BackendOtherwise, go for the Director
Backends and Directors
![Page 11: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/11.jpg)
Backends and Directors
Define a backend
# Only hit the upload serversbackend myopera { .host = "upload.my.opera.com"; .port = "80";}
![Page 12: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/12.jpg)
Backends and Directors
Define a director
director myopera round-robin { .backend {
.host = "b1.opera.com";
.port = "80"; } .backend {
.host = "b2.opera.com";
.port = "80"; } ...}
![Page 13: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/13.jpg)
Backends and Directors
...and then use them
sub vcl_recv { ... set req.backend = myopera; ...}
![Page 14: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/14.jpg)
sub vcl_recv {
set req.backend = myopera; set req.grace = 3m;
# URL patterns based cache. # Avoid possible mixups. if(req.http.host !~ "^my\.opera\.com$") { pass; }
vcl_recv() / 1
![Page 15: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/15.jpg)
if (req.url ~ "^/community/users/avatar\.pl/[0-9]+$" || req.url ~ "^/.+/avatar\.pl$" || req.url ~ "^/.+/picture\.pl\?xscale=100$" || req.url ~ "^/desktopteam/xml/atom/blog/?$" || req.url ~ "^/desktopteam/xml/rss/blog/?$" || req.url ~ "^/community/api/users/friends\.pl\?user=.+$" || req.url ~ "^/community/api/users/groups\.pl\?user=.+$") { unset req.http.Cookie; unset req.http.Authorization; lookup;}
vcl_recv() / 2
![Page 16: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/16.jpg)
... # Check for cookie only after always-cache URLs if (req.http.Cookie ~ "(opera_session|opera_persistent_)") { pass; }
# DANGER, Will Robinson! Caching the front-page # At this point, lots of Google Analytics cookies will go in. # No problem. It's stuff used by Javascript if (req.url ~ "^/community/$") { lookup; }
pass;}
vcl_recv() / 3
![Page 17: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/17.jpg)
sub vcl_fetch {
set obj.http.X-Varnish-URL = req.url; set obj.grace = 3m;
if (obj.http.Set-Cookie) { set obj.http.X-Varnish-Cacheable = "no, set-cookie"; pass; }
if (req.request != "GET") { set obj.http.X-Varnish-Cacheable = "no, !GET"; pass; }
vcl_fetch() / 1
![Page 18: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/18.jpg)
if (req.http.host !~ "^my\.opera\.com$") { set obj.http.X-Varnish-Cacheable = "no, !my.opera.com"; pass;}
if (req.url ~ "^/community/users/avatar\.pl/[0-9]+$" || req.url ~ "^/[A-Za-z0-9]+/avatar\.pl$" || ... ) { unset obj.http.Set-Cookie; set obj.http.X-Varnish-Cacheable = "yes, url"; set obj.ttl = 24h; deliver;}
vcl_fetch() / 2
![Page 19: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/19.jpg)
vcl_hash()sub vcl_hash { # Default Varnish behavior set req.hash += req.url; set req.hash += req.http.host; # Have a different cached frontpage per language if (req.url ~ "^/community/$") { set req.http.X-FrontPage-Language = regsub( req.http.Cookie, "^.*?language=([^;]*?);*.*$", "\1" ); set req.hash += "lang:"; set req.hash += req.http.X-FrontPage-Language; } hash;}
![Page 20: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/20.jpg)
• Developed a testing tool (varnish-test)– outputs a TAP stream and some debug info– works best if varnish is specially tuned
• Can quickly check if a test/production instance is performing correctly or having problems
• Invoked as a simple script:varnis h-tes t --profile=tes ts .url --hos t=b1
Testing Varnishhow to avoid nightmares...
![Page 21: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/21.jpg)
# Frontpag e
/ N O_C OOK IE S V AR N IS H_C AC HED
/ N O_C OOK IE S V AR N IS H_N OT_C AC H ED Hos t: my.c n.opera .c om
/ N O_C OOK IE S V AR N IS H_C AC HED C ookie:lang uag e=it
# B log s
/des ktopteam/blog / N O_C OOK IE S V AR N IS H_N OT_C AC HE D
# Avatars
/c ommunity/us ers /avatar/817271 N O_C OOK IE S V AR N IS H_C AC HE D
/c ommunity/us ers /avatar/442 N O_C OOK IE S V AR N IS H_C AC HE D
/g raphic s /avatar.g if N O_C OOK IE S V AR N IS H_N OT_C AC HE D
Testing Varnishcaching test list
![Page 22: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/22.jpg)
• We can specify exactly how the varnish instance should behave.– Production acceptance tests– Test new varnish versions, new OS distributions– Fine tune config changes quickly with no impact on production
• Midway through there's a request that logs in as a test user.From then on, we can verify what resources are cached whena user is logged in. Some resources should be cached in any case.
Testing Varnishcaching test list
![Page 23: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/23.jpg)
Testing Varnishsample run
...ok 289 - Got response from backend for /community/ (from ...) ok 290 - Correct status line# Adding header [Cookie] => [language=it]# ----------# GET http://cache01.my.opera.com:6081/community/# Host: my.opera.com# ------------ok 291 - 2nd request: got response from backend for /community/ (from...)ok 292 - Correct status line# X-Varnish: 1211283813 1211283812# X-Varnish-Status: hit# X-Varnish-Cacheable: yes, language cookie# X-Varnish-URL: /community/ok 293 - URL '/community/' was handled correctly by varnish# cookie_header:ok 294 - URL '/community/' has correct cookies (or no cookies)1..294
All tests successful.
X-Varnish: 1211283813 1211283812X-Varnish-Status: hitX-Varnish-Cacheable: yes, language cookieX-Varnish-URL: /community/
![Page 24: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/24.jpg)
• varnishlog– Reads shared memory log info and displays it– Full instance log, on My Opera, 1 day is about 15 Gb– You can get an emulated Apache-style access.log from it
• varnishncsa– Displays requests to Varnish as Apache access logs– Can read from an archived log by varnishlog
• varnishstat– Displays realtime stats (hit ratio, space allocated, connections,...)
Monitoring Varnishbuilt-in tools
![Page 25: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/25.jpg)
• Munin plugins– Hit ratio– Requests rate– Backend traffic
• Nagios plugins– Nothing special, TCP connection to port 6081
Monitoring Varnishexternal tools
![Page 26: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/26.jpg)
Monitoring Varnish
![Page 27: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/27.jpg)
Monitoring Varnish
![Page 28: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/28.jpg)
Monitoring Varnish
![Page 29: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/29.jpg)
• My Opera front page caching• My Opera files server?• Working on a prototype thumbnail server
Next steps
![Page 30: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/30.jpg)
• Redpill-Linpro website– http://varnish.projects.linpro.no– Bug tracking, documentation and community support– Users and developers mailing lists
• Commercial support and training– http://www.varnish-cache.com
References and more information
![Page 31: My Opera meets Varnish, Dec 2009](https://reader033.vdocuments.us/reader033/viewer/2022052505/554fb1e4b4c905ad218b52c3/html5/thumbnails/31.jpg)
• At Opera, there's several teams using Varnish in production• If you want to know more, contact me: [email protected]
Questions?