### Intro Scalability - Not about speed/performance - E.g. no disk operations - Scalable system doesn't change when the size of the problem changes - Vertical (bigger servers) Vs Horizontal (more servers) ### Infrastructure Web server - nginx/fcgi - apache + mod_python (YUCK)/fcgi/mod_wsgi - lighttpd - cherrypy/wsgi - cherokee Database - PostgreSQL - MySQL (nice replication) - NoSQL: distributed Monitoring - Pingdom - Self-hosted: Zabbix, Ganglia, Munin, Nagios - Server load, CPU, RAM, DB QPS, Cache status, Queue status Load balancer (nginx haproxy) - Shared-nothing architecture (responsibility down the stack) - Also hardware ones ($30K) -- need two - Software ones do cool things (caching, re-proxying, auth, mod_rewrite) Static serving: - nginx - Amazon S3 distributed - Static media browser caching + Use the "expire" method + Static media versioning + /1/css/foo.css or css/foo.css?v=1 ### Programming Caching - First-level: Varnish: (anonymous) - Full-page caching (per-site caching, no GET/POST) - Per-view caching - HTML fragment caching - Low-level caching: Objects (django-orm-caching -- CachedModel class) - Low-level caching: Raw class Person(models.Model): ... def get_favs(self): key = ‘favs_for_%s’ % self.user.id favs = cache.get(cache_key) if not favs: favs = self.favs.all() cache.set(key, favs) return favs - Denormalization + Joins are evil + Frameworks make it easier to do heavy stuff + List of IDs (Joins are bad) - Tough: Invalidation Queues - RabbitMQ (ZeroMQ, Ghetto) - Celery + AMQP - Persistence Stay clean - Good code quality - SQL introspection: + Person.objects.filter(...).query.as_sql() + django.db.connection.queries + django-debug-toolbar: timer, HTTP headers, GET/POST variables, SQL queries, signals - Be effective: Call stuff outside your code (e.g. django settings) Semi-static data - How to update static data? ### Practices Django-addons - Why? Multiple versions of your product - Decouple using Signals - Too many hooks are bad Migrations - Django South - Version-by-version schema evolution - Data evolution Testing - Unit tests: Nose, mg test - Hudson, buildbot - Coverage tests - tests/translations/test_managers with regexxes and multiple languages