the static asset pipeline

optimized production experience, uncompromised development flow

@bogdancostea

@bogdancostea

software developer and technical lead

10+ years experience

Java, .Net, ruby

desktop, rich client, web

infrastructure and test automation

multi-language, multi-region deployments

agile and lean methods

 

intro

there's no place like home and no environment like production

this is the place where your app was meant to be, it was born to go live

 

every step you took since the first user story, the first screen mockup and the first line of code has been directed at this one point - deployment to production

 

we like to build things fast so we tend to forget that the things we build are not meant to be built, they are meant to be used

environments

being agile, we accept change, and when it happens, we want to do it fast, and above all, we want it done right

the best way to ensure that it's both fast and right is to have separated and as-automated-as-possible environments

 

these typically include:

  • development - this is where the programmers live
  • testing - this is where QA and product management lives
  • production - this is where users live

 

more on environments

depending on your flow you may have more environments, such as staging, acceptance, demo, etc. - that's up to you

 

for this talk we'll simplify everything and focus on the production and development environments - this is meant for brevity

 

so where's the problem?

the problem

everything is fine and dandy until you realize that for a web app there's an inherent mismatch between different environment concerns

development

  • developer productivity
  • code visibility
  • tool and library integration
  • test isolation
  • debugging

production

  • user experience
  • intellectual property
  • consistency
  • resource usage
  • scaling

we need a middle ground

development:

  • it's a good thing to have JavaScript classes/modules split up in separate files - it's good for productivity and testing, it's clean and we love it
  • it's a good thing to have your styling split up just like the components that are being styled, SASS keeps the styling small, clean and configurable
  • it's nice to have just one copy of an image used for menus, tooltips and help page

production:

  • ​​why is the login page requesting 1.4MB of data in 56 requests, taking a whole 4 seconds to load? 
  • this thing is slow
  • did we forget to compile to CSS before deployment again?

the pipeline

that's where the asset pipeline comes into play

 

the role of the asset pipeline is to:

  • preprocess dynamic assets into static assets (think SASS and CoffeeScript)
  • concatenate multiple raw assets into single assets for faster loading
  • lint and minify
  • optimize images for size and compatibility

 

when used properly, an asset pipeline will give you the best of both worlds

building the asset pipeline

this should be done from the start.

 

it's not impossible to do it on an existing codebase but if there are hidden asset dependencies of if the code expects some assets to be served in a specific way it's going to be really nasty...

 

let's start

environments

first things first - you need to separate your environments

 

your build tool, whatever that may be, must:

  • know that you have different environments - dev, prod
  • run build processes based on those different environment definitions

 

as a quick note, it's really good to have an automated build :)

your assets

your assets are source files

 

these files, although most of them may be static must be treated as source files, checked in to version control

these files will not be served directly to clients

 

this means that you must define 2 separate locations:

- assets location - this is where you pick up your assets for processing

- public location - this is where you output the result and where the browser has access to them 

caveats

the way that you code your assets, especially JS may make or break your process

 

JS variables can and will conflict if you do not take into account that all your files will be concatenated and that the result will be loaded all at once

 

also, you must take into consideration that various assets may have dependencies and that you should respect the dependencies when loading - use AMD, require.js

the pipeline

the pipeline usually consists of:

  • pre-processing - compile assets to final form
  • concatenation - turn muliple assets into one to optimize round-trips
  • minification  - optimize the assets for bandwith usage
  • image optimization / multimedia processing
  • compression - prepare pre-gzipped files to avoid server-side compression
  • public asset versioning (bumping) - to support cache eviction

the input

the process

why bump?

producing and using single JavaScript and CSS artifacts gets really useful when you cache them server-side and client-side

 

well, this gets painful when you release a new version and you get mismatched versions of your artifacts because of cache refresh issues

 

bumping versions on every release allows you to produce unique artifacts that have a different filename (fingerprinted), avoiding caching issues

delivery

asset delivery can highly influence user experience, so taking the following into account is a must:

  • CDNs help. using mulptiple resources from multiple CDNs erases the benefits
  • client-side caching is a must. cache retention must be as long as possible
  • assets have considerable impact on resource usage - more files -> less throughput

before

after

pipeline stack

 

  • NodeJS - https://nodejs.org/
  • RequireJS ( + Almond) - http://requirejs.org/
  • SASS - http://sass-lang.com/
  • Grunt - http://gruntjs.com/
  • Grunt tasks: grunt-contrib-concat, grunt-contrib-connect, grunt-contrib-cssmin, grunt-contrib-sass, grunt-concat-css, grunt-contrib-requirejs, grunt-text-replace, grunt-contrib-htmlmin
  • willpower

thank you

 

speed is a feature

user experience is a feature

 

don't let your users down

 

 

@bogdancostea