Holograms and Data Science

This will be a quick and different post, recording one of these A-HA moments we sometimes have, before I forget it. It is a random thought that’s got more than 140 characters. A tweet would not do it.

I just finished reading The Black Hole War, by Leonard Susskind (one of the most important physicists alive today). It is a great book if you are interested in modern physics. Here is an excerpt of my review on Goodreads:

(…). For non-physicists like me, this was a fantastic introduction on what we currently know about quantum gravity and its relation with other areas of science. As a bonus, it also (finally) helped me start grasping string theory, and better understand entropy, the event horizon (complementarity, information paradox) and the holographic principle.

My A-HA moment came while the book discussed the Holographic Principle. Coincidence or not, I have been studying a whole bunch of Machine Learning, and things converged. I love when (seemingly) completely different areas of science suddenly converge. It reinforces my belief that there must be something common underlying all of it (call it Grand Unified Theory, or God, or as you please).

Principal Component Analysis and Dimensionality Reduction are the Data Science (Machine Learning) topics in particular that seem to be correlated with the Holographic Principle. They are techniques of removing redundancy in data sets and extracting the minimal data necessary to represent something. It allows you to reduce the number of columns in a database without losing meaningful information, for example. The main technique can be seen as a linear algebra transformation: a projection of the N-dimensional data onto a smaller K-dimensional space.

In a sense, holograms are the same thing: a projection (encoding) of a higher dimensional space onto a smaller dimensional space (3D to 2D, for example). The Holographic Principle states that all information inside a N-dimensional space is contained into its (N-1)-dimensional boundary. For example, all the information inside a volume (3D) is contained (or described) onto its surface/area (2D).

This is fascinating. Holograms are fascinating. It could mean that not all dimensions (e.g.: rows/columns in your database) are necessary to perfectly describe any information, and removing that redundancy is precisely what Dimensionality Reduction (and PCA) tries to do. I wonder if we can use the Holographic Principle to find ways to do Dimensionality Reduction without losing any information (a loss-less compression, if you will).

Linear algebra is another fascinating aspect of all this. Projecting data and extracting meaningful information (compression) always seems to involve the calculation of Eigenvectors and Eigenvalues. Google is constantly calculating Eigenvectors that power your searches. Somehow, Eigenvectors also seem to be central to all of this.

I am sure this is not something I’m inventing or discovering and there are plenty of papers about it out there. I only happen to not have stumbled upon any of them. A quick search tells me that Holography and Dimensionality Reduction is being used in many different areas of science, including genetics and biology. If you know of any of such papers (proving or disproving my random thought), let me know in the comments.

My DockerCon 2014 talk: Thoughts on interoperable containers

Recently, I finally had a chance to read Presentation Zen: Simple Ideas on Presentation Design and Delivery, by Garr Reynolds. I wish I had done if before, after so many years of giving less-than-ideal talks and ignoring recommendations from presenters I admire, who give presentations I highly respect.

It is usual to post your slides somewhere public right after you give a presentation. I’m sure that Docker organizers are going to publish my slides somewhere, but here they are if you really want to see them and don’t want to wait. However, if you agree with some of the guidelines from Presentation Zen, slides from a good presentation will not have a lot of useful information. Slides are there just as a visual aid to the story being told during a presentation. They should be highly visual and help illustrate your points. A slide deck is not a document.

I want to try something different this time. This blog post is an attempt of a proper handout of my slides. I hope it is a bit more useful than a bunch of slides with beatiful pictures and small sentences to all of you who could not attend DockerCon and didn’t see my talk. Or for those of you that attended (thank you!) and want to review what I said.

We want to run our apps, unmodified, everywhere. I too have been trying to find ways to make it happen, and I want to share some of the things I have discovered so far.

I run Linux Containers at Heroku. Lots of them. Heroku has been running Linux Containers (dynos) for more than 3 years. Maybe 4 years, I don’t know exactly when the move from chroot jails was made.

Docker has so many different uses! Every other week I discover people using Docker in a different way. For example, some people are now distributing CLI tools as Docker containers. Here is (more or less) how Ruby VMs are being built these days at Heroku (thanks to Terence and ENTRYPOINT):

docker run hone/ruby-builder --version 2.1.0p123

# a new ruby VM is now available in a local directory

Compare that with a ruby-builder binary that would do the same thing: the containerized version includes all the dependencies and works reliably everywhere Docker is available.

We are interested in a particular usage for Docker though. Docker to run portable, server side, web applications following the 12factor.net guidelines.

“Write once, run everywhere” is the dream. In reality, each group (app developers, PaaS providers, Docker developers, LXC developers, Linux Kernel developers, etc.) has different priorities. Different trade-offs and optimizations are being made by each one of them.

Developers want apps: “hey provider, just run my damn container!”.
PaaS providers want scale: they must scale as a business, with sustainable operations, while being fast and secure. Arbitrary Code Execution as a Service.
Docker wants to be the right tool for many different use cases. It is a toolkit that will enable many different things, but we can’t expect Docker developers to solve all the problems. They are too busy already!

I have been navigating this problem space for a while now, playing all the different roles. At the end, besides building a PaaS at Heroku, I am also an app developer and I enjoy hacking on open source container managers (Docker, LXC, the Linux kernel).

Of all the slides with beautiful pictures, this is the only one I can truly say that I made it myself. 🙂

I have been trying to reconcile all sides and would like to share some things I learned. Ultimately, I want to hear from you: tell me what is bullshit and what wrong assumptions I’m making, and/or if you agree. You can also contribute and help make this happen with ideas, design and code.

The container shipping analogy is the first thing that comes to mind. You build a full container locally then ship it to Platform and Infrastructure providers. A Docker container would run anywhere.

trying to make Docker secure for multi-tenant scenarios is a can of worms

– darren0, at #docker-dev

I agree with darren0. Most of the challenges I faced so far when trying to make it work as a Platform provider are differences between local or small Docker environments (single or a few apps on a few servers) versus multitenant environments where boxes are packed with containers from multiple different apps (tenants).

At Heroku, we run millions of containers (apps). It is not hard to imagine why trade-offs need to be different than with a smaller environment.

One of these challenges is root access. When you build a container locally or in your build servers, you have root access inside it.

Root access is controversial. For some people it is obvious why containers can’t have it in a shared (multitenant) environment, or any production environment for that matter. Others have a hard time accepting the idea it is dangerous and believe that containers should be safe enough to cover all issues. Or they don’t want to care and just want the runtime environment to solve all the problems.

App developers often want root access because:

apt get install ..., they want to install packages, tools and libraries.
vi /etc/..., they want to edit important configuration files.
mount -t fancy ..., they want to mount filesystems and use what the kernel has to offer.
modprobe something, they want to load modules and use what the kernel has to offer #2.
iptables -A INPUT ..., they want to configure firewall rules, port mappings, …

The big problem is that root access in a container is also root access on the host system. The same host machine that runs many other containers for different apps/tenants. If anyone can – for any reason – escape a container, they would be able to do anything they wanted with other containers in the same box: look at other containers data, code, and potentially escalate to other components of the infrastructure.

This used to be much easier. To be honest, things are much better nowadays, and I will admit that there is a lot of FUD. Jérôme Petazzoni gave a nice talk about it earlier this year, go check it out:

LXC, Docker, Security

Still, you need to always be careful and do the right thing when running containers. Thankfully, Docker, LXC and Heroku (the container managers I’m familiar with) are in general doing a good job IMO: dropping capabilities, using AppArmor/SELinux/grsecurity, kernel namespaces, cgroup controllers, etc. But anyone still can do bad things on top of it and leak resources inside containers. Just as an example, I’ve seen people mentioning they are bind mounting the Docker socket (/var/run/docker.sock) inside containers. Which basically gives containers full privileges on the host.

But, escaping containers is not the only problem with root access. In the Linux Kernel, there are still many parts that haven’t been containerized. Some places in the kernel will hold global locks, or expose global resources. Some examples are the /sys and /proc filesystems. Docker does a great job preventing known problems, but if you are not careful:

# don't do this in a machine you care about, the host will halt
docker run --privileged ubuntu sh -c "echo b > /proc/sysrq-trigger"

If you are not careful protecting /sys and /proc (and again, Docker by default does the right thing AFAIK), any container can bring a whole host down, affecting other tenants running on the same box. Even when container managers do everything they can to protect the host, it might not be enough. Root users inside containers can still call syscalls or operations usually only available to root, that will cause the kernel to hold global locks and impact the host. I wouldn’t be surprised if we keep finding more of such parts of the kernel.

This is also one of the reasons that many people are choosing to run one container per box (the other being better resource isolation), physical or virtual. One container per VM is a nice way to leverage the advantages of both containers and hypervisors.

I am happy that this is getting better and better with time, and most of these concerns are not really a big deal anymore, as long as you use bleeding edge versions of Docker, the Linux Kernel, and AppArmor/SELinux, etc. I can see a light at the end of the tunnel that would allow Providers to permit root access inside containers. Some Providers even started already running code as root inside containers. However, it increases the surface area of attack considerably, and we can’t blame providers for trying to avoid malicious tenants (or even unintentionally) DoS’ing their boxes or getting access to other containers (apps).

User namespaces (a.k.a. unprivileged containers) were recently added to the Linux Kernel to provide safe root access usage inside containers. It allows regular (unprivileged) user ids on a host machine to be mapped to arbitrary user ids inside a new namespace.

That way, root users (uid=0) inside a container (namespace) can be mapped to regular unprivileged users outside of the container. Root users inside user namespaces are not necessarily root on the whole box. It is a better implementation of fakeroot enforced and controlled by the kernel. Nice!

This is very recent though, and will probably take some time to stabilize. It is not hard to find exploits from earlier this year (a few months ago).

Because of the way it works, by design, it can also be made dangerous. By default, root users inside new user namespaces have all the kernel capabilities (superpowers) inside that namespace (container).

Providers relying on user namespaces to provide (fake)root access will still need to drop capabilities, otherwise (fake)root inside containers will still be able to call syscalls and operations on the kernel that only root users can call. With that, they can potentially DoS boxes or escape containers. Another simple example is mknod and mount: if a container is not protected appropriately, even with user namespaces, a (fake)root user could just get raw access to local disks in a box, mount them, and start reading all unencrypted data in the box, including data from other containers. Again, Docker does the right thing AFAIK, as long as containers don’t run as --privileged. But, Docker doesn’t support user namespaces yet. I’m sure Docker developers are thinking about it before blindly adding support.

Another attack vector is code like this:

if (getuid() == 0) {
  // do root stuff
}

Who has not done this before? Blame me, I certainly have. This is how auth was intended to be done on Linux systems, before capabilities and user namespaces were added. Nowadays, in order to be safe, that code would need to check capabilities in a specific user namespace.

What code like this is out there? Who knows… but if you are uid 0 in a namespace you will be able to pass all these validations.

When a root user has all its capabilities dropped in a new user namespace, it is very similar to a regular non-privileged user. None of the things a user would want root access for are possible. It might as well be a regular, non-privileged user. Most (if not all) of the things that require root will not work anyway.

Just don’t run as root?

As an app developer, why should I care? Providers then can just run app code as unprivileged users and end of story!

Unfortunately not: setuid binaries are a problem in this scenario. Binaries with that permission (bit) set are executed as the owner of the file, not with the permissions of the user that runs the binary.

Injecting arbitrary code as setuid binaries owned by root is a very well known old attack to execute code as root on UNIX systems. Container images built locally can contain anything. If PaaS providers accept arbitrary images to run, they must be very careful to remove all setuid binaries owned by root, or disable setuid completely on all filesystems (with the nosuid flag and AppArmor/SELinux, for example). Anyone can inject any setuid binary owned by root in container images being built locally.

Providers also need to be extra careful with what gets injected into containers, e.g.: bind mounts with --volume or --volumes-from. A filesystem without nosuid leaking into a container allows malicious users to create and execute arbitrary setuid binaries.

These binaries are sometimes useful and some apps may require them. setuid is what allows unprivileged users to execute useful things only traditionally available to superusers, like ping (requires raw socket access), tcpdump (requires promiscuous mode), etc. These will probably not be available in many multitenant container Platforms.

This all makes me start thinking that we may need a way to specify constraints on container images that particular runtime environments impose. Different Providers will make different choices on what they accept or not, and it may be useful to capture these requirements, or container capabilities as container metadata.

We would need something like a Restrictions API for containers, that could be validated at runtime by Providers, and/or during build by container managers, or during docker push by registries, …

Here are some other examples of requirements that may need to be imposed on containers by some runtime environments:

Networking: the number of network interfaces available, which ports are reachable/routable from the outside, how many IP addresses are available, public vs. private IPs, firewall rules, etc.
Ephemeral disks: some Providers may not provide persistent disks for containers.
Arch, OSes: which architectures are supported (eg.: x86_64, arm).
Image size: there probably will be a max amount of disk a container can occupy, or a max size for the container image to be downloaded.

container-rfc is an attempt to define a standard container image format and metadata that would work on multiple container managers.

Container images, in my experience, have 2-5GB.

Heroku is very dynamic. Containers are constantly being cycled and are always moving. Thousands of containers are being launched per minute. Having to download a 2-5GB image every time a container is being launched does not sound reasonable.

Docker solves this problem with layers.

Deltas are overlaid on top of a base image. All containers are formed from a hierarchy of read-only layers, and a private writeable layer on top of them. Read-only layers are shared between all containers that use them on a host, and are cached locally.

When many containers use the same base image and share some layers, they are downloaded only once. If providers can restrict the number of base images they support (Restrictions API, again?) known base images can be pre-cached in runtime servers and containers can be launched very quickly, as only the layers (deltas) specific to each app are going to be downloaded.

Even so, Providers will probably need a way to limit the max size of a layer, or a set of layers.

This works well, until base images (or shared layers) need to be updated. We do that constantly at Heroku, for example with security updates. Every time a base image or shared layer gets updated, all containers using them need to be rebuilt, to pick up the changes.

Heroku wouldn’t be able to quickly respond to security incidents like Heartbleed if we had to rebuild millions of containers every time the base image needs to be updated.

Compare that with what we currently do at Heroku.

It is a more traditional approach. Similar to how packages are traditionally installed on Operating Systems. Apps are just unpacked on top of a read-only base image, inside containers. The read-only base image is shared (bind mounted) between all containers, and downloaded only once on each box.

When containers are being launched, only the app package (a.k.a. slug) is downloaded.

When the base image needs to be updated, a new version is sent to all runtime instances, and new containers are simply launched on top of a new read-only base image. No rebuild operations are required on any apps.

It is also possible to do that with Docker, but it does not follow the traditional model of shipping containers (docker push + docker pull). The community may come up with good ways to solve this problem in the future, see dotcloud/docker#332 for example.

# idea: make a container point to a new base image?
docker save myapp | docker load --rebase=new-base-image

Honestly, the whole idea of supporting an arbitrary image format everywhere reminds me a bit of what happened with VM image formats a while ago. Many people were trying to find a common format for VMs that would work on any hypervisor. They failed. Do you remember VMDK vs VHD vs QCow vs QCow2 vs …?

I can’t say yet if the idea of shipping whole containers everywhere is a rabbit hole. Maybe it is, maybe it isn’t. I’ve been experimenting with it and others should do as well.

But let’s take a step back. How about instead of putting restrictions on whole container images and distribute them everywhere, we turn portable apps into containers? We shift the focus back to the app code to be distributed and make sure that apps are portable and can run anywhere, then we map apps to execution runtimes, like Docker containers.

Runtime environments don’t need to be just containers. Portable apps (12factor.net) can then be mapped to raw VMs, or even bare metal servers.

What we are missing in this context is something standard to prepare these portable apps for different runtime environments.

Buildpacks are a possible way to make this happen. Initially, buildpacks were created to compile (build) 12factor apps into a package that can be executed inside Heroku containers (dynos). But buildpacks are very simple and flexible: just a few executables (bin/detect, bin/compile and bin/release) that are used to build an app for a runtime environment.

Many different PaaS providers adopted buildpacks as their mechanisms to build app code. Each language runtime (Ruby, Python, Node.js, etc.) has a different buildpack.

I propose that we extend the concept of a buildpack: given a base image, it is something that maps (transforms) apps into runtime environments (e.g.: a Docker container) during builds.

One way to implement this idea for Docker, is to make each buildpack a container image that can be used as a parent image by 12factor apps:

$ cat my-portable-app/Dockerfile
FROM heroku/heroku-buildpack-ruby

Buildpack images can then use ONBUILD triggers to do everything required to turn an app into an executable Docker container:

$ cat heroku/heroku-buildpack-ruby/Dockerfile
# Heroku's base image based on Ubuntu 10.04
FROM heroku/cedar

ADD . /buildpack
ONBUILD ADD . /app
ONBUILD RUN /buildpack/bin/compile /app
ONBUILD ENV PORT 5000
ONBUILD EXPOSE 5000

I am exploring heavily with this and I’ve already learned that ONBUILD has limitations (dotcloud/docker#5714). I hope to share more soon.

I am also sure there are other possible ways to use buildpacks to turn apps into executable docker containers. This is just an example (not even complete). Others are doing similar things:

Buildstep, from Flynn/Dokku.
Google *-runtime images act as buildpack images too, though they don’t use heroku buildpacks – yet :-).
Radial tries to map 12factor guidelines to Docker.

I’m not trying to undervalue containers (and Docker!). They are still an amazing way of running apps. But once we shift the focus from “shipping containers” to “shipping apps” again, we open possibilities to run our apps in more runtime environments. As an example, some of my apps have a Makefile to build them locally using a buildpack, so that I can run them locally, on my dev machine (no VMs, no containers, just plain old process execution):

#!/usr/bin/env make -f

buildpath := .build
buildpackpath := $(buildpath)/pack
buildpackcache := $(buildpath)/cache

build: $(buildpackpath)/bin
    $(buildpackpath)/bin/compile . $(buildpackcache)

$(buildpackcache):
    mkdir -p $(buildpath)
    mkdir -p $(buildpackcache)
    curl -O https://codon-buildpacks.s3.amazonaws.com/buildpacks/kr/go.tgz
    mv go.tgz $(buildpath)

$(buildpackpath)/bin: $(buildpackcache)
    mkdir -p $(buildpackpath)
    tar -C $(buildpackpath) -zxf $(buildpath)/go.tgz

make is not the most straightforward thing, but this should be simple enough. make build will build my code using a buildpack. Int his case, the code is written in Go, but it could be any language/framework that has a buildpack. The Makefile just downloads a buildpack and calls bin/compile, the standard buildpack interface to build apps.

This makefile can be seen as a buildpack that maps an app (source code) into a runtime environment (single binary), given a base image (the OS on my laptop).

Here is another example we use at Heroku to run apps in raw machines or VMs (hey, sometimes we need to do it too!):

ruby = "https://codon-buildpacks.s3.amazonaws.com/buildpacks/heroku/ruby.tgz"

app_container "myapp" do
  buildpack ruby
  git_url "git@mycompany.com:myapp.git"
end

define :app_container,
       name: nil,
       buildpack: nil,
       git_url: nil do
  # ...

  execute "#{name} buildpack compile" do
    command "#{dir}/.build/pack/bin/compile #{dir} .build/cache"
  end
end

It is a Chef recipe that builds an app inside a machine using a buildpack, by calling bin/compile. Again!

If an app can be built with a buildpack, then it can potentially run on multiple runtime envinronments, including Docker containers, Heroku, Cloud Foundry, your own machines via Chef recipes, etc.

My talk was not intended to provide any final answers. In my personal journey, I’ve been bouncing between these two concepts: container centric and app centric, and so far I have found that both have their pros and cons. I’m leaning more towards the app centric model, but I’m biased working as a PaaS provider. Ultimately, I hope we can find something most people are happy with and that everyone can use to run their apps (almost) anywhere.

Memory inside Linux containers

Or why don’t free and top work in a Linux container?

Lately at Heroku, we have been trying to find the best way to expose memory usage and limits inside Linux containers. It would be easy to do it in a vendor-specific way: most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others.

An implementation of Linux containers could easily inject one or more of those files inside containers. Here is an hypothetical example of what Heroku, Docker and others could do:

# create a new dyno (container):
$ heroku run bash

# then, inside the dyno:
(dyno) $ cat /sys/fs/cgroup/memory/memory.stat
cache 15582273536
rss 2308546560
mapped_file 275681280
swap 94928896
pgpgin 30203686979
pgpgout 30199319103
# ...

/sys/fs/cgroup/ is the recommended location for cgroup hierarchies, but it is not a standard. If a tool or library is trying to read from it, and be portable across multiple container implementations, it would need to discover the location first by parsing /proc/self/cgroup and /proc/self/mountinfo. Further, /sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location. Thinking about it, /sys/fs/cgroup/self would not be a bad idea.

If we decide to go down that path, I would personally prefer to work with the rest of the Linux containers community first and come up with a standard.

I wish it were that simple.

The problem

Most of the Linux tools providing system resource metrics were created before cgroups even existed (e.g.: free and top, both from procps). They usually read memory metrics from the proc filesystem: /proc/meminfo, /proc/vmstat, /proc/PID/smaps and others.

Unfortunately /proc/meminfo, /proc/vmstat and friends are not containerized. Meaning that they are not cgroup-aware. They will always display memory numbers from the host system (physical or virtual machine) as a whole, which is useless for modern Linux containers (Heroku, Docker, etc.). Processes inside a container can not rely on free, top and others to determine how much memory they have to work with; they are subject to limits imposed by their cgroups and can’t use all the memory available in the host system.

This causes a lot of confusion for users of Linux containers. Why does free say there is 32GB free memory, when the container only allows 512MB to be used?

With the popularization of linux container technologies – Heroku, Docker, LXC (version 1.0 was recently released), CoreOS, lmctfy, systemd and friends – more and more people will face the same problem. It is time to start fixing it.

Why is this important?

Visibility into memory usage is very important. It allows people running applications inside containers to optimize their code and troubleshoot problems: memory leaks, swap space usage, etc.

Some time ago, we shipped log-runtime-metrics at Heroku, as an experimental labs feature. It is not a portable solution though, and does not expose the information inside containers, so that monitoring agents could read it. To make things worse, most of the monitoring agents out there (e.g.: NewRelic?¹) rely on information provided by free, /proc/meminfo, etc. That is plain broken inside Linux containers.

On top of that, more and more people have been trying to maximize resource usage inside containers, usually by auto-scaling the number of workers, processes or threads running inside them. This is usually a function of how much memory is available (and/or free) inside the container, and for that do be done programmatically, the information needs to be accessible from inside the container.

More about `/proc`

In case you wondered, none of the files provided by the cgroup filesystem (/sys/fs/cgroup/memory/memory.*) can be used as a drop-in replacement (i.e.: bind mounted on top of) for /proc/meminfo, or /proc/vmstat. They have different formats and use slightly different names for each metric. Why memory.stat and friends decided to use a format different from what was already being used at /proc/meminfo is beyond my comprehension.

Some of the contents of a /proc filesystem are properly containerized, like the /proc/PID/* and /proc/net/* namespaces, but not all of them. Unfortunately, /proc in general is considered to be a mess. From the excellent “Creating Linux virtual filesystems” article on LWN:

Linus and numerous other kernel developers dislike the ioctl() system call, seeing it as an uncontrolled way of adding new system calls to the kernel. Putting new files into /proc is also discouraged, since that area is seen as being a bit of a mess. Developers who populate their code with ioctl() implementations or /proc files are often encouraged to create a standalone virtual filesystem instead.

I went ahead and started experimenting with that: procg is an alternative proc filesystem that can be mounted inside linux containers. It replaces /proc/meminfo with a version that reads cgroup specific information. My goal was for it to be a drop-in replacement for proc, without requiring any patches to the Linux kernel. Unfortunately, I later found that it was not possible, because none of the functions to read memory statistics from a cgroup (linux/memcontrol.h and mm/memcontrol.c) are public in the kernel. I hope to continue this discussion on LKML soon.

Others have tried similar things modifying the proc filesystem directly, but that is unlikely to be merged to the mainstream kernel if it affects all users of the proc filesystem. It would either need to be a custom filesystem (like procg) or a custom mount option to proc. E.g.:

mount -t proc -o meminfo-from-cgroup none /path/to/container/proc

FUSE

There is also a group of kernel developers advocating that this would be better served by something outside of the kernel, in userspace, making /proc/meminfo be a virtual file that collects information elsewhere and formats it appropriately.

FUSE can be used to implement a filesystem in userspace to do just that. Libvirt went down that path with its libvirt-lxc driver. There were attempts to integrate a FUSE version of /proc/meminfo into LXC too.

Even though there is a very nice implementation of FUSE in pure Go, and that I am excited with the idea to contribute a plugin/patch to Docker using it, at Heroku we (myself included) have a lot of resistance against using FUSE in production.

This is mainly due to bad past experiences with FUSE filesystems (sshfs, s3fs) and the increased surface area for attacks. My research so far has revealed that the situation may be much better nowadays, and I would even be willing to give it a try if there were not other problems with using fuse to replace the proc filesystem.

I am also not comfortable with making my containers dependent on an userspace daemon that serves FUSE requests. What happens when that daemon crashes? All containers in the box are probably left without access to their /proc/meminfo. Either that, or having to run a different daemon per container. Hundreds of containers in a box would require hundreds of such daemons. Ugh.

/proc is not the only issue: `sysinfo`

Even if we could find a solution to containerize /proc/meminfo with which everyone is happy, it would not be enough.

Linux also provides the sysinfo(2) syscall, which returns information about system resources (e.g. memory). As with /proc/meminfo, it is not containerized: it always returns metrics for the box as a whole.

I was surprised while testing my proc replacement (procg) that it did not work with Busybox. Later, I discovered that the Busybox’s implementation of free does not use /proc/meminfo. Guess what? It uses sysinfo(2). What else out there could also be using sysinfo(2) and be broken inside containers?

ulimit, setrlimit

On top of cgroup limits, Linux processes are also subject to resource limits applied to them individually, via setrlimit(2).

Both cgroup limits and rlimit apply when memory is being allocated by a process.

systemd

Soon, cgroups are going to be managed by systemd. All operations on cgroups are going to be done through API calls to systemd, over DBUS (or a shared library).

That makes me think that systemd could also expose a consistent API for processes to query their available memory.

But until then…

Solution?

Some kernel developers (and I am starting to agree with them) believe that the best option is an userspace library that processes can use to query their memory usage and available memory.

libmymem would do all the hard work of figuring out where to pull numbers from (/proc vs. cgroup vs. getrlimit(2) vs. systemd, etc.). I am considering starting one. New code could easily benefit from it, but it is unlikely that all existing tools (free, top, etc.) will just switch to it. For now, we might need to encourage people to stop using those tools inside containers.

I hope my unfortunate realization – figuring out how much memory you can use inside a container is harder than it should be – helps people better understand the problem. Please leave a comment below and let me know what you think.

To be fair, I don’t really know what NewRelic is doing these days, I am just using them as an example. They may be reading memory metrics in a different way (maybe aggregating information from proc/*/smaps). Pardon my ignorance. ↩

Software Development Guidelines

From Merrian-Webster:

guide·line (noun): an indication or outline of policy or conduct

U.S. Dept. of Veterans Affairs:

A guideline is a statement by which to determine a course of action. A guideline aims to streamline particular processes according to a set routine or sound practice. By definition, following a guideline is never mandatory. Guidelines are not binding and are not enforced.

These definitions are very important. What I am listing here are my guidelines, a set of items that usually drive my particular style of coding and engineering. They are not rules. As someone once wisely pointed out (no references, sorry):

“guidelines account for judgement, rules don’t”

They are also my guidelines. It is ok if they do not fit in your case, I am not trying to describe the definitive way of writing, maintaining and operating software. Most of the items here should be obvious for many people, but I hope this helps you think about your own guidelines and understand more others.

After a quick disclaimer, let us go to the list. There is a lot more I would like to publish someday, but I will try to not make this too long and painful.

Extremism

There is a big chance that ying-yang is in my blood; my father is Chinese. One of the most important lessons I learned so far is that extremism is bad, everything needs balance.

This is at the top of my list, mainly because it nicely applies to the other items as well. I consider them to be good ideas, but I will not just blindly apply them to every situation. I usually do, unless I can come up with a very good justification not to.

The recursiveness of this item is also beautiful. It means that you can even be extreme (passionate?) about something, as long as you can give a damn good reason for it. You should not be extreme about not being extreme.

Take for example my deep hate for inheritance (in the OO context): there are good reasons for it. But it is not true that I would never use inheritance: the fact that it should be avoided is something to keep in mind, not something to block you from delivering software.

While we are on the topic, I really like how Go approaches inheritance.

Enabling vs. Directing

I took this very important lesson from some good discussions I had with my friend Tiago many years ago, back when we were coworkers in Germany.

Enabling means that something can be used in many different ways. Even in ways that we have not even considered yet. Humans are creative and will come up with different ways of using and applying an enabling idea. Directing however, means something that is designed to be used in a particular way, or an specific action.

A good example is how to design Java annotations. Here is how we could annotate a class to define that its objects need to be saved to a database (i.e. they are persistent):

@Save
public class Person {
  // ...
}

This is an example of a directing implementation, it specifically tells that objects of this class need to be saved. Using that information for anything else would be awkward.

Here is how the Java Persistence API (thanks to Hibernate) defines it:

@Entity
public class Person {
  // ...
}

This information could be used by many different components on a system. For instance, a logging component could read and use it to determine that some of its fields need to be filtered from logs (such as passwords), just because persistent entities usually contain sensitive data. A formatting component could use different colors for entity objects. There are all sorts of different uses for that information and creative people will come up with even more.

It does not mean that all of those uses will be correct, but that enables more from your code and makes it potentially more flexible.

This concept applies to almost any decision we need to do related to software development. Think about shipping a new feature, designing a component, planning an api, coding styleguides, people and project management styles, technology (framework?) choices, etc. There is probably a more enabling than directing way of doing them all in each context, which would empower people more.

In practice, it can be very hard. I have experienced many cases where it was really hard to tell if we were directing or enabling people. Keeping this in mind already helps a lot though.

Premature Optimization

This has been discussed a lot already, no need to talk too much about it. We all know that “Premature optimization is the root of all evil”.

However, quoting Albert Einstein:

“Everything should be made as simple as possible, but not simpler.”

Do not use premature optimization as an excuse to be lazy, or irresponsible about what you ship. Keep it in mind, but balance it up with your previous experience and feedback from others. There are times when you just know it is going to bite you soon.

Which leads us to the next item…

Productionization

I have learnt a lot of this over the past years running large production services (both at Locaweb and now at Heroku).

From the beginning, think about how your code is going to run in production. Do you understand the platform (runtime) it is going to run on? Are you confortable in troubleshooting hard problems? While we are on the topic, how are you planning to debug those problems?

Are you collecting metrics of how it is being used and how it services its requests? Even if you do not officially publish a SLA, understanding your service times is invaluable to dig into issues that will come with production load. Remember to track not only mean values, mean alone is useless. Always mix it up with variance, or rather with percentiles. Try to target for high percentiles – 95th and 99th are good targets, depending on what your scale is.

Last but not least: are you confident that you can notice (and be notified of) problems before your customers start complaining about your service on twitter? Metrics and automated monitoring are very important to run production services.

Testing

It does not really matter if automated or not. Yes, I said it. I have seen successful software (for whatever that means) both ways. Testing what you ship is just being responsible about it. And that includes having the proper infrastructure to do it: staging environments, gradual rollouts, feature flagging, etc.

Do not get me wrong. Automated testing should be usually preferred, but it is not the ultimate goal. I have seen many “evangelists” speaking hours about how automated testing is important, while the kernel of the operating system they use prefers a more traditional approach with lots of manual (or semi-automated) tests, Q/A testing teams and not many (or even zero) automated tests in its codebase.

Quick note about test (or behavior) driven development: it has more to do with software design methods than with the tests themselves. IMO it is a good practice, which works for me sometimes, but not always in every project, everyday.

Dependency Inversion

I am really proud of what some of my heavy Java development days have taught me.

Designing loosely coupled components is a big one. Dependency Injection, Dependency Inversion Principle and Inversion of Control are all related topics that to me mean a simple thing: design for single (or few) responsibilities.

It means that when you are writing that piece of code (be it a method, function, class, module, script or whatever), focus it on a single responsibility, and keep it small. If it gets to big, break it. If it needs resources to do its job, do not go after the resources it needs there, receive (inject) them instead.

In summary: instead of going after your dependencies, inject them. When a component needs to go after a resource it needs, it usually means that:

The component is now responsible for its dependency lifecycle: this adds more responsibility. Now that it opens that damn connection to your DB (or your queue service), it needs to decide when to close, and needs to know how to do it. Also, what happens when you need to share the same connection in other parts of the system?
The resource needs to be globally accessible: if the component did not create the resource it needs, or if that resource is shared in many parts of the codebase, it needs to be globally available. Hiding the resource creation behind factories does not help when the factory objects themselves need to be globally reachable. I hope I do not need to talk much about how globally accessible things are bad, but it is usually hard to test components using them, and it leads to tighter coupling. Changing that global resource gets harder with more stuff using it.

When you invert control and inject dependencies instead, you have the opportunity to centralize resource management in a single place. That single place centralizes the (single) responsibility of deciding where to inject that resource and how to share it. In more fancy environments, something like this can be called Dependency Injection Framework, but it does not need to be a big bloated framework once you understand the mechanics. In fact, this does not require a framework at all, it is just plain old method/function/constructor invocation with proper parameters.

All in all, let us please not forget about balance. This is not a rule, there are times when what you need is just a goddamn simple function.

Do Not Block The Event Loop

Evented programming (or event-driven programming) is very popular these days and one of the side effects is that some projects will want to be evented without careful consideration.

Going evented or not is one of the big decisions involved in writing code, with big implications. Things blocking the event loop are usually very hard to debug. When you go that route, all code called (including external libraries) needs to be aware of the event loop and not to block it.

There are many advantages though, notably it leads to much more lightweight servers which support much higher concurrency levels. When I am deciding if I should do event-driven code or not, here is what I consider (feel free to add a comment below with your own thoughts):

Is it an event-driven runtime/platform? Nodejs, for example, has been designed from the ground up to be evented. Meaning that all libraries and code written to run on it are already aware of the event loop. If the platform was not designed to be evented (like Ruby with EventMachine) much more care must be taken to not call code which will block the event loop. It is hard to control all the code in libraries included in your project. Take that into consideration.
Consider evented if the piece of code you are writing is mostly a data multiplexer, meaning that it just takes data from one side and sends it to another, acting like a pipe, distributor, load balancer, or router. This type of component is usually I/O bound.
Avoid evented if the piece of code you are writing is CPU bound and does a lot of processing. The chance it will block the event loop is much higher. I have seen projects having to resort to threads (or external processes) to move that part out of the event loop, often leading to spaghetti concurrency very hard to follow. Part evented, part threaded.

There Is Always Something To Learn

As I learn more, I expect this list to change, but I would say that it currently contains the factors influencing my development style the most. It was a very healthy exercise to think about what defines me as a programmer. I hope it is for you too, try it out!

Running Java Web Applications on Heroku Cedar Stack

Update: Heroku now does support Java.

Heroku ~~does not officially support Java applications yet~~ (yes, it does). However, the most recently launched stack comes with support for Clojure. Well, if Heroku does run Clojure code, it is certainly running a JVM. Then why can we not deploy regular Java code on top of it?

I was playing with it today and found a way to do that. I admit, so far it is just a hack, but it is working fine. The best (?) part is that it should allow any maven-based Java Web Application to be deployed fairly easy. So, if your project can be built with maven, i.e. mvn package generates a WAR package for you, then you are ready to run on the fantastic Heroku Platform as a Service.

Wait. There is more. In these polyglot times, we all know that being able to run Java code means that we are able to run basically anything on top of the JVM. I’ve got a simple Scala (Lift) Web Application running on Heroku, for example.

There are also examples of simple spring-roo and VRaptor web applications. I had a lot of fun coding on that stuff, which finally gave me an opportunity to put my hands on Clojure. There are even two new Leiningen plugins: lein-mvn and lein-herokujetty. 🙂

VRaptor on Heroku

Let me show you what I did with an example. Here is a step-by-step to deploy a VRaptor application on Heroku Celadon Cedar:

Go to Heroku and create an account if you still do not have.
We are going to need some tools. First Heroku gems:
```
$ gem install heroku
$ gem install foreman
```
Then Leiningen, the clojure project automation tool. On my Mac, I installed it with brew:
```
$ brew update
$ brew install leiningen
```
The vraptor-scaffold gem, to help us bootstrap a new project:
```
$ gem install vraptor
```

That is it for the preparation. We may now start dealing with the project.

First, we need to create the project skeleton:
```
$ vraptor new <project-name> -b mvn
$ lein new <project-name>
$ cd <project-name>
```
The lein command is not strictly necessary, but it helps with .gitignore and other minor stuff.

Now, the secret sauce. This is the ugly code that actually tries to be smart and tricks Heroku to do additional build/compilation steps:

$ wget https://gist.github.com/raw/1129069/f7ffcda9c42a3004fa8d3c496d4f13887c11ebf4/heroku_autobuild.clj -P src

Or if you do not have wget installed:

$ curl -L https://gist.github.com/raw/1129069/f7ffcda9c42a3004fa8d3c496d4f13887c11ebf4/heroku_autobuild.clj > src/heroku_autobuild.clj

You also need to tweak the Leiningen project definition – project.clj. The template is here. Please remember to adjust your project name. It must be the same that you are using in the pom.xml. Or you may download it directly if you prefer:
```
$ curl -L https://gist.github.com/raw/1129081/5790e173307d28a8df256e0aaa5a9fe7757e922f/project.clj > project.clj
```
Unfortunately, Leiningen comes bundled with an old version of the Maven/Ant integration. The embedded maven is currently at version 2.0.8, which is old. The versions of some plugins configured in the default pom.xml from vraptor-scaffold are incompatible with this old version of maven.
The best way to solve it for now is to remove all <version> tags from items inside the <plugins> section of your pom.xml. This is specific to VRaptor and other frameworks may need different modifications. See below for spring-roo and Lift instructions.

If even after removing versions from all plugins, you still get ClassNotFoundException: org.apache.maven.toolchain.ToolchainManager errors, try cleaning your local maven repository (the default location is $HOME/.m2/repository). The pom.xml that I used is here.
Now, try the same command that Heroku uses during its slug compilation phase. It should download all dependencies and package your application in a WAR file inside the target directory.
```
$ lein deps
```
Confirm that the WAR was built into target/. It must have the same name as defined in your project.clj.
Create your Heroku/Foreman Procfile containing the web process definition:
```
web: lein herokujetty
```
And test with:
```
$ foreman start
```
A Jetty server should start and listen on a random port defined by foreman. Heroku does the same.
Time to push your application to Heroku. Start editing your .gitignore: add target/ and tmp/, and please remove pom.xml. Important: The pom.xml must not be ignored.

You may also remove some unused files (created by leiningen):

$ rm -rf src/<project-name>
# do not remove src/main src/test and src/heroku_autobuild.clj!

Create a git repository:

$ git init
$ git add .
$ git commit -m "initial import"

Push everything to Heroku and (hopefully) watch it get built there!
```
$ heroku create --stack cedar
$ git push -u heroku master
```
When it ends, open another terminal on the project directory, to follow logs:
```
$ heroku logs -t
```
Open the URL that heroku creates and test the application. You may also spawn more instances and use everything Heroku has to offer:
```
$ heroku ps:scale web=2
$ heroku ps:scale web=4
$ heroku ps:scale web=0
```
Quick tip: you do not have SSH access directly to Dynos, but this little trick is very useful to troubleshoot issues and to see how slugs were compiled:
```
$ heroku run bash
```

spring-roo on Heroku

Once you understand the process for a VRaptor application, it should be easy for others as well. Just follow the simple step-by-step here, to create a simple spring-roo maven application.

Then, before running lein deps or foreman start, you must adjust your pom.xml, to remove plugins incompatible with the older mvn that is bundled in Leiningen. Here is the pom.xml that I am using (with some of the plugins downgraded).

You can see my spring-roo application running on Heroku here.

Scala/Lift on Heroku

The same for Scala/Lift: follow the instructions to bootstrap an application with maven. One simple change to the pom.xml is required. My version is here.

You also need to turn off the AUTO_SERVER feature of H2 DB. It makes H2 automatically starts a server process, which binds on a tcp port. Heroku only allows one port bound per process/dyno, and for the web process, it must be the jetty port.

To turn it off, change the database URL inside src/main/scala/bootstrap/liftweb/Boot.scala. I changed mine to a in-memory database. The URL must be something similar to jdbc:h2:mem:<project>.db.

My Lift application is running on Heroku here.

I hope it helps. Official support for Java must be in Heroku’s plans, but even without it yet, we’ve got a lot of possibilities.

DSLs: one interface, multiple implementations

One interface, multiple implementations is one of these design concepts I like most. It’s basically polymorphism in its pure form!

Coding a little bit a while ago, I realized how DSLs could reinforce this idea. My example here is a simple implementation for queue consumers, as a simple Ruby internal DSL:

class GitRepositoryCloner < Consumer
  queue "RepositoriesToBeCloned"
  exclusive true

  handle do |message|
    # git clone repository!
    # I'm pretty sure github has something alike
  end
end

And to enqueue a message, we could do:

GitRepositoryCloner.publish("rails")

GitRepositoryCloner is a simple consumer of the queue named RepositoriesToBeCloned. One of the times I did something similar to this, I needed support for exclusive consumers. Then, my choices were ActiveMQ as the messaging middleware together with the Stomp protocol.

Using them as an example, let’s take a look on a possible implementation for the Consumer class, using the stomp gem:

module ActiveMQ
  class Consumer

    def self.queue(name)
      @queue_name = name
    end

    def self.exclusive(bool)
      @exclusive = bool
    end

    def self.handle(&blk)
      @callback = blk
    end

    def self.listen
      broker = Stomp::Client.new(Config[:broker])
      broker.subscribe(@queue_name, 
          :'activemq.exclusive' => @exclusive) do |message|
        @callback.call(message)
        broker.acknowledge(message)
      end
    end

    def self.publish(message)
      broker = Stomp::Client.new(Config[:broker])
      broker.publish(@queue_name, message, :persistent => true)
    end

  end
end

Consumer = ActiveMQ::Consumer

The last line is where we choose the ActiveMQ::Consumer as the default implementation.

The beautiful aspect of this little internal DSL, composed only of three methods (queue, exclusive and handle), is that it defines an interface. Here, I have seen a common misconception from many developers coming from Java, C# and similar languages which have the interface construct. An interface in Object Orientation is composed of all accessible methods of an object. In other words, the interface is the object’s face to the rest of the world. We are not necessarily talking about a Java or C# interface construct.

In this sense, these three methods (queue, exclusive and handle) are the interface of the Consumer internal DSL (or class object, as you wish).

Let’s say for some reason, we would like to switch our messaging infrastructure to something else, like Resque, which Github uses and is awesome. Resque’s documentation says that things are a little bit different for Resque consumers. They must define a @queue class attribute and must have a perform class method.

As we would do with regular Java/C# interfaces, let’s make another implementation respecting the previous contract:

module Resque
  class Consumer

    def self.queue(name)
      @queue = name
    end

    def self.exclusive(bool)
      self.extend(Resque::Plugins::LockTimeout) if bool
    end

    def self.handle(&blk)
      self.send(:define_method, :perform, &@blk)
    end

    def self.listen
      raise "Not ready yet" unless self.respond_to?(:perform)
    end

    def self.publish(message)
      Resque.enqueue(self, message)
    end

  end
end

There you can see how the implementations differ. The exclusive consumer feature is provided by the LockTimeout plugin. In this case, instead of passing the activemq.exclusive parameter to the connection, we must use the Resque::Plugins::LockTimeout module, as the documentation says. Another key difference is in the message handling process. Instead of passing a handler block to the subscribe method, Resque consumers are required to define a perform method, which we are dynamically creating with some metaprogramming: Class#define_method(name).

Finally, here is how we switch our messaging backend to Resque, without any changes to the consumer classes (GitRepositoryCloner in this example):

Consumer = Resque::Consumer

That’s it: one interface, two (multiple) implementations.

Ruby and dependency injection in a dynamic world

It’s been many years I’ve been teaching Java and advocating dependency injection. It makes me design more loosely coupled modules/classes and generally leads to more extensible code.

But, while programming in Ruby and other dynamic languages, the different mindset always intrigued me. Why the reasons that made me love dependency injection in Java and C# don’t seem to apply to dynamic languages? Why am I not using DI frameworks (or writing simple wiring code as I did many times before) in my Ruby projects?

I’ve been thinking about it for a long time and I don’t believe I’m alone.

DI frameworks are unnecessary. In more rigid environments, they have value. In agile environments like Ruby, not so much. The patterns themselves may still be applicable, but beware of falling into the trap of thinking you need a special tool for everything. Ruby is Play-Doh, remember! Let’s keep it that way.

— Jamis Buck

Even being a huge fan of DI frameworks in the Java/C# world (particularly the lightweights), I completely agree with Jamis Buck (read his post, it’s very good!). This is a recurrent subject in talks with really smart programmers I know and I’ve tried to explain my point many times. I guess it’s time to write it down.

The whole point about Dependency Injection is that it is one of the best ways to achieve Inversion of Control for object dependencies. Take the Carpenter object: his responsibility is to make and repair wooden structures.

class Carpenter {
    public void repair(WoodenStructure structure) {
        // ...
    }
    public WoodenStructure make(Specification spec) {
        // ...
    }
}

Because of his woodwork, the carpenter often needs a saw (dependency). Everytime the carpenter needs the saw, he may go after it (objects who take care of their dependencies on their own).

class Carpenter {
    public void repair(WoodenStructure structure) {
        PowerSource powerSource = findPowerSourceInsideRoom(this.myRoom);
        Saw saw = new ElectricalSaw(powerSource);

        // ok, now I can *finally* do my real job...
    }
}

The problem is that saws could be complicated to get, to assemble, or even to build. The saw itself may have some dependencies (PowerSource), which in turn may also have some dependencies… Everyone who needs a saw must know where to find it, and potencially how to assemble it. What if saw technology changes and some would rather to use hydraulic saws? Every object who needs saws must then be changed.

When some change requires you to mess with code everywhere, clearly it’s a smell.

Have you also noticed that the carpenter has spent a lot of time doing stuff which isn’t his actual job? Finding power sources and assembling saws aren’t his responsibilities. His job is to repair wooden structures! (Separation of Concerns)

One of the possible solutions is the Inversion of Control principle. Instead of going after their dependencies, objects receive them somehow. Dependency Injection is the most common way:

class Carpenter {
    private Saw saw;
    public Carpenter(Saw saw) {
        this.saw = saw;
    }
    public void repair(WoodenStructure structure) {
        // I can focus on my job!
    }
}

Now, the code is more testable. In unit tests where you don’t care about testing saws, but only about testing the carpenter, a mock of the saw can be provided (injected in) to the carpenter.

In the application code, you can also centralize and isolate code which decides what saw implementation to use. In other words, you now may be able to give the responsibility to do wiring to someone else (and only to him), so that objects can focus on their actual responsibilities (again Separation of Concerns). DI framework configuration is a good example of wiring centralized somewhere. If the saw implementation changes somehow you have just one place to modify (Factories help but don’t actually solve the problem – I’ll leave this discussion for another post).

class GuyWithResponsibilityToDoWiring {
    public Saw wireSaw() {
        PowerSource powerSource = getPowerSource();
        return new Saw(powerSource);
    }
    public PowerSource getPowerSource() { ... }
}

Ok. Plain Old Java Dependency Injection so far. But what about Ruby?

Sure you might do dependency injection in Ruby (and its dynamic friends) exactly the same way as we did in Java. But, it took some time for me to realize that there are other ways of doing Inversion of Control in Ruby. That’s why I never needed a DI framework.

First, let’s use a more realistic example: repositories that need database connections:

class Repository {
    private Connection connection;
    public Repository(Connection connection) {
        this.connection = connection;
    }
    public Something find(Integer id) {
        return this.connection.execute("SELECT ...");
    }
}

Our problem is that many places need a database connection. Then, we inject the database connection as a dependency in everywhere it is needed. This way, we avoid replicating database-connection-retrieval code in such places and we may keep the wiring code centralized somewhere.

In Ruby, there is another way to isolate and centralize common behavior which is needed in many other places. Modules!

module ConnectionProvider
  def connection
    # open a database connection and return it
  end
end

This module can now be mixed, or injected in other classes, providing them its functionality (wiring). The “module injection” process is to some degree similar to what we did with dependency injection, because when a dependency is injected, it is providing some functionality to the class too.

Together with Ruby open classes, we may be able to centralize/isolate the injection of this module (perhaps even in the same file it is defined):

# connection_provider.rb

module ConnectionProvider
  def connection
    # open a database connection and return it
  end
end

# reopening the class to mix the module in
class Repository
  include ConnectionProvider
end

The effect inside the Repository class is very similar to what we did with dependency injection before. Repositories are able to simply use database connections without worrying about how to get or to build them.

Now, the method that provides database connections exists because the ConnectionProvider module was mixed in this class. This is the same as if the dependency was injected (compare with the Java code for this Repository class):

# repository.rb

class Repository
  def find(id)
    connection.execute("SELECT ...")
  end
end

The code is also very testable. This is due to the fact that Ruby is a dynamic language and the connection method of Repository objects can be overridden anywhere. Particularly inside tests or specs, the connection method can be easily overridden to return a mock of the database connection.

I see it as a different way of doing Inversion of Control. Of course it has its pros and cons. I can tell that it’s simpler (modules are a feature of the Ruby language), but it may give you headaches if you have a multithreaded application and must use different implementations of database connections in different places/threads, i.e. to inject different implementations of the dependency, depending on where and when it’s being injected.

Opening classes and including modules inside them is a global operation and isn’t thread safe (think of many threads trying to include different versions of the module inside the class). I’m not seeing this issue out there because most of Ruby applications aren’t multithreaded and run on many processes instead; mainly because of Rails.

Regular dependency injection still might have its place in Ruby for these cases where plain modules aren’t enough.

IMO, it’s just much harder to justify. In my experience modules and metaprogramming are usually just enough.

Ruby indentation for access modifiers and their sections

Ruby programmers are very flexible (and permissive) when talking about Ruby code indentation. Most of rubyists I know prefer to indent with two spaces instead of tabs (soft tabs in some editors). There are even some style guides and code conventions published for the language, but none of them is official and they talk too little about code indentation practices.

Things go even worse when we have to choose how to indent private, protected and public sections of classes. Programmers and projects I’ve seen so far seem to adopt different styles. I’ll try to summarize them here:

1. Indent as a new block

This is the most common pattern I see out there. Some programmers prefer to treat sections created by access modifiers as new blocks and indent them:

class MyClass

  def the_public_method
    # ...
  end

  private

    def first_private_method
      # ...
    end

    def second_private_method
      # ...
    end

  protected

    def the_protected_method
      # ...
    end

end

Pros:

Visibility: easy to see if a method has a non-standard access modifier (non-public in most cases).
Produces a very readable code.
Used in the rails codebase (mainly older code).

Cons:

Semantically wrong: access modifiers do not create new scopes. They are simple method calls. Deeper explanation in the next item.
Methods inside the same scope with different indentation levels.
(opinion) One more level of indentation. When you have classes inside modules it starts being a problem: “indentation hell” :-). Your code ends up using its 80 columns very fast and you start having to break lines more often.

2. No indentation

Ruby access modifiers are simple method calls. The Module class (from which Ruby classes inherit) has the private, protected and public methods, that when called with no arguments, simply change the visibility of subsequent defined methods. You may confirm this by testing that the following code works:

class MyClass

  self.send(:private)

  def the_private_method
    # ...
  end

  def another_private_method
    # ...
  end

end

Try to call any of these methods in an instance of the MyClass class and you will confirm they are private.

Because access modifiers are simple method calls, they don’t create a new scope. Semantically speaking, they shouldn’t create another level of indentation. This is even advocated by one of the most important Ruby style guides.

class MyClass

  def the_public_method
    # ...
  end

  private

  def first_private_method
    # ...
  end

  def second_private_method
    # ...
  end

  protected

  def the_protected_method
    # ...
  end

end

Because of its correctness, this was my preferred style until recently, when I found the next ones.

Pros:

It is semantically correct: method calls don’t create new scopes, then method calls shouldn’t increase indentation levels.
(opinion) this style makes access modifiers look like python decorators, Java annotations and C# attributes. IMO, it’s one of the nice things about Ruby: its ability to reproduce most of other languages features, without requiring new syntax or fancy constructs. It’s a simple method call.

Cons:

Hard to see if a method has any non-standard access modifier. In classes with many methods (code smell!) you must constantly scroll to see what access modifier is applied.
Some would argue that this could be solved with proper syntax highlighting or visual method decoration. But it requires editor/IDE support, then I’m considering it as a disadvantage.

3. if-else style

We have a similar case in Ruby: the if keyword creates a new block and has associated else and elsif statements, which are also new blocks. The convention suggests the following indentation (note that if, elsif and else are in the same indentation level):

if something?
  2.times { play }
  jump(2.meters)
elsif other?
  sing and dance
else
  cry
  walk
end

Some of the code recently pushed to Rails 3.0 use the same indentation style for classes with access modifiers. I liked it:

class MyClass

  def the_public_method
    # ...
  end

private

  def first_private_method
    # ...
  end

  def second_private_method
    # ...
  end

protected

  def the_protected_method
    # ...
  end

end

Pros:

Easy to see access modifiers inside classes. With a fast look it is easy to identify sections of private, public and protected methods. They look like blocks.
Readable code. Similar to other Ruby constructs.
Used in the rails codebase (mainly newer code).

Cons:

Still semantically wrong. Access modifiers are inside the scope created by the class keyword. They should be indented.
if, elsif and else are keywords and have special meaning. Access modifiers aren’t and should follow the same convention of other method calls.

4. Indentation with one space

C++ also splits private, protected and public methods in sections. The construct is very similar to Ruby:

class MyClass {
 public:
  MyClass();
  void ThePublicMethod(int n);
  void Print(ostream &output) const;

 private:
  int *Items;
  bool FirstPrivateMethod();
  int SecondPrivateMethod();
};

While reading Google’s style guide for C++ code, their recommendation for public, private and protected sections indentation caught my attention. They suggest that you indent the private, protected and public keywords with only one space.

It seemed a bit awkward in the beginning. But soon, I started liking it because it makes sections easy to identify, access modifiers remain indented inside classes and methods stay all in the same indentation level, as they are all in the same scope.

class MyClass

  def the_public_method
    # ...
  end

 private

  def first_private_method
    # ...
  end

  def second_private_method
    # ...
  end

 protected

  def the_protected_method
    # ...
  end

end

Pros:

Easy to see access modifiers and sections inside classes.
Semantically correct.
All methods remain in the same indentation level.
Google style guide.

Cons:

People are always fighting for 2 spaces vs 4 spaces vs tabs indentation. One space? Very uncommon.
Special treatment for regular method calls. Statements inside the same scope with different indentation levels

My choice

Currently, I tend to like the 3rd (new Rails style) and 4th (Google) styles more. I somehow feel they give the best deal, considering their advantages and disadvantages. Being able to easily identify sections and access modifiers inside classes is very important to me.

In my current project, the team adopted the Google style. I’m happy with it, but you must be flexible in the beginning to adopt one-space-indentation. I won’t lie, it feels strange until you get used. Another issue I have is that my TextMate indent Ruby code with two spaces and treat them as one “step” when navigating with keyboard cursors. Then, I have to type a few more keystrokes to indent access modifiers the way I want. It’s hard to explain, you will have to try it yourself.

And you, what’s your opinion?
How do you indent access modifiers and their sections inside classes?

Soda is cheaper than Water

(at least here in Brazil)

These prices were taken from http://www.paodeacucar.com.br, Sao Paulo, in the date of this post. Compare them:

Soda (Brazilian Guarana):

Versus standard natural Water:

The soda has approximately a R$ 0,92 price/liter ratio, versus R$ 1,25 price/liter of the natural water bottle. Some years ago, I would never imagine soda being so much cheaper than water. This just show us how the world is really turning fast.

Even being more expensive, I have no doubt that water is simpler than soda, and more healthy too. Hey, they use water in soda production process!

My message:

Even when the flavored (more complicated) solution seems to be cheaper and more attractive, many times the simpler solution is more healthy.

Think about it when making your next decision about technology and software design. Please.

Status report: new job, new life

I’m sorry my last post was about three months ago. But, I have a good excuse: I’m just married!
(and I took a nice and fast honeymoon vacation)

Besides that, after three happy years, full time at Caelum, here is the shocking news: I’m now part of the Locaweb team!

This was (and is still being) a very hard decision. People close to me know, that I have a nice and strong relationship with Caelum. Man, I love the company!

I’m feeling very strange, because I’m sad I’m no more full time at Caelum and, at the same time, I’m extremely excited with the new challenges which are about to come. My decision to join Locaweb, which is a really good place to work, just proves I’m very anxious to make a good job there. Other than that, I’m going to sit near very known people as my friend Fabio Akita and Daniel Cukier, just to cite some.

I’m joining the talented Cloud Computing team and I hope I can help them to improve the Cloud Server product. An enormous responsibility!

Locaweb is a huge company and I must admit I’m a little bit scared with the size of things there. They have many teams working on a big diversity of challenging products. They even have their own Datacenter! Locaweb already embraced agile and people there are very open minded, because as a hosting provider, they have to deal with almost all kind of technology.

Because of my work prior to Locaweb, I know the expectations on me are quite high. I like being honest and just to be clear, I’m not better than anyone. I’m relatively new to this area, so I still have too much to learn. Fortunately, I already know many people at Locaweb and I really believe they will help me start being productive. Additionally, there are areas I know I can give something. I’m joining to help, not to prove anything to anyone.

It’s also important to say, that I couldn’t fully leave Caelum. I’m still part of the team, as an instructor, and I keep helping them in many areas, such as the book, courses, textbooks, internal discussions, events, talks and where else I can.

That’s it. Comments, questions and feature suggestions are, as always, welcome. As a cloud provider, I’m happy to hear what would you like to see in a cloud product and what can we do to help you scale and earn profit.

From now on, I stop referring to Locaweb as “they” and instead as “we at Locaweb …”.

Fabio Kung

software, programming, computers, technology

Menu

Holograms and Data Science

Memory inside Linux containers

The problem

Why is this important?

More about `/proc`

FUSE

/proc is not the only issue: `sysinfo`

ulimit, setrlimit

systemd

Solution?

Software Development Guidelines

Extremism

Enabling vs. Directing

Premature Optimization

Productionization

Testing

Dependency Inversion

Do Not Block The Event Loop

There Is Always Something To Learn

DSLs: one interface, multiple implementations

Ruby and dependency injection in a dynamic world

Ruby indentation for access modifiers and their sections

1. Indent as a new block

Pros:

Cons:

2. No indentation

Pros:

Cons:

3. if-else style

Pros:

Cons:

4. Indentation with one space

Pros:

Cons:

My choice

Soda is cheaper than Water

Menu

Just don’t run as root?

The problem

Why is this important?

More about /proc

FUSE

/proc is not the only issue: sysinfo

ulimit, setrlimit

systemd

Solution?

Extremism

Enabling vs. Directing

Premature Optimization

Productionization

Testing

Dependency Inversion

Do Not Block The Event Loop

There Is Always Something To Learn

VRaptor on Heroku

spring-roo on Heroku

Scala/Lift on Heroku

1. Indent as a new block

Pros:

Cons:

2. No indentation

Pros:

Cons:

3. if-else style

Pros:

Cons:

4. Indentation with one space

Pros:

Cons:

My choice

More about `/proc`

/proc is not the only issue: `sysinfo`