Diving into Dependencies in Elixir

When we migrated our API from PHP to Elixir, we expected it would just be a matter of time until we ran into an issue with a dependency - Elixir is a much newer language, and has fewer mature libraries. Our full-text search engine, Sphinx, ended up being the first dependency that gave us issues. In this article we'll talk about how we were able to quickly and easily fix these problems for ourselves (and everyone else using Sphinx).

When you run into an apparent bug in a MySQL driver in PHP, the easy options quickly become exhausted: verify the bug is actually in driver code, and try updating the driver. Debugging further usually involves dusting off your copy of K&R and diving into C code. Due to the fact that mariaex, the de-facto Elixir MySQL library, is written in pure Elixir, we were able to look through the code and make fixes with very little additional context required.

All it took to change our project’s dependency was to fork mariaex and tweak one line in our project’s mix.exs. We just needed to set override: true in order to correctly resolve other dependencies which also depend on mariaex.

Original mix.exs #

{:mariaex, "~> x.y.z"}

Local Development #

{:mariaex, path: "../mariaex", override: true}

Being able to inspect and sift through the code in a controlled environment made the debugging simple. The issue was a bug in parsing the handshake packet during the initial connection to Sphinx, which doesn’t manifest when using MySQL Server. In short, we isolated the problem and fixed it.

It took some time to connect with the maintainer of mariaex and get that patch upstreamed. Once we got in touch, there were some interface-breaking changes which we needed to iron out before we could merge. In the meantime, we just maintained our own internal fork with another small change to mix.exs:

Private Fork #

{:mariaex,
 git: "git://github.com/TeachersPayTeachers/mariaex.git",
 branch: "master",
 override: true}

The solution above let us proceed rapidly without being blocked on mariaex releasing a new version, but we ultimately decided to proceed in a different direction. Since our first contribution was merged, we have run into and fixed further issues with Sphinx communication via the MySQL4.1 protocol. Our fixes are not interface-compatible with mainline mariaex, and the cost of maintaining our fixes is increasing with every new commit to either repo. Sphinx recently released its HTTP API to beta, motivating us to investigate a switch to an HTTP client library and drop our mariaex fork in the near future.

Our experience with Elixir has opened up a new development pattern for us. As an organization, we don't feel constrained by deficiencies or bugs in open source packages that we use. We can adopt any library; if it doesn't work properly, we can quickly dive in and fix it.