Recently we announced an MCP server for DocDB. While observing our user base adopt the tool it occurred to me that it closely matches the paradigm of the Unix pipe. It is, fundamentally, a new category of “glue” that I expect to be widely adopted in the next few years.

The Unix pipe (|) is a magical character that allows you to compose powerful one-off pipelines by combining several programs together. It illustrates an philosophy of widely referred to as the “Unix philosophy.” This philosophy can be boiled down to the idea that “the power of a system comes more from the relationships among programs than from the programs themselves.”

This philosophy has four tenets, but the first two are most relevant to what I see LLM tooling doing right now.

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.
2. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.

These two principles are very much internalized by the tools available in the command line. It’s something that I have very much internalized as I’ve spent much of my career working from a command line interface (CLI). ¹

Claude² kicks this idea up a notch. Now, we have a general purpose tool that starts from a basis of fuzzy randomness and allows for users to not only evaluate text but create their own deterministic runtimes and scheduled routines, redirecting output from one utility to multiple other utilities, combining them in ways that creates additional value. Previously, one had to write the programs and stand up the infrastructure to do run it regularly. Now you can accomplish the same thing in a modern harness.

Now, there are quite a few caveats here. First, LLMs are by their very nature stochastic. The same input today might result in different output tomorrow. This can be challenging for some data pipelines that require predictable behavior. Technologists will need to become familiar with understanding when a job to be done cannot be accomplished probabilistically and must be built deterministically.

Second, Claude routines are typically built and managed by a single user. This is a problem if the builder/maintainer leaves the company and the routine has become business critical. This creates a “ghost infrastructure” of business critical pipelines that will be mostly invisible to traditional IT teams. This will be brittle and cause much consternation over the next decade.³

Even with the caveats, the usefulness of this paradigm guarantees that it will see wide adoption soon. Ideally, we will see additional infrastructure built to address the potential problems while still reaping the advantages.

Header Image by Heino Esiner on Unsplash