Bashing irreproducibility with shournal

Abstract

1The Linux shell is arguably one of the most important computational tools across various scientific disciplines. Its high flexibility makes it the platform of choice for many file operations and smaller scripting tasks. Also, many stand-alone programs are called from the Linux shell - typically followed by multiple command-line parameters. However, in larger analysis projects, keeping track of the work quickly becomes challenging, as a typical shell workflow involves the iterative execution of commands with many parameters, modification of scripts, and editing of configuration files. Too often, researchers find themselves in the uncomfortable position that a computational result generated a few weeks ago can no longer be reproduced, despite having taken great care documenting the work manually. On the other hand, there is a lack of tools able to record the researchers shell activity automatically with reasonably low runtime- and storage overhead. To close this critical gap, we developed shournal, a program that tightly integrates with the Linux shell and automatically records every shell command along with the files it reads or writes. Besides logging command- and file metadata, such as working directory, file path, and checksums, shournal can be configured to archive scripts or configuration files that are not regularly under version control via git or svn.