CLTL

Publications

CLTL strives for sharing resources and software and making sure our research results are replicable. We therefore have the following guidelines. Principles 1. and 2. are required for any publication by a member of the CLTL group.

Principles

  1. All publications are indicated on the CLTL website with a link to the actual publication (if permitted).
  2. All published experiments must be replicable. For any experiment, the following should be stored and made accessible where possible:
    • Your original research question and hypothesis
    • Data
    • Resources
    • Software
    • Results
    • Readme with necessary instructions
  3. Ideally, the historic development of the experiment is also provided.
    • Your research question and hypotheses may change during the course of your research. You should always be open about your original hypothesis, if your questions and hypotheses change, explain why.

Recommended Workflow

  1. Define your research question and hypothesis (or hypotheses). (obligatory)
  2. Before you start working on an experiment, create a directory in your favorite version control system that you can use to run your experiment. (recommended)
  3. Setup the experiment in this directory, so that it contains all code, scripts and data created especially for the experiment and points clearly to all other code and resources you use. (recommended)
  4. Add a file called \textit{notes.txt} where you briefly describe the steps you take, reasons for adapting code or rerunning the experiment. (recommended)
  5. Check in your setup and results after each run. (recommended)
  6. When you are done and have the results you’ll describe in a paper, publish the experiment carefully checking the points listed below. (obligatory)

Publishing experiments

For any published experiment, all materials required to repeat the experiment must be archived and, whenever possible, made accessible.

Software, resources and datasets can be stored at the CLTL github and CLTL SciStor.

The following components are typically required to ensure replicability of an experiment. To be on the safe side, ask a colleague to carry out the experiment from scratch to verify if you have provided all components and described all steps clearly.

Accessibility

Unless project partners require otherwise, all software, data and resources developed at CLTL should be publicly available.
We generally release software and resources under the Apache license.

If you use data, software or resources you cannot make available:

  1. Make sure you clearly indicate what you used (including the version) and how one may gain access to it (e.g. where to get the license).
  2. If you made any changes, make sure that anyone who has the original data/software/resource can recreate the version you used either by running a script you provide or by following a detailed description.

Historic development

Our 8-page limited papers are typically too short to provide all experimental details. The steps that were taken to get to a final (reported) experiment can provide useful insights to other researchers and help prevent errors in your own experiment.

A full report of the historic development starts with the hypothesis that inspired the research, including a brief explanation of where you based your hypothesis on. It reports each step that was taken, including errors.

Ideally, there would be:

If you setup a version control system for your experiment when you begin working, it will be less work to publish the experiment once your done (and you’re probably dealing with deadline stress…).