What it can do
The features, grouped by what you need them for. You do not have to use them all: each study switches on only the parts it needs.
Finding and keeping participants
- Sign-up forms in Turkish and English, with eligibility questions and a waiting list when the study is full.
- Consent forms where each item is agreed to separately (for example "my chat messages may be analysed"), with every version kept.
- A personal portal for each participant: tasks, team, uploads, questionnaires, hand-ins, and their own data to download.
- Withdrawal and erasure on request, carried out across every part of the system.
Working in teams
- Balanced teams, formed by the ToolKit and checked by you.
- Reshuffles between phases, for studies on how teams adapt.
- A Zulip chat per team, with accounts and nicknames created for participants.
- A shared Etherpad document per team and task, with every change kept.
Recording the process
- Stone Guest, a keystroke logger for macOS and Windows that works offline and saves one file per session.
- Window categories (translation, source, search, other), so you know where the time went.
- Pad changes matched to keystrokes, so you know exactly where in the text each key landed.
- Optional: skin conductance from an EmbracePlus wristband, and the participant's own ChatGPT or Claude conversations around a session.
- Use of machine translation and AI tools, tagged by app and website, with a study policy (allowed, forbidden, logged).
Questionnaires and tests
- A question bank with Likert scales, choices, open answers, numbers and dates, and simple "show this if" rules.
- Ready instruments: LexTALE, NASA-TLX, a background questionnaire, a post-task reflection, a team process questionnaire.
- Personal links with opening and closing times, sent by email or shown on the portal.
Running the study
- Automation: reminders and messages on a schedule or when something happens, with quiet hours.
- An outbox of everything sent, and every message recorded as an intervention.
- Quality flags for missing uploads, short sessions, chat activity without a session, and more.
- Notes, memos and incidents, with protocol deviations kept for the report.
- Deliverables: drafts and final translations handed in as numbered versions, with deadlines.
Analysing
- The Task Segment Framework: 513 published process indicators, plus your own and the ToolKit's extensions.
- A lock on your choice of indicators before analysis, recorded with a fingerprint.
- Session replay, keystroke by keystroke, and coding rounds with agreement statistics.
- Inputlog-style pauses, bursts and revisions.
- Reply networks and response times from the team chats.
- Rating rounds for the translations, with a rubric or error annotation, and inter-rater agreement.
- Tidy CSV exports with a codebook, ready for R or Python.
Reporting and sharing
- A protocol report that drafts your methods section and a preregistration update, as Word or Markdown.
- Dataset releases for OSF: the parts you choose, new codes for every participant, a codebook and checksums.
- Interviews, including cued retrospective interviews with the session replay as the cue.
- Certificates of participation and a record of compensation.
Looking after the system
- Installed on one server with one command, upgraded from the panel, with automatic rollback if an upgrade fails.
- Nightly backups and a monthly restore test.
- Health checks and email alerts for the things that can go wrong.
- An audit log of who did what, which cannot be edited.
Some of these (rating rounds, interviews, certificates) arrive with the first full version of the ToolKit.