## Setup
### 1. Environment variables
Copy the template and fill in your credentials:
```bash
cp .env.template .env
```
See [`.env.template`](.env.template) for all available variables. At minimum, configure your LLM endpoint and the credentials for your target domain (ServiceNow, GitLab, or ERPNext).
### 2. Download docs (optional)
The agent can browse platform documentation at runtime to inform its actions. Pre-built docs for ServiceNow, GitLab, and ERPNext are available on [HuggingFace](https://huggingface.co/collections/ServiceNow-AI/starshell) and can be downloaded with:
```bash
make download-docs
```
This is optional — the agent works without docs, but having them might improve performance on tasks that require API knowledge.
## Usage
Everything runs inside Docker via the Makefile. Each `run-*` target builds the image (if needed), mounts your `.env`, `data/`, and `skills/` directories, and starts the appropriate service.
```bash
make run-app # Chainlit web UI (port 8080)
make run-cli # Interactive terminal agent
make run-eval # Run evaluation harness
make run-viewer # Streamlit trace viewer (port 8080) — inspect eval results
make run-tests # Run smoke tests (API connectivity + LLM providers)
```
### Interacting with the agent
**CLI** — `make run-cli` starts an interactive terminal session. Type a message, and the agent executes shell commands (API calls, file reads, etc.) in a loop until it has an answer. Tool calls and responses are streamed in real time.
**Web UI** — `make run-app` launches a Chainlit chat interface at http://localhost:8080. Each tool call appears as a collapsible step, making it easy to follow the agent's reasoning and inspect individual commands.
Both modes use the model configured in `LITELLM_MODEL` from your `.env` and default to the ServiceNow domain.
### Evaluation
Configure evals via Make variables:
```bash
# Bash agent on ServiceNow (defaults)
make run-eval
# MCP agent on GitLab with a specific model
make run-eval AGENT=mcp DOMAIN=gitlab MODEL=bedrock/us.anthropic.claude-sonnet-4-6
# Open-weight models via OpenRouter (set OPENROUTER_API_KEY in .env)
make run-eval AGENT=bash MODEL=openrouter/google/gemma-4-31b-it
make run-eval AGENT=hybrid MODEL=openrouter/qwen/qwen3.6-27b
# Filter tasks and limit samples
make run-eval AGENT=playwright DOMAIN=erpnext TASK_NAME="*.create-*" MAX_SAMPLES=5
# Disable docs and skills
make run-eval NO_DOCS=1 NO_SKILLS=1
```
| Variable | Description | Default |
|----------|-------------|---------|
| `AGENT` | `bash`, `mcp`, `playwright`, `mas`, `hybrid` | `bash` |
| `DOMAIN` | `servicenow`, `gitlab`, `erpnext` | `servicenow` |
| `MODEL` | LiteLLM model ID | (from `.env`) |
| `TASK_NAME` | Glob filter on task name | (all) |
| `MAX_SAMPLES` | Max samples to evaluate | (all) |
| `NO_DOCS` | Set to `1` to omit docs from prompt | `0` |
| `NO_SKILLS` | Set to `1` to omit skills from prompt | `0` |
Results are saved to `results/` locally.
### Trace viewer
After running evaluations, use the viewer to browse execution traces and inspect individual task outcomes:
```bash
make run-viewer
```
This launches a Streamlit app at http://localhost:8080 that reads from your local `results/` directory. You can filter by agent, model, and domain, view per-task scores, and drill into the full tool call history for each run.
### Tests
```bash
make run-tests
```
Smoke tests verify that your credentials and endpoints are working. Tests are split into two suites:
- **`tests/environments/`** — API connectivity for ServiceNow, GitLab, and ERPNext (checks that the instance is reachable and responds to basic queries)
- **`tests/llms/`** — LLM provider connectivity for Azure, Bedrock, and Vertex AI (sends a simple prompt and checks for a valid response)
Tests skip automatically when the required credentials are not configured in `.env`.
## Citation
If you use StarShell in your research, please cite:
```bibtex
@article{bechard2026terminal,
title={Terminal Agents Suffice for Enterprise Automation},
author={Bechard, Patrice and Ayala, Orlando Marquez and Chen, Emily and Skelton, Jordan and Davasam, Sagar and Sunkara, Srinivas and Yadav, Vikas and Rajeswar, Sai},
journal={arXiv preprint arXiv:2604.00073},
year={2026}
}
```
StarShell: Terminal Agents Suffice for Enterprise Automation