Project Silkworm

Snapshot from the pipeline's local stats, covering runs from 2026-06-08 to 2026-08-18. The Hub dataset is the source of truth for current size.

How it works

A loop that generates, filters, batches and uploads. No human annotation.

1. Sample a prompt. topics.py picks a domain, a topic (about 50 per domain) and one of 30 prompt templates, e.g. “What is {topic} and why does it matter?”
2. Generate twice. generator.py calls Llama 3.1 8B Instruct through the HF Inference API: once as a thorough expert (chosen, up to 512 tokens), once as a hurried, superficial assistant (rejected, up to 256 tokens). Retries use exponential backoff.
3. Filter. Pairs are dropped if the chosen answer is under 200 characters, the rejected under 30, or both are identical. Prompts already in the dataset are skipped.
4. Batch and upload. Accepted pairs go to data/pending.jsonl. Every 50 pairs uploader.py merges them into a Parquet file and pushes it to the Hub.
5. Watch it run. main.py shows a live terminal dashboard; Ctrl+C stops cleanly and uploads the final batch.

Domains

Example pairs

Real rows from the dataset (chosen answers trimmed for display).

Chosen answers are fluent but not fact-checked. Some contain errors. See limitations below.

Use the dataset

Columns: prompt, chosen, rejected, topic, timestamp. Works with TRL's DPOTrainer.

from datasets import load_dataset

ds = load_dataset("iPwnds/Project-Silkworm", split="train")
print(ds[0]["prompt"])

Licence: CC BY 4.0, plus the Llama 3.1 Community License because outputs come from Llama 3.1. Code is MIT.

Run your own

Needs Python 3.10+ and a Hugging Face token with write access to a dataset repo.

git clone https://github.com/lumenworksco/-Project-Silkworm.git Project-Silkworm
cd Project-Silkworm
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # set HF_TOKEN and HF_DATASET_REPO
python main.py

Limitations