> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/YerevaNN/ChemLactica/llms.txt
> Use this file to discover all available pages before exploring further.

# Training Data

> Overview of the PubChem training corpus and data format used for ChemLactica models

## Training Corpus

All ChemLactica and Chemma models are trained on a comprehensive corpus derived from [PubChem](https://pubchem.ncbi.nlm.nih.gov/), the world's largest collection of freely accessible chemical information.

> **Corpus Size**
>
> **40 billion tokens**
>
> Covering 100M+ molecules

> **[Dataset on HuggingFace](https://huggingface.co/datasets/yerevann/PubChemForLM)**
>
> **yerevann/PubChemForLM**
>
> Publicly available dataset

## Data Format

### JSONL Structure

The training data is stored in JSONL (JSON Lines) format, with each line representing one molecular entry:

```python chemlactica/get_dataset.py theme={null}
training_data_files = glob.glob(training_data_dir + "/*.jsonl")

dataset = IterableDataset.from_generator(
    samples_generator,
    gen_kwargs={
        "files": training_data_files,
        "shared_jsonl_files": shared_jsonl_files,
    },
)
```

### Example Entry

Each molecule in the corpus contains multiple fields:

```json theme={null}
{
  "CID": 523129,
  "SMILES": "CCCCCCOC(=O)NC1=CC=C(C=C1)N=NC2=CC=CC=C2",
  "synonyms": [{"name": "p-Phenylazo carbanilic acid, n-hexyl ester"}],
  "related": [
    {"SMILES": "CCCCOC(=O)NC1=CC=C(C=C1)N", "similarity": 0.7}
  ],
  "SAS": 2.0,
  "WEIGHT": 325.1,
  "TPSA": 63.06823,
  "CLOGP": 6.231230123,
  "QED": 0.46,
  "NUMHDONORS": 1,
  "NUMHACCEPTORS": 4,
  "NUMHETEROATOMS": 5,
  "NUMROTATABLEBONDS": 8,
  "NOCOUNT": 5,
  "NHOHCOUNT": 1,
  "RINGCOUNT": 2,
  "HEAVYATOMCOUNT": 24,
  "FRACTIONCSP3": 0.32,
  "NUMAROMATICRINGS": 2,
  "NUMSATURATEDRINGS": 0,
  "IUPAC": "hexyl N-(4-phenyldiazenylphenyl)carbamate",
  "experimental": [
    {
      "PROPERTY_NAME": "Kovats Retention Index",
      "PROPERTY_VALUE": "Semi-standard non-polar: 4303"
    }
  ]
}
```

## Data Processing

### Text Formatting

Raw JSON data is converted to formatted text strings for training:

```python chemlactica/utils/text_format_utils.py theme={null}
def generate_formatted_string(compound_json, rng, model_config):
    key_value_pairs = []
    
    # SMILES may appear first (50% chance)
    key = "SMILES"
    value = compound_json.get(key, "")
    if rng.integers(2) == 0:
        if value:
            key_value_pairs.append(format_key_value(key, value, rng))
            del compound_json[key]
    
    # Shuffle remaining keys for variety
    keys = list(compound_json.keys())
    rng.shuffle(keys)
    
    for key in keys:
        key_value_pairs.append(format_key_value(key, compound_json[key], rng))
    
    compound_formatted_string = "".join(key_value_pairs) + model_config.separator_token
    return compound_formatted_string
```

> [!NOTE]
>
> The order of properties is randomized during training to prevent the model from learning order-dependent patterns.

### Property Formatting

Different types of data are formatted with special tags:

```python chemlactica/utils/text_format_utils.py theme={null}
def format_key_value(key, value, rng):
    if key == "related":
        # Similar molecules with Tanimoto scores
        if len(value) > 10:
            value = rng.choice(value, size=10, replace=False, shuffle=False)
        for pair in value:
            rounded_sim = "{:.2f}".format(float(pair["similarity"]))
            formatted_string += f"[SIMILAR]{pair['SMILES']} {rounded_sim}[/SIMILAR]"
            
    elif key == "experimental":
        # Experimental properties
        for pair in value:
            formatted_string += f"[PROPERTY]{pair['PROPERTY_NAME']} {pair['PROPERTY_VALUE']}[/PROPERTY]"
            
    elif key == "synonyms":
        # Chemical names
        for val in value:
            formatted_string += f"[SYNONYM]{val['name']}[/SYNONYM]"
    else:
        # Standard properties (QED, SAS, etc.)
        if SPECIAL_TAGS[key].get("type") is float:
            value = "{:.2f}".format(float(value))
        start = SPECIAL_TAGS[key]["start"]
        end = SPECIAL_TAGS[key]["end"]
        return f"{start}{value}{end}"
```

### Example Formatted Output

```text theme={null}
[START_SMILES]CCCCCCOC(=O)NC1=CC=C(C=C1)N=NC2=CC=CC=C2[END_SMILES]
[SAS]2.00[/SAS]
[WEIGHT]325.10[/WEIGHT]
[TPSA]63.07[/TPSA]
[CLOGP]6.23[/CLOGP]
[QED]0.46[/QED]
[SIMILAR]CCCCOC(=O)NC1=CC=C(C=C1)N 0.70[/SIMILAR]
[SYNONYM]p-Phenylazo carbanilic acid, n-hexyl ester[/SYNONYM]
[NUMHDONORS]1[/NUMHDONORS]
[NUMHACCEPTORS]4[/NUMHACCEPTORS]
</s>
```

## Data Types

The corpus supports different data types:

```python chemlactica/utils/dataset_utils.py theme={null}
DIR_DATA_TYPES = ["pubchem", "assay", "assay-split", "custom"]
```

> **PubChem Data**
>
> Standard PubChem molecular data with computed properties and similar molecules.

> **Assay Data**
>
> Experimental assay data with biological activity measurements:
>
> ```
> [ASSAY_NAME]Assay Name[/ASSAY_NAME]
> [ASSAY_DESC]Description[/ASSAY_DESC]
> [VAR_NAME]Variable[/VAR_NAME]
> [VAR_VAL]Value[/VAR_VAL]
> ```

> **Custom Data**
>
> User-provided molecular data in the same JSONL format.

## Dataset Loading

### For Pre-training

```python chemlactica/get_dataset.py theme={null}
def get_dataset(
    train_type,
    training_data_dirs,
    valid_data_dir,
    dir_data_types,
    train_config,
    model_config,
    shared_jsonl_files,
    evaluate_only,
    slurm_eval,
    shuffle_buffer_size,
):
    if train_type == "pretrain":
        assert len(training_data_dirs) == len(dir_data_types)
        train_dataset_dict = {}
        
        for i, (training_data_dir, dir_data_type) in enumerate(
            zip(training_data_dirs, dir_data_types)
        ):
            training_data_files = glob.glob(training_data_dir + "/*.jsonl")
            dataset = IterableDataset.from_generator(
                samples_generator,
                gen_kwargs={
                    "files": training_data_files,
                    "shared_jsonl_files": shared_jsonl_files,
                },
            )
            dataset = process_dataset(
                dataset=dataset,
                train_config=train_config,
                model_config=model_config,
                process_batch_sizes=(50, 50),
                is_eval=False,
                assay=is_assay_split,
            )
```

### For Fine-tuning

```python theme={null}
if train_type == "sft":
    dataset = load_dataset(training_data_dirs[0])
```

## Tokenization

### Custom Tokenizer

ChemLactica uses specialized tokenizers optimized for chemical notation:

```python chemlactica/utils/utils.py theme={null}
default_tokenizer_path = "./chemlactica/tokenizer/ChemLacticaTokenizer66"

def get_tokenizer(tokenizer_path):
    return create_tokenizer(tokenizer_path)

def create_tokenizer(tokenizer_path):
    tok = AutoTokenizer.from_pretrained(tokenizer_path)
    tok.add_bos_token = False
    tok.padding_side = "right"
    return tok
```

### Vocabulary Size

> **Chemlactica**
>
> **50,000 tokens**
>
> Optimized for chemistry

> **Chemma**
>
> **256,000 tokens**
>
> Larger vocabulary from Gemma

### Special Tokens

The tokenizer includes chemistry-specific special tokens:

```python chemlactica/utils/utils.py theme={null}
def get_start2end_tags_map(tokenizer_path: str = default_tokenizer_path):
    with open(os.path.join(tokenizer_path, "special_tokens_map.json"), "r") as _f:
        special_tokens_map = json.load(_f)
    additional_tokens = special_tokens_map.get("additional_special_tokens", None)
    n = len(additional_tokens)
    return {
        additional_tokens[i]: additional_tokens[n // 2 + i] for i in range(n // 2)
    } | {"[START_SMILES]": "[END_SMILES]"}
```

## Data Statistics

### Corpus Coverage

* **100M+ unique molecules** from PubChem
* **40B tokens** total
* **Multiple properties** per molecule (avg 10-15 properties)
* **Similar molecules** included for \~70% of compounds
* **Chemical synonyms** and IUPAC names when available

### Property Distribution

All molecules include:

* SMILES notation (100%)
* Core properties: SAS, WEIGHT, TPSA, CLOGP, QED (>95%)
* Structural counts (>95%)
* Similar molecules with Tanimoto scores (\~70%)
* Chemical names and synonyms (\~60%)
* Experimental properties (varies)

## Training Configuration

### Block Size

```yaml theme={null}
block_size: 2048  # Maximum sequence length
```

> [!TIP]
>
> The 2048 token context window is sufficient for most molecules and their associated properties. Very large molecules or those with many similar molecules may be truncated.

### Data Processing Parameters

```python theme={null}
process_batch_sizes=(50, 50)  # Batch sizes for processing
is_eval=False                  # Training vs evaluation mode
assay=False                    # Standard PubChem vs assay data
```

## Accessing the Dataset

### From Hugging Face

```python theme={null}
from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("yerevann/PubChemForLM")

# Load streaming for large-scale training
dataset = load_dataset("yerevann/PubChemForLM", streaming=True)
```

### Local JSONL Files

```python theme={null}
import glob
from datasets import load_dataset

training_data_files = glob.glob("path/to/data/*.jsonl")
dataset = load_dataset(
    "text",
    data_files={"train": training_data_files},
    streaming=True
)
```

## Data Quality

> **Validation**
>
> * All SMILES validated with RDKit
> * Properties computed from validated molecular structures
> * Similar molecules verified with Tanimoto similarity
> * Duplicate molecules removed via canonical SMILES

> **Preprocessing**
>
> * Empty fields removed
> * Float values rounded to 2 decimal places
> * Related molecules limited to 10 per entry
> * Property order randomized for training diversity

## Next Steps

> **[Model Architectures](/concepts/models)**
>
> Learn about the models

> **[Pre-training](/guides/pretraining)**
>
> Train your own model

> **[Fine-tuning](/guides/fine-tuning)**
>
> Adapt to your dataset

> **[SMILES Format](/concepts/smiles-format)**
>
> Understand molecular representation


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.