Skip to main content
Datasets in Data Lab provide a reusable data layer for working with model inputs, outputs, training data, and production inference data. You can upload your own data, connect data from S3-compatible storage, create datasets from inference logs, and reuse existing datasets for fine-tuning.

​
What is a dataset?

A dataset is a structured collection of rows with a defined schema. Datasets in Data Lab can come from several sources:
  • Uploaded datasets: data uploaded directly to Data Lab.
  • S3-connected datasets: data connected from S3-compatible object storage.
  • Imported inference logs: historical Chat Completions imported from your Token Factory usage.
  • Derived datasets: datasets created by filtering or transforming another dataset.
Once created, datasets appear in the Datasets section of Data Lab and can be inspected, filtered, downloaded, or used for fine-tuning.

​
Creating a dataset

​
Upload a dataset

You can upload structured data directly from your computer. To upload a dataset:
  1. Open Data Lab → Datasets.
  2. Click Add dataset.
  3. Select the file you want to upload.
  4. Specify the dataset name.
  5. Start the upload.
For large datasets, Data Lab uses multipart upload so the file can be transferred in multiple parts instead of requiring a single large request (better to use API). The dataset becomes available after the upload has been processed successfully.

​
Connect a dataset from S3

You can also work with data stored in S3-compatible object storage. This is useful when your datasets are already stored externally or are too large to move between systems as part of every workflow. Data Lab works with the dataset and its metadata without creating an additional raw-data copy of the source S3 object. This allows you to:
  • keep your existing storage architecture;
  • avoid unnecessary data duplication;
  • inspect and process datasets through Data Lab;
  • use the same data for fine-tuning.

​
Create a dataset from inference logs

Data Lab can convert historical Chat Completions into a dataset. You can select a time range and model and create a structured dataset containing available request and response information. Imported data may include:
  • id: string
  • choice_id: integer
  • prompt: json
  • completion: json
  • dialogue: json
  • started: string
  • completed: string
  • raw_prompt: json
  • raw_completion: json
  • completion_tokens: integer
  • prompt_tokens: integer
  • ai_project_id: string
  • model_flavor_id: string
  • tenant_user_account_id: string
  • error_string: string
If Zero Data Retention (ZDR) was enabled when requests were made, those inference logs are not available for import.

​
Supported formats

Data Lab currently supports the following dataset formats:

​
JSON Lines

JSON Lines (JSONL) is supported both for direct uploads and for datasets connected from S3-compatible storage. Each line in a JSONL file must contain a valid JSON object. Example:
JSONL is well suited for model-related datasets because it can represent both simple records and nested structures such as conversation messages.

​
Parquet

Parquet datasets are supported when connected from S3-compatible storage. Direct upload of Parquet files is currently not supported. Parquet is recommended for large structured datasets already stored in object storage, particularly when efficient storage and analytical processing are important.