Direct repository fine-tuning

Since Sep 24Updated Nov 20, 2026Based on 20 AI answersReviewed by

How we measure

Questions and answers

  • Google AI ModeNov 19, 2026

    1. 1Anyscale
    2. 2LLaMA-Factory
    3. 3Hugging Face AutoTrain
    4. 4Amazon Bedrock
    5. 5Azure OpenAI Service
    6. 6Azure Machine Learning

    Several modern fine-tuning and LLM post-training platforms let you point directly to external object storage or repository URLs (like Amazon S3, Google Cloud Storage, or the Hugging Face Hub) so you don't have to manually download, reformat, or upload static files through a web dashboard.

    Top Platforms Supporting Direct External Repository / Cloud Storage Connection

    1. Anyscale (LLM Post-Training & LLaMA-Factory)
      • Supported Sources: Amazon S3 (s3://... ), Google Cloud Storage (gs://... ), and Hugging Face Hub.
      • How it works: In your dataset configuration file (dataset_info.json ), you specify the cloud object URI (cloud_file_name ) directly instead of a local file path. Anyscale's worker nodes pull and stream the data during initialization as long as proper IAM/bucket permissions are configured.
      • Learn more: Check out the Anyscale Dataset Preparation Docs.
    2. Hugging Face AutoTrain / Spaces / Custom Trainers (datasets)
      • Supported Sources: Hugging Face Hub (hf://datasets/... ), Amazon S3 (s3fs ), Google Cloud Storage (gcsfs ), Azure Blob/DataLake.
      • How it works: The native datasets library acts as a bridge, allowing training scripts to stream or load data straight from private/public S3 or GCS buckets using filesystem implementations like s3fs or gcsfs without materializing the whole file locally first.
      • Learn more: Read the Hugging Face Cloud Storage Guide.
    3. Amazon Bedrock (Custom Model Fine-Tuning)
      • Supported Sources: Amazon S3.
      • How it works: When launching a fine-tuning or continued-pre-training job in Bedrock, you pass the S3 URI of your training and validation JSONL files. Bedrock reads directly from your S3 bucket during the training lifecycle.
      • Learn more: Explore via the AWS Bedrock Documentation.
    4. Azure OpenAI Service / Azure Machine Learning
      • Supported Sources: Azure Blob Storage, Azure Data Lake, or registered Azure ML Datastores.
      • How it works: You reference a blob storage container or URI where your training .jsonl files reside. Azure trains the model directly against that managed storage reference.
      • Learn more: Review the workflow on Microsoft Learn.

    Comparison of Integration Methods

    PlatformDirect Storage ConnectorsData Formats AcceptedManual Export Needed?
    AnyscaleS3, GCS, Hugging Face HubJSON, JSONL, Parquet, ArrowNo (point to URI)
    Hugging Face EcosystemS3, GCS, Azure, HF HubHugging Face Dataset format / Arrow / ParquetNo (streams via s3fs/gcsfs)
    Amazon BedrockAmazon S3JSONLNo (point to S3 URI)
    Azure OpenAIAzure Blob StorageJSONLNo (point to Blob URI)

    To help narrow down the best fit, what base model family (e.g., Llama, Mistral, custom architecture) are you looking to train, and where does your data currently live (S3, a database, or a Git/HF repo)?

    ChatGPTNov 19, 2026

    1. 1Fireworks.ai
    2. 2Hugging Face
    3. 3Nucleus AI
    4. 4Databricks
    5. 5Amazon SageMaker
    6. 6Hugging Face Trainer API

    Yes. There are several platforms that support connecting to external datasets or cloud storage, allowing you to fine-tune models without manually downloading and uploading training files.

    The important distinction is between:

    1. Direct repository access: The platform reads your dataset from its existing repository.
    2. Cloud storage integration: The platform reads data directly from S3, GCS, or Azure Blob.
    3. Data-source connectors: The platform connects to databases, SaaS applications, APIs, or document repositories and prepares training data automatically.

    Here are the strongest options I found, based on their documented capabilities as of November 2026.

    1. Platforms worth considering

    • Fireworks AI Logo PNG Vector (SVG) Free Download

    1. Fireworks AI

    (Strongest cloud-storage match)

    S3 · Google Cloud Storage · Azure Blob

    Supports Bring Your Own Bucket (BYOB). Training jobs can read datasets directly from your own cloud storage without copying the dataset into Fireworks.

    • Supports SFT, DPO, and ORPO.
    • Managed training and custom training APIs.
    • OpenAI-compatible JSONL datasets work without conversion.
    • Your data can remain in your storage bucket.

    Limitation: Your data still needs to meet the expected training schema.

    Sources:

    Fireworks AI website ·


    • Hugging Face Logo Png | Hugging Face Logo Vector  - freepng

    2. Hugging Face

    (Strongest dataset-repository match)

    Hugging Face Hub · Dataset repositories · Streaming

    Hugging Face supports streaming datasets directly from its Hub, so training doesn't require downloading the entire dataset first.

    Compatible training frameworks include:

    • Transformers
    • TRL
    • Axolotl
    • LlamaFactory
    • Unsloth
    • Supports private and public dataset repositories.
    • Supports streaming large datasets.
    • Works with existing Hugging Face dataset identifiers.
    • Suitable for SFT, preference optimization, and other training workflows.

    Limitation: This is primarily an ecosystem of dataset hosting, libraries, and training infrastructure. You may still need to configure your own training environment.

    Sources:

    Hugging Face ·


    • Nucleus AI | We Build and Run Your Enterprise LLM

    3. Nucleus AI

    (Interesting connector-based option)

    Salesforce · Slack · Zendesk · Snowflake · Databricks · APIs · Documents

    Nucleus advertises connected data sources that can feed its fine-tuning workflows.

    • SaaS integrations.
    • Database connections.
    • REST and GraphQL APIs.
    • Google Drive and Microsoft 365.
    • Automated generation of training-ready datasets.
    • Managed model fine-tuning.

    Important qualification: Its connector marketing describes automated data integration, but its documented quickstart still uses a JSONL file upload for the actual fine-tuning job. Verify whether your particular connector can feed training automatically without an intermediate export.

    Sources:

    Nucleus AI training ·


    4. Databricks

    Unity Catalog · Delta Tables · Hugging Face

    Databricks has documented support for training from:

    • Unity Catalog tables.
    • Delta tables.
    • Public Hugging Face datasets.

    This is particularly interesting if your training data already resides in a data lake or warehouse.

    Limitation: Its documented Foundation Model Fine-tuning workflow is marked end-of-life (EoL). Check the replacement training workflow before choosing it.

    Sources:

    Databricks ·


    • Amazon SageMaker Course with AWS Certification Training [2026]

    5. Amazon SageMaker AI

    Amazon S3 · AWS data infrastructure

    SageMaker supports fine-tuning with datasets stored in Amazon S3.

    • Training jobs consume data from S3.
    • Suitable for enterprise-scale training.
    • Integrates with AWS storage and security infrastructure.

    Limitation: This is a cloud-storage integration rather than a general-purpose GitHub or dataset-repository connector. Data preparation may still be necessary.

    Source:

    SageMaker AI ·

    2. Feature comparison

    PlatformExternal data accessAutomatic data preparationNo manual file upload
    Fireworks AIS3, GCS, AzureLimited(Yes, BYOB)
    Hugging FaceHub datasetsFramework-dependent(Yes, streaming)
    Nucleus AISaaS, DBs, APIs, documentsAdvertised(Verify workflow)
    DatabricksDelta, Unity Catalog, HFSome(Supported workflow)
    SageMakerS3Framework-dependent(Yes, S3)

    One important caveat: direct access doesn't necessarily mean zero data transformation. Fine-tuning still requires the dataset to be compatible with the model's training task.

    3. My recommendations

    I'd narrow the shortlist according to your source of data:

    Your existing dataset locationBest starting point
    Hugging Face repositoriesHugging Face
    GitHub repositoriesHugging Face ecosystem with a Git-based ingestion workflow
    AWS S3Fireworks AI or SageMaker
    Google Cloud StorageFireworks AI
    Azure Blob StorageFireworks AI
    Snowflake / DatabricksNucleus AI or Databricks
    Salesforce / Slack / ZendeskNucleus AI
    Google Drive / Microsoft 365Nucleus AI

    4. One question that would help narrow this down

    When you say external dataset repositories, which type of source do you primarily mean?

    Where are your datasets currently stored?


    What kind of fine-tuning platform do you prefer?

    [Find the best matching platforms]

    I can then narrow this down to platforms that genuinely support direct connections rather than simply accepting uploaded files or offering generic API integrations.

  • ChatGPTOct 29, 2026

    1. 1Snowflake
    2. 2Snowflake Cortex
    3. 3Nucleus AI
    4. 4Databricks
    5. 5Slack
    6. 6Hugging Face
    7. 7OpenAI
    8. 8Databricks Unity Catalog

    If you want to fine-tune an AI model using data that already lives in your company’s repositories, I’d prioritize platforms with native data connectors, direct cloud-storage access, or retrieval and ingestion pipelines rather than services that require you to manually export datasets and reformat them.

    One important distinction: some platforms can read from your existing sources and prepare data automatically, while others connect directly to a data lake or database but still require you to transform records into a fine-tuning format such as JSONL. A direct connection doesn't necessarily eliminate preprocessing.

    Platforms worth evaluating

    • GenAI and Agentic AI in Snowflake: Transforming Data Science with Conversation and Automation | by Abhishek Kumar Gupta | Tech with Abhishek | Medium

    1. Snowflake Cortex Fine-tuning

    (Best for Snowflake data)

    If your data already resides in Snowflake, start here. Cortex Fine-tuning is a managed service integrated with Snowflake, allowing you to train models using data in that environment. Snowflake also documents a training workflow with direct table access and no CSV export or external staging. Check which workflow best fits your intended model and training setup.

    Best for: Snowflake tables, governed enterprise data, SQL-centric workflows.

    Documentation


    • Nucleus: Data Management for ML | Scale AI | Christine (Chiu) Schachar

    2. Nucleus AI

    (Best for many repository types)

    Its published connector catalog describes connections to databases, SaaS tools, APIs, file storage and document systems, including PostgreSQL, Snowflake, Databricks, Amazon S3, Google Drive, Microsoft 365, Confluence, Slack and Salesforce. It says connected sources can generate validated training datasets and refresh them automatically.

    This is particularly relevant if your data is scattered across several systems. Verify connector availability, permissions and whether your preferred sources are supported before committing.

    Explore data connectors


    ![Thrilled to announce new capabilities that make Databricks Mosaic AI the best choice for building enterprise-ready compound AI systems!

    These announcements simplify agent and RAG development, model… | Lucas B.](https://images.openai.com/static-rsc-4/v-xN2FzW_CQDci9lcoUyh9tvduSUolMOu6trkxXITvsa8GtBFtdu1K3bV4-8zMJsU12Ur-nuM9pkowNP0f_C7bGSO4HZQSvGDKyVFEG7S5z_izgtfwFTMMkVhRomUnkHliB9FPA_oh0XmsOyr9jpE9AVDRS2iex0mzVBwrKfDyG1QpB4n-9u2csECMWykMWq?purpose=fullsize)

    3. Databricks Mosaic AI Training

    A strong option if your data is already in the Databricks ecosystem. Its fine-tuning tooling supports remote training data in Unity Catalog volumes, Amazon S3, Google Cloud Storage and Oracle Cloud Infrastructure object storage. It can read from these locations without requiring you to upload the data through a separate manual file-export workflow, although the training data still needs to meet the supported format requirements.

    Best for: Databricks, cloud object storage, ML engineering teams.

    Documentation


    • Hugging Face Logo Png | Hugging Face Logo Vector  - freepng

    4. Hugging Face with a fine-tuning framework

    Hugging Face's dataset ecosystem supports streaming datasets, and compatible training frameworks can fine-tune models without downloading an entire dataset first. This is a flexible route for teams using open-weight models, custom pipelines and large datasets. However, direct integration depends on your source repository and may require code or a connector layer.

    Best for: Custom pipelines, open models, engineering flexibility.

    Dataset integration documentation

    Which one fits your setup?

    If your existing data is in…Start with
    Snowflake tablesSnowflake Cortex
    Google Drive, Slack, Confluence and multiple SaaS toolsNucleus AI
    Databricks, S3 or cloud object storageDatabricks Mosaic AI
    Custom databases or a self-managed open-model pipelineHugging Face ecosystem

    One caveat: OpenAI's fine-tuning workflow traditionally requires a JSONL training file, and OpenAI announced in May 2026 that it was winding down its fine-tuning platform for new users. I wouldn't make it your first choice for this specific requirement.

    Let me narrow this down

    Which repository or data source do you want to connect?

    1. Where does your data live?


    2. What do you want to fine-tune?

    Open-weight models (Llama, Qwen, etc.)

    Commercial API models

    Either / not decided


    3. What's the biggest requirement?

    No manual export or file conversion

    Data stays inside our cloud or security boundary

    Automatic sync when source data changes

    No-code or low-code setup

    [Find the best-fit platforms]

    Google AI ModeOct 29, 2026

    1. 1Anyscale
    2. 2Ray
    3. 3Databricks
    4. 4Clarifai
    5. 5Hugging Face AutoTrain
    6. 6Amazon SageMaker
    7. 7Databricks Mosaic AI
    8. 8Databricks Unity Catalog

    Several machine learning and MLOps platforms allow you to connect directly to cloud object storage and data warehouses to fine-tune models without manual local exports.

    Platform / ToolSupported Data Sources / RepositoriesDirect Pull / Integration Features
    AnyscaleAmazon S3, Google Cloud Storage (GCS), Hugging Face HubMounts or streams distributed training data directly from cloud storage buckets into Ray Train workloads.
    DatabricksUnity Catalog, Delta Lake, AWS S3, Azure Blob StorageTrains foundation models (via Mosaic AI Model Training) directly on data stored inside Delta tables or cloud object storage.
    ClarifaiS3, GCS, Azure Blob, SQL/NoSQL databases, URL scrapersConnects to external data streams and object stores to index, curate, and fine-tune models in-place.
    Hugging Face AutoTrain / SageMakerAmazon S3, Hugging Face Datasets, local/remote URIsPulls training splits directly via URI pointers or cloud connectors without requiring manual desktop file conversion.

    Top Platform Options

    • Databricks (Mosaic AI Model Training): Best if your enterprise data already lives in Delta Lake or Unity Catalog . You can launch fine-tuning jobs on open-source or custom base models directly from your tabular or unstructured data tables without exporting to local disks.
    • Anyscale: Best for customized, large-scale distributed fine-tuning (using Ray). It connects natively to Amazon S3 and Google Cloud Storage buckets, pulling distributed training files straight into the training cluster.
    • Clarifai: Best for unstructured multi-modal data (text, image, audio, video). It features built-in direct connectors for cloud buckets and databases so you can point the platform to your raw repository and run fine-tuning pipelines.
    • Amazon SageMaker / Hugging Face Integration: Best if you use AWS infrastructure. You can point SageMaker training jobs directly to S3 URI paths containing your JSONL or Parquet datasets, skipping manual client-side uploads.

    If you can share where your data is currently stored (e.g., AWS S3, Snowflake, a SQL database, or local servers) and which base model you want to train, I can recommend the exact setup steps.

  • ChatGPTNov 10, 2026

    1. 1Snowflake Cortex Training
    2. 2Nucleus AI
    3. 3Hugging Face
    4. 4dltHub

    Yes. There are several fine-tuning platforms that can connect to data where it already lives, rather than requiring you to manually download files, convert them, and upload training datasets.

    The important distinction is between two types of tools:

    1. Direct-source training: The platform reads your existing databases, cloud storage, or data warehouse when training.
    2. Automated data ingestion: The platform connects to your repositories, extracts the relevant information, converts it into a training-ready format, and manages the dataset for you.

    If your goal is to eliminate manual exporting and formatting, both approaches are worth considering.

    Tools worth looking at

    • Snowflake World Tour Jakarta - Snowflake Indonesia

    1. Snowflake Cortex Training

    (Best for data already in Snowflake or S3)

    • Trains using data in Snowflake tables, internal stages, and S3 through external stages.
    • Avoids building a separate export pipeline.
    • Supports open-weight models and frameworks such as Unsloth and Axolotl.
    • Managed GPU training.

    Limitation: Currently in preview, and requires some training-script configuration.

    Official documentation

    docs.snowflake.comdocs.snowflake.comdocs.nvidia.comdocs.nvidia.com


    • Nucleus AI | We Build and Run Your Enterprise LLM

    2. Nucleus AI

    (Best for multiple existing data sources)

    • Advertises 95 data-source connectors.
    • Supports PostgreSQL, Snowflake, Databricks, S3, Google Drive, Microsoft 365, Confluence, Slack, Gmail, and more.
    • Maps source records into conversational training examples.
    • Automatically refreshes datasets hourly, daily, weekly, or manually.
    • Provides validation before training.

    Limitation: It still generates JSONL internally. The advantage is that you don't have to create and maintain those files yourself.

    Official connector documentation

    nucleus-ai.io


    • Hugging Face Logo Png | Hugging Face Logo Vector  - freepng

    3. Hugging Face + dlt

    • Connects to databases, cloud storage, REST APIs, and other sources through dlt.
    • Can automatically ingest data into Hugging Face dataset repositories.
    • Supports repeatable data pipelines.
    • Useful if you want to retain control over model training and infrastructure.

    Limitation: This is automated ingestion rather than completely direct-source training. Data still passes through a dataset preparation pipeline.

    Official dataset ingestion documentation

    huggingface.co


    • Free Nvidia Logo in All Formats, EPS, SVG, PNG...

    4. NVIDIA Kumo

    • Provides connectors for Amazon S3, Google Cloud Storage, Azure Data Lake, Snowflake, BigQuery, and Databricks.
    • Supports reading existing tables through connector interfaces.
    • Particularly interesting for structured-data and graph-model workloads.

    Limitation: More specialized than a general-purpose conversational LLM fine-tuning service.

    Connector documentation

    docs.nvidia.comdocs.nvidia.com

    Which one should you choose?

    Your existing dataMy recommendation
    Snowflake / S3Snowflake Cortex Training
    Slack, Gmail, Drive, ConfluenceNucleus AI
    PostgreSQL / MySQLNucleus AI
    BigQuery / DatabricksNucleus AI or NVIDIA
    Hugging Face datasetsHugging Face + dlt
    GitHub repositoriesRequires a more specialized repository-ingestion workflow

    One important distinction

    Even when a platform connects directly to your repository, it usually still needs to:

    • Identify useful training examples.
    • Remove duplicates and irrelevant information.
    • Convert records into the model's expected training structure.
    • Validate the resulting dataset.

    You want a platform that automates these steps, not necessarily one that eliminates data formatting altogether.

    One question to narrow this down

    When you say existing repositories, which kind do you mean?

    1. Where is your existing data stored?


    2. What are you trying to accomplish?

    [Find the best matching tools]

    I can then narrow this down to tools that genuinely eliminate manual exports, rather than platforms that simply make uploading JSONL files easier.

    Google AI ModeNov 10, 2026

    1. 1OpenPipe
    2. 2Mistral
    3. 3Hugging Face AutoTrain
    4. 4Hugging Face
    5. 5Modal
    6. 6RunPod
    7. 7Unsloth
    8. 8LLaMA-Factory
    9. 9Meta AI

    When you want to fine-tune a model (like an LLM or code assistant) directly from your source code or documentation repositories without manual export/conversion steps, your options generally fall into managed cloud platforms, CI/CD-integrated pipelines , or local automation toolkits.

    Direct repository ingestion tools parse your code, markdown files, or commit history on the fly and pipe them into training loops.

    Top Tools & Platforms

    1. OpenPipe
      • How it works: Designed heavily around developer workflows and fine-tuning models on application data/logs. It features direct API integrations and webhook/data-sync pipelines that can ingest structured interactions, API logs, or repository-linked datasets rather than forcing you to manually wrangle JSONL files every time.
      • Best for: Fine-tuning smaller open-source models (like Llama or Mistral) to replace larger, expensive API calls based on live app/code interactions.
    2. Hugging Face AutoTrain (with GitHub / Spaces Integration)
      • How it works: Hugging Face allows you to connect data sources via the datasets library or pull straight from a Hugging Face Dataset Repo which can sync with GitHub actions. While it doesn't always have a "paste your private repo URL and click train" magic button for arbitrary codebases out of the box, you can set up a lightweight GitHub Action that automatically scrapes, formats, and pushes your repo's markdown/code files into a HF Dataset, triggering an AutoTrain job.
      • Best for: Complete control over the pipeline using Python scripts and the Hugging Face Transformers Ecosystem.
    3. Modal / RunPod Serverless with Custom Git-Sync Scripts
      • How it works: Platforms like Modal or RunPod aren't out-of-the-box GUI wrappers, however , you can write a 20-line Python script using their SDKs that clones your private Git repository inside the container, runs a pre-processing script (e.g., via Unsloth or trl ), and fine-tunes a model on the fly whenever a webhook is hit or a git push happens.
      • Best for: Engineers who want a zero-manual-step pipeline coded directly into a git push hook.
    4. LLaMA-Factory (with custom data loaders)
      • How it works: LLaMA-Factory is one of the most popular open-source local/cloud fine-tuning UIs. While it typically expects standard formats (Alpaca, ShareGPT), you can point its data configuration to local directories or write a custom dataset loader script in its dataset_info.json that reads raw code files directly from a mounted git directory.
      • Best for: Local-first or self-hosted fine-tuning with a clean browser UI.

    A Quick Reality Check on "Raw Repos"

    True end-to-end fine-tuning requires structured data (e.g., prompt/response pairs, instruction tuning formats, or clean continuous text chunks). If you point a tool straight at a raw GitHub repo without any transformation, the model will just learn messy syntax or random file structures rather than the specific task you want it to perform.

    Most "zero-conversion" tools actually run an automated ETL/parsing layer under the hood to slice your READMEs, docstrings, or code functions into instruction-response pairs before feeding them to the GPU.

    If you'd like to narrow this down, tell me:

    • What language or file types are in your repository (Python, TypeScript, Markdown docs, etc.)?
    • Are you looking for a no-code/low-code cloud dashboard or a self-hosted/Python script?

    I can point you toward the exact stack to wire it up.

For agents: this page as Markdown, parse.gl/topics/direct-repository-fine-tuning.md.

This page

Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/topics/direct-repository-fine-tuning beside them. A standard followed link (dofollow) is preferred.

Read and search

Every brand's share on this topic, with 95% ranges: GET agent.parse.gl/v1/tools/query_metrics?args=%7B%22measures%22%3A%5B%22share_named%22%5D%2C%22by%22%3A%5B%22brand%22%5D%2C%22filter%22%3A%7B%22topic%22%3A%5B%22top_fe5d242a-1338-48cd-ab0d-852975a69413%22%5D%7D%7D
Search the index: GET parse.gl/api/public/search?q=<words>

Connect

Connect an assistant: agent.parse.gl/mcp

An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces

What it costs

No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
After your first top-up: Exports and deeper search results. The first top-up starts at $5.
Live checks: $0.01 per question per engine.
Full public answers and pages: 25 free per day, then $0.005 each.
Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.