Direct repository fine-tuning
Since Sep 24Updated Nov 20, 2026Based on 20 AI answers
Reviewed by Dimitry Apollonsky
Brands AI recommends
Also mentioned rarely: Google Cloud Gemini Enterprise Agent Platform, Nucleus AI, Databricks Unity Catalog, Fireworks.ai, Amazon Bedrock, LLaMA-Factory, Anyscale, Hugging Face Trainer API, Databricks Mosaic AI, GitHub Actions, Modal, OpenPipe, RunPod, Axolotl, Google Cloud BigQuery, Snowflake, Unsloth, Microsoft Foundry, Clarifai, Databricks Model Training, Snowflake Cortex Training, Snowflake Cortex Fine-tuning, DVC, ForgeAI, LangSmith, Meta-pytorch, Ray, Snowflake ArcticTraining, Snowflake Cortex, OpenAI, Azure OpenAI Service, Google Cloud Storage, Meta AI, Amazon S3, Azure Machine Learning, Mistral, OpenAI API, Together AI, AWS, Amazon Nova, Azure Data Lake Storage, Azure Blob Storage, Baseten, Claude Code, Claude, Claude Code Action, Cursor, dltHub, Amazon SageMaker Unified Studio, Meta LLaMA 4, Gemini, LitGPT, DocsGPT, GitLab, LangChain, OpenAI Platform, Qwen, Slack
- DatabricksChatGPTChatGPT 60%Google AI ModeGoogle AI Mode 20%
- Hugging FaceChatGPTChatGPT 50%Google AI ModeGoogle AI Mode 10%
- Amazon SageMakerChatGPTChatGPT 40%Google AI ModeGoogle AI Mode 20%
- Hugging Face HubChatGPTChatGPT 30%Google AI ModeGoogle AI Mode 30%
- Hugging Face AutoTrainChatGPTChatGPT 0%Google AI ModeGoogle AI Mode 60%
Questions and answers
Google AI ModeNov 19, 2026
Several modern fine-tuning and
LLM post-training platformslet you point directly to external object storage or repository URLs (like Amazon S3, Google Cloud Storage, or the Hugging Face Hub) so you don't have to manually download, reformat, or upload static files through a web dashboard.Top Platforms Supporting Direct External Repository / Cloud Storage Connection
- Anyscale (LLM Post-Training & LLaMA-Factory)
- Supported Sources: Amazon S3 (
s3://...), Google Cloud Storage (gs://...), and Hugging Face Hub. - How it works: In your dataset configuration file (
dataset_info.json), you specify the cloud object URI (cloud_file_name) directly instead of a local file path. Anyscale's worker nodes pull and stream the data during initialization as long as proper IAM/bucket permissions are configured. - Learn more: Check out the Anyscale Dataset Preparation Docs.
- Supported Sources: Amazon S3 (
- Hugging Face AutoTrain / Spaces / Custom Trainers (
datasets)- Supported Sources: Hugging Face Hub (
hf://datasets/...), Amazon S3 (s3fs), Google Cloud Storage (gcsfs), Azure Blob/DataLake. - How it works: The native
datasetslibrary acts as a bridge, allowing training scripts to stream or load data straight from private/public S3 or GCS buckets using filesystem implementations likes3fsorgcsfswithout materializing the whole file locally first. - Learn more: Read the Hugging Face Cloud Storage Guide.
- Supported Sources: Hugging Face Hub (
- Amazon Bedrock (Custom Model Fine-Tuning)
- Supported Sources: Amazon S3.
- How it works: When launching a fine-tuning or continued-pre-training job in Bedrock, you pass the S3 URI of your training and validation JSONL files. Bedrock reads directly from your S3 bucket during the training lifecycle.
- Learn more: Explore via the AWS Bedrock Documentation.
- Azure OpenAI Service / Azure Machine Learning
- Supported Sources: Azure Blob Storage, Azure Data Lake, or registered Azure ML Datastores.
- How it works: You reference a blob storage container or URI where your training
.jsonlfiles reside. Azure trains the model directly against that managed storage reference. - Learn more: Review the workflow on Microsoft Learn.
Comparison of Integration Methods
Platform Direct Storage Connectors Data Formats Accepted Manual Export Needed? Anyscale S3, GCS, Hugging Face Hub JSON, JSONL, Parquet, Arrow No (point to URI) Hugging Face Ecosystem S3, GCS, Azure, HF Hub Hugging Face Dataset format / Arrow / Parquet No (streams via s3fs/gcsfs)Amazon Bedrock Amazon S3 JSONL No (point to S3 URI) Azure OpenAI Azure Blob Storage JSONL No (point to Blob URI) To help narrow down the best fit, what base model family (e.g., Llama, Mistral, custom architecture) are you looking to train, and where does your data currently live (S3, a database, or a Git/HF repo)?
ChatGPTNov 19, 2026
Yes. There are several platforms that support connecting to external datasets or cloud storage, allowing you to fine-tune models without manually downloading and uploading training files.
The important distinction is between:
- Direct repository access: The platform reads your dataset from its existing repository.
- Cloud storage integration: The platform reads data directly from S3, GCS, or Azure Blob.
- Data-source connectors: The platform connects to databases, SaaS applications, APIs, or document repositories and prepares training data automatically.
Here are the strongest options I found, based on their documented capabilities as of November 2026.
1. Platforms worth considering
1. Fireworks AI
(Strongest cloud-storage match)
S3 · Google Cloud Storage · Azure Blob
Supports Bring Your Own Bucket (BYOB). Training jobs can read datasets directly from your own cloud storage without copying the dataset into Fireworks.
- Supports SFT, DPO, and ORPO.
- Managed training and custom training APIs.
- OpenAI-compatible JSONL datasets work without conversion.
- Your data can remain in your storage bucket.
Limitation: Your data still needs to meet the expected training schema.
Sources:
2. Hugging Face
(Strongest dataset-repository match)
Hugging Face Hub · Dataset repositories · Streaming
Hugging Face supports streaming datasets directly from its Hub, so training doesn't require downloading the entire dataset first.
Compatible training frameworks include:
- Transformers
- TRL
- Axolotl
- LlamaFactory
- Unsloth
- Supports private and public dataset repositories.
- Supports streaming large datasets.
- Works with existing Hugging Face dataset identifiers.
- Suitable for SFT, preference optimization, and other training workflows.
Limitation: This is primarily an ecosystem of dataset hosting, libraries, and training infrastructure. You may still need to configure your own training environment.
Sources:
3. Nucleus AI
(Interesting connector-based option)
Salesforce · Slack · Zendesk · Snowflake · Databricks · APIs · Documents
Nucleus advertises connected data sources that can feed its fine-tuning workflows.
- SaaS integrations.
- Database connections.
- REST and GraphQL APIs.
- Google Drive and Microsoft 365.
- Automated generation of training-ready datasets.
- Managed model fine-tuning.
Important qualification: Its connector marketing describes automated data integration, but its documented quickstart still uses a JSONL file upload for the actual fine-tuning job. Verify whether your particular connector can feed training automatically without an intermediate export.
Sources:
4. Databricks
Unity Catalog · Delta Tables · Hugging Face
Databricks has documented support for training from:
- Unity Catalog tables.
- Delta tables.
- Public Hugging Face datasets.
This is particularly interesting if your training data already resides in a data lake or warehouse.
Limitation: Its documented Foundation Model Fine-tuning workflow is marked end-of-life (EoL). Check the replacement training workflow before choosing it.
Sources:
5. Amazon SageMaker AI
Amazon S3 · AWS data infrastructure
SageMaker supports fine-tuning with datasets stored in Amazon S3.
- Training jobs consume data from S3.
- Suitable for enterprise-scale training.
- Integrates with AWS storage and security infrastructure.
Limitation: This is a cloud-storage integration rather than a general-purpose GitHub or dataset-repository connector. Data preparation may still be necessary.
Source:
2. Feature comparison
Platform External data access Automatic data preparation No manual file upload Fireworks AI S3, GCS, Azure Limited (Yes, BYOB) Hugging Face Hub datasets Framework-dependent (Yes, streaming) Nucleus AI SaaS, DBs, APIs, documents Advertised (Verify workflow) Databricks Delta, Unity Catalog, HF Some (Supported workflow) SageMaker S3 Framework-dependent (Yes, S3) One important caveat: direct access doesn't necessarily mean zero data transformation. Fine-tuning still requires the dataset to be compatible with the model's training task.
3. My recommendations
I'd narrow the shortlist according to your source of data:
Your existing dataset location Best starting point Hugging Face repositories Hugging Face GitHub repositories Hugging Face ecosystem with a Git-based ingestion workflow AWS S3 Fireworks AI or SageMaker Google Cloud Storage Fireworks AI Azure Blob Storage Fireworks AI Snowflake / Databricks Nucleus AI or Databricks Salesforce / Slack / Zendesk Nucleus AI Google Drive / Microsoft 365 Nucleus AI 4. One question that would help narrow this down
When you say external dataset repositories, which type of source do you primarily mean?
Where are your datasets currently stored?
What kind of fine-tuning platform do you prefer?
[Find the best matching platforms]
I can then narrow this down to platforms that genuinely support direct connections rather than simply accepting uploaded files or offering generic API integrations.
Pages cited in this answer 17
- fireworks.aiFine-tuning and evaluation on one stack
- fireworks.aiTrain Your Own AI Models: SFT, DPO, and RL | Fireworks
- fireworks.aiOwn your specialized intelligence | Fireworks
- huggingface.coLibraries · Hugging Face
- huggingface.coUploading datasets · Hugging Face
- huggingface.coHugging Face – The AI community building the future.
- huggingface.coLibraries · Hugging Face
- nucleus-ai.ioTrain | Nucleus AI
- docs.nucleus-ai.ioQuickstart
- nucleus-ai.sgIntegrations - Connect your data sources to your LLM
- docs.databricks.comCreate a training run using the Foundation Model Fine-tuning API (EoL) | Databricks on AWS
- docs.databricks.comPrepare data for Foundation Model Fine-tuning (EoL) | Databricks on AWS
- docs.databricks.comFoundation Model Fine-tuning (EoL) | Databricks on AWS
- databricks.comDatabricks: Leading Data and AI Platform for Enterprises
- docs.databricks.comDatabricks documentation | Databricks on AWS
- docs.aws.amazon.comFine-Tune a Model - Amazon SageMaker AI
- aws.amazon.comThe center for all your data, analytics, and AI – Amazon SageMaker – AWS
- Anyscale (LLM Post-Training & LLaMA-Factory)
ChatGPTOct 29, 2026
If you want to fine-tune an AI model using data that already lives in your company’s repositories, I’d prioritize platforms with native data connectors, direct cloud-storage access, or retrieval and ingestion pipelines rather than services that require you to manually export datasets and reformat them.
One important distinction: some platforms can read from your existing sources and prepare data automatically, while others connect directly to a data lake or database but still require you to transform records into a fine-tuning format such as JSONL. A direct connection doesn't necessarily eliminate preprocessing.
Platforms worth evaluating
1. Snowflake Cortex Fine-tuning
(Best for Snowflake data)
If your data already resides in Snowflake, start here. Cortex Fine-tuning is a managed service integrated with Snowflake, allowing you to train models using data in that environment. Snowflake also documents a training workflow with direct table access and no CSV export or external staging. Check which workflow best fits your intended model and training setup.
Best for: Snowflake tables, governed enterprise data, SQL-centric workflows.
2. Nucleus AI
(Best for many repository types)
Its published connector catalog describes connections to databases, SaaS tools, APIs, file storage and document systems, including PostgreSQL, Snowflake, Databricks, Amazon S3, Google Drive, Microsoft 365, Confluence, Slack and Salesforce. It says connected sources can generate validated training datasets and refresh them automatically.
This is particularly relevant if your data is scattered across several systems. Verify connector availability, permissions and whether your preferred sources are supported before committing.

3. Databricks Mosaic AI Training
A strong option if your data is already in the Databricks ecosystem. Its fine-tuning tooling supports remote training data in Unity Catalog volumes, Amazon S3, Google Cloud Storage and Oracle Cloud Infrastructure object storage. It can read from these locations without requiring you to upload the data through a separate manual file-export workflow, although the training data still needs to meet the supported format requirements.
Best for: Databricks, cloud object storage, ML engineering teams.
4. Hugging Face with a fine-tuning framework
Hugging Face's dataset ecosystem supports streaming datasets, and compatible training frameworks can fine-tune models without downloading an entire dataset first. This is a flexible route for teams using open-weight models, custom pipelines and large datasets. However, direct integration depends on your source repository and may require code or a connector layer.
Best for: Custom pipelines, open models, engineering flexibility.
Dataset integration documentation
Which one fits your setup?
If your existing data is in… Start with Snowflake tables Snowflake Cortex Google Drive, Slack, Confluence and multiple SaaS tools Nucleus AI Databricks, S3 or cloud object storage Databricks Mosaic AI Custom databases or a self-managed open-model pipeline Hugging Face ecosystem One caveat: OpenAI's fine-tuning workflow traditionally requires a JSONL training file, and OpenAI announced in May 2026 that it was winding down its fine-tuning platform for new users. I wouldn't make it your first choice for this specific requirement.
Let me narrow this down
Which repository or data source do you want to connect?
1. Where does your data live?
2. What do you want to fine-tune?
Open-weight models (Llama, Qwen, etc.)
Commercial API models
Either / not decided
3. What's the biggest requirement?
No manual export or file conversion
Data stays inside our cloud or security boundary
Automatic sync when source data changes
No-code or low-code setup
[Find the best-fit platforms]
Pages cited in this answer 7
- docs.snowflake.comFine-tuning (Snowflake Cortex) | Snowflake Documentation
- snowflake.comFine-Tune LLMs on Your Data with Snowflake ArcticTraining and ML Jobs
- nucleus-ai.ioData connectors | Nucleus AI
- docs.mosaicml.comFinetuning - Mosaic AI Training
- huggingface.coLibraries · Hugging Face
- developers.openai.comSupervised fine-tuning | OpenAI API
- openai.comIntroducing improvements to the fine-tuning API and expanding our custom models program | OpenAI
Google AI ModeOct 29, 2026
Several machine learning and MLOps platforms allow you to connect directly to cloud object storage and data warehouses to fine-tune models without manual local exports.
Platform / Tool Supported Data Sources / Repositories Direct Pull / Integration Features Anyscale Amazon S3, Google Cloud Storage (GCS), Hugging Face Hub Mounts or streams distributed training data directly from cloud storage buckets into Ray Train workloads. Databricks Unity Catalog, Delta Lake, AWS S3, Azure Blob Storage Trains foundation models (via Mosaic AI Model Training) directly on data stored inside Delta tables or cloud object storage. Clarifai S3, GCS, Azure Blob, SQL/NoSQL databases, URL scrapers Connects to external data streams and object stores to index, curate, and fine-tune models in-place. Hugging Face AutoTrain / SageMaker Amazon S3, Hugging Face Datasets, local/remote URIs Pulls training splits directly via URI pointers or cloud connectors without requiring manual desktop file conversion. Top Platform Options
- Databricks (Mosaic AI Model Training): Best if your enterprise data already lives in Delta Lake or Unity Catalog . You can launch fine-tuning jobs on open-source or custom base models directly from your tabular or unstructured data tables without exporting to local disks.
- Anyscale: Best for customized, large-scale distributed fine-tuning (using Ray). It connects natively to Amazon S3 and Google Cloud Storage buckets, pulling distributed training files straight into the training cluster.
- Clarifai: Best for unstructured multi-modal data (text, image, audio, video). It features built-in direct connectors for cloud buckets and databases so you can point the platform to your raw repository and run fine-tuning pipelines.
- Amazon SageMaker / Hugging Face Integration: Best if you use AWS infrastructure. You can point SageMaker training jobs directly to S3 URI paths containing your JSONL or Parquet datasets, skipping manual client-side uploads.
If you can share where your data is currently stored (e.g., AWS S3, Snowflake, a SQL database, or local servers) and which base model you want to train, I can recommend the exact setup steps.
ChatGPTNov 10, 2026
Yes. There are several fine-tuning platforms that can connect to data where it already lives, rather than requiring you to manually download files, convert them, and upload training datasets.
The important distinction is between two types of tools:
- Direct-source training: The platform reads your existing databases, cloud storage, or data warehouse when training.
- Automated data ingestion: The platform connects to your repositories, extracts the relevant information, converts it into a training-ready format, and manages the dataset for you.
If your goal is to eliminate manual exporting and formatting, both approaches are worth considering.
Tools worth looking at
1. Snowflake Cortex Training
(Best for data already in Snowflake or S3)
- Trains using data in Snowflake tables, internal stages, and S3 through external stages.
- Avoids building a separate export pipeline.
- Supports open-weight models and frameworks such as Unsloth and Axolotl.
- Managed GPU training.
Limitation: Currently in preview, and requires some training-script configuration.
docs.snowflake.comdocs.snowflake.comdocs.nvidia.comdocs.nvidia.com
2. Nucleus AI
(Best for multiple existing data sources)
- Advertises 95 data-source connectors.
- Supports PostgreSQL, Snowflake, Databricks, S3, Google Drive, Microsoft 365, Confluence, Slack, Gmail, and more.
- Maps source records into conversational training examples.
- Automatically refreshes datasets hourly, daily, weekly, or manually.
- Provides validation before training.
Limitation: It still generates JSONL internally. The advantage is that you don't have to create and maintain those files yourself.
Official connector documentation
3. Hugging Face + dlt
- Connects to databases, cloud storage, REST APIs, and other sources through dlt.
- Can automatically ingest data into Hugging Face dataset repositories.
- Supports repeatable data pipelines.
- Useful if you want to retain control over model training and infrastructure.
Limitation: This is automated ingestion rather than completely direct-source training. Data still passes through a dataset preparation pipeline.
Official dataset ingestion documentation
4. NVIDIA Kumo
- Provides connectors for Amazon S3, Google Cloud Storage, Azure Data Lake, Snowflake, BigQuery, and Databricks.
- Supports reading existing tables through connector interfaces.
- Particularly interesting for structured-data and graph-model workloads.
Limitation: More specialized than a general-purpose conversational LLM fine-tuning service.
docs.nvidia.comdocs.nvidia.com
Which one should you choose?
Your existing data My recommendation Snowflake / S3 Snowflake Cortex Training Slack, Gmail, Drive, Confluence Nucleus AI PostgreSQL / MySQL Nucleus AI BigQuery / Databricks Nucleus AI or NVIDIA Hugging Face datasets Hugging Face + dlt GitHub repositories Requires a more specialized repository-ingestion workflow One important distinction
Even when a platform connects directly to your repository, it usually still needs to:
- Identify useful training examples.
- Remove duplicates and irrelevant information.
- Convert records into the model's expected training structure.
- Validate the resulting dataset.
You want a platform that automates these steps, not necessarily one that eliminates data formatting altogether.
One question to narrow this down
When you say existing repositories, which kind do you mean?
1. Where is your existing data stored?
2. What are you trying to accomplish?
[Find the best matching tools]
I can then narrow this down to tools that genuinely eliminate manual exports, rather than platforms that simply make uploading JSONL files easier.
Pages cited in this answer 6
Google AI ModeNov 10, 2026
When you want to fine-tune a model (like an LLM or code assistant) directly from your source code or documentation repositories without manual export/conversion steps, your options generally fall into managed cloud platforms, CI/CD-integrated pipelines , or local automation toolkits.
Direct repository ingestion tools parse your code, markdown files, or commit history on the fly and pipe them into training loops.
Top Tools & Platforms
- OpenPipe
- How it works: Designed heavily around developer workflows and fine-tuning models on application data/logs. It features direct API integrations and webhook/data-sync pipelines that can ingest structured interactions, API logs, or repository-linked datasets rather than forcing you to manually wrangle JSONL files every time.
- Best for: Fine-tuning smaller open-source models (like Llama or Mistral) to replace larger, expensive API calls based on live app/code interactions.
- Hugging Face AutoTrain (with GitHub / Spaces Integration)
- How it works: Hugging Face allows you to connect data sources via the
datasetslibrary or pull straight from a Hugging Face Dataset Repo which can sync with GitHub actions. While it doesn't always have a "paste your private repo URL and click train" magic button for arbitrary codebases out of the box, you can set up a lightweight GitHub Action that automatically scrapes, formats, and pushes your repo's markdown/code files into a HF Dataset, triggering an AutoTrain job. - Best for: Complete control over the pipeline using Python scripts and the Hugging Face Transformers Ecosystem.
- How it works: Hugging Face allows you to connect data sources via the
- Modal / RunPod Serverless with Custom Git-Sync Scripts
- How it works: Platforms like Modal or RunPod aren't out-of-the-box GUI wrappers, however , you can write a 20-line Python script using their SDKs that clones your private Git repository inside the container, runs a pre-processing script (e.g., via Unsloth or
trl), and fine-tunes a model on the fly whenever a webhook is hit or a git push happens. - Best for: Engineers who want a zero-manual-step pipeline coded directly into a
git pushhook.
- How it works: Platforms like Modal or RunPod aren't out-of-the-box GUI wrappers, however , you can write a 20-line Python script using their SDKs that clones your private Git repository inside the container, runs a pre-processing script (e.g., via Unsloth or
- LLaMA-Factory (with custom data loaders)
- How it works: LLaMA-Factory is one of the most popular open-source local/cloud fine-tuning UIs. While it typically expects standard formats (Alpaca, ShareGPT), you can point its data configuration to local directories or write a custom dataset loader script in its
dataset_info.jsonthat reads raw code files directly from a mounted git directory. - Best for: Local-first or self-hosted fine-tuning with a clean browser UI.
- How it works: LLaMA-Factory is one of the most popular open-source local/cloud fine-tuning UIs. While it typically expects standard formats (Alpaca, ShareGPT), you can point its data configuration to local directories or write a custom dataset loader script in its
A Quick Reality Check on "Raw Repos"
True end-to-end fine-tuning requires structured data (e.g., prompt/response pairs, instruction tuning formats, or clean continuous text chunks). If you point a tool straight at a raw GitHub repo without any transformation, the model will just learn messy syntax or random file structures rather than the specific task you want it to perform.
Most "zero-conversion" tools actually run an automated ETL/parsing layer under the hood to slice your READMEs, docstrings, or code functions into instruction-response pairs before feeding them to the GPU.
If you'd like to narrow this down, tell me:
- What language or file types are in your repository (Python, TypeScript, Markdown docs, etc.)?
- Are you looking for a no-code/low-code cloud dashboard or a self-hosted/Python script?
I can point you toward the exact stack to wire it up.
For agents: this page as Markdown, https://parse.gl/topics/direct-repository-fine-tuning.md. JSON, MCP · free for the last 90 days, $0.01 a check
This page
Read and search
Connect
An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces
What it costs
Prices: https://parse.gl/pricing.md
Map of the site: https://parse.gl/llms.txt