Best practices for infrastructure provisioning in an IDP
Integrating infrastructure provisioning into an Internal Developer Platform (IDP) is about making infrastructure self-service for developers while keeping security, governance, cost, and reliability under platform-team control.
The strongest approach is to treat infrastructure as a product offered through your IDP, rather than simply adding Terraform commands to an existing CI/CD pipeline.
For most teams, I recommend a combination of Infrastructure as Code (IaC), reusable infrastructure templates, policy-as-code, GitOps or pull-request-based workflows, and centralized observability.
1. Recommended architecture
Developer / Application Team
Requests a database, namespace, queue, or environment
IDP Self-Service Catalog
Golden paths · Templates · Approved infrastructure modules
CI/CD Provisioning Pipeline
Plan & review
Policy & security
Identity & access
Approval gates
IaC Execution Layer
Terraform / OpenTofu, cloud APIs, Kubernetes operators, or Crossplane
Provisioned Infrastructure
What are the standard patterns for automating infrastructure provisioning inside an Internal Developer Platform for CI/CD?
What are the recommended approaches for handling infrastructure provisioning within a CI/CD pipeline when building an IDP?
Other topics in Internal Developer Platforms and CI/CD
Feedback loop: publish resource status, ownership, cost, audit events, and operational links back into the IDP catalog.
The key architectural principle is to separate the developer experience from the infrastructure execution and governance layers. Developers request capabilities; the platform decides how those capabilities are safely delivered.
Create versioned modules for common capabilities: managed databases, Kubernetes namespaces, object storage, networking, and ephemeral test environments. Offer safe defaults rather than asking developers to configure every low-level resource.
Standardize encryption, networking, tagging, logging, and backups.
Pin module versions and test upgrades before publishing them.
2. Use a predictable, reviewable provisioning workflow
A typical pipeline should validate and preview infrastructure changes before applying them.
Pull request
Format, lint, validate, security scan, module tests
Plan
Generate a change preview and estimate cost
Policy and approval
Enforce controls and require appropriate review
Apply
Execute with restricted deployment identity
Verify and publish
Health checks, outputs, audit trail, IDP status
Run formatting, validation, and policy checks automatically. Require stronger approval for production, destructive operations, and high-risk network or identity changes. Avoid requiring manual approval for every low-risk development resource.
Treat compliance as a pipeline requirement, not a checklist performed after deployment.
Examples of enforceable policies:
Approved regions and resource types.
Mandatory encryption and private networking.
Required ownership, environment, and cost-center tags.
Restricted public access and privileged IAM permissions.
Budget limits and production-specific controls.
Evaluate what you can before provisioning. Keep runtime cloud security controls as a separate layer; pipeline policies alone are not a complete security boundary.
Tools include Open Policy Agent (OPA), Conftest, Checkov, and supported policy frameworks in your IaC platform.
4. Make identity and state management foundational
Use workload identity federation, such as OIDC, where supported instead of long-lived cloud access keys.
Apply least-privilege permissions per workload, environment, and deployment stage.
Store Terraform state remotely with access controls, encryption, backups, and locking where supported.
Separate production and non-production state and permissions.
Never commit state files, credentials, or sensitive outputs to Git.
Your IDP should expose capabilities, not raw Terraform configuration.
A developer might select “Create development PostgreSQL database,” then provide a name, environment, size, and data classification. The platform maps these inputs to an approved module and provisioning workflow.
Every provisioned resource should have:
An owner and associated application or service.
A lifecycle status and provisioning history.
Links to logs, dashboards, documentation, and operational runbooks.
A clear path to update, decommission, or request help.
If using Backstage, templates and catalog metadata can provide the developer-facing experience while a separate, controlled workflow handles infrastructure execution.
6. Separate application delivery from infrastructure lifecycle
Application CI/CD and infrastructure provisioning are related but should not share unrestricted permissions or blindly run the same deployment steps.
Provision or update infrastructure when infrastructure definitions change.
Deploy application code independently when possible.
Publish infrastructure outputs through controlled mechanisms.
Use dependency-aware orchestration when infrastructure must exist before an application deployment.
Make retries safe and operations idempotent.
For Kubernetes-centric platforms, consider GitOps for reconciling declared state. Avoid having several independent controllers or pipelines compete to manage the same resource.
7. Build in drift detection, observability, and cleanup
Provisioning is not complete when the pipeline turns green. Track the resource throughout its lifecycle.
Detect configuration drift and unauthorized manual changes.
Monitor failed runs, provisioning duration, and cloud API errors.
Record who requested, approved, and executed each change.
Apply TTLs and automatic cleanup to temporary environments.
Provide a controlled decommissioning workflow that checks dependencies, retention requirements, and backups before deletion.
For platforms using continuous reconciliation, tools such as Crossplane can monitor managed resources and reconcile actual state toward the declared desired state.
8. Measure the platform as a product
Track outcomes rather than just pipeline success.
Developer experience: time from request to usable resource.
Reliability: provisioning success rate and recovery time.
Governance: policy violations, exceptions, and drift.
Efficiency: reusable-module adoption and manual interventions.
Cost: spend by service, team, environment, and idle resources.
Use these measurements to prioritize improvements to the IDP catalog and its golden paths.
3. Choose the right technology stack
A practical stack depends on whether your IDP is primarily a developer portal, a Kubernetes control plane, or an infrastructure automation platform.
Layer
Recommended options
Purpose
Developer portal
Backstage
Service catalog, templates, ownership
Infrastructure as Code
Terraform or OpenTofu
Declarative provisioning
IaC execution
HCP Terraform or CI runners
State, plans, execution and approvals
Policy and security
OPA / Conftest, Checkov
Policy enforcement and configuration scanning
GitOps
Argo CD or Flux
Continuous reconciliation of Kubernetes resources
Kubernetes-native control plane
Crossplane
Self-service infrastructure APIs and reconciliation
Secrets and identity
Cloud workload identity, OIDC, a secrets manager
Short-lived credentials and access control
Observability
Your existing metrics, logs, tracing and audit stack
My default recommendation: start with Backstage (or your existing IDP), Terraform/OpenTofu modules, your current CI system, remote state, OIDC, and policy checks. Add a dedicated infrastructure control plane such as Crossplane if continuous reconciliation and Kubernetes-native infrastructure APIs solve a real need.
You do not need to adopt every tool in the table.
4. A practical implementation roadmap
Phase 1
Foundation
Weeks 1–2: Establish safe provisioning
Standardize IaC repositories and reusable modules.
Configure remote state and workload identity.
Add validation, scanning, and plan generation.
Define environment separation and ownership conventions.
Phase 2
Integration
Weeks 3–4: Connect the IDP
Create two or three high-value self-service templates.
Connect catalog requests to the provisioning pipeline.
Publish resource status and outputs back to the catalog.
Introduce production approvals and policy enforcement.
Phase 3
Scale
Weeks 5–8: Operationalize
Add drift detection and resource cleanup.
Implement cost allocation and lifecycle controls.
Establish module versioning and release practices.
Measure adoption, provisioning time, failures, and developer feedback.
Illustrative timeline. Adjust it to your existing IaC maturity, security requirements, and team capacity.
5. Common mistakes to avoid
Giving developers unrestricted cloud credentials: use controlled self-service interfaces and scoped identities.
Building a generic Terraform form: developers should request capabilities, not become Terraform operators.
Automatically applying every change: use risk-based approvals, especially for production and destructive operations.
Sharing one state file across unrelated environments: isolate state and execution permissions to reduce blast radius.
Treating infrastructure as disposable without lifecycle controls: protect persistent data and require safeguards for deletion.
Overengineering the IDP before validating demand: start with a small number of useful golden paths and expand based on adoption.
6. Tailor this to your environment
I can make this more concrete by recommending the architecture and pipeline stages for your specific setup.
Select your current setup. You can leave defaults where they fit.
1. IDP / developer portal
Backstage
Existing custom IDP
No IDP yet — designing from scratch
Another IDP
2. Cloud or platform
AWS
Azure
Google Cloud
Multi-cloud
Kubernetes / on-premises
3. CI/CD system
GitHub Actions
GitLab CI/CD
Jenkins
Azure DevOps
Other
4. Infrastructure tooling
Terraform
OpenTofu
Crossplane
Cloud-native IaC (such as CloudFormation or Bicep)
