As I’ve been learning about taking backend applications from development to production, one area that stood out to me is configuration and secret management.
During development, it’s easy to work with an app.env file and keep values such as database URLs, token keys, and application configuration locally. But once an application moves toward production, that approach raises an important question:
Where should production configuration actually live?
I recently explored this problem using AWS Secrets Manager, AWS CLI, jq, GitHub Actions, Docker, and Amazon ECR. What I found interesting was that the topic wasn't just about storing passwords securely. It connected several different concepts—AWS IAM, CI/CD, shell environments, Docker, and application startup—into one deployment workflow.
Here are the main things I learned.
Moving Secrets Out of the Repository
The first thing that became clear to me was that production configuration should not live directly inside the Git repository.
A production application may need values such as:
DB_SOURCE=...
DB_DRIVER=postgres
SERVER_ADDRESS=...
ACCESS_TOKEN_DURATION=15m
TOKEN_SYMMETRIC_KEY=...
Some of these are ordinary configuration values, while others are highly sensitive.
The problem becomes especially obvious with values such as database connection strings and token-signing keys. Keeping those values in source control creates an unnecessary security risk.
The approach I learned was to move this sensitive configuration into AWS Secrets Manager.
Instead of thinking of the repository as the place where all configuration lives, I started thinking of the architecture as:
Source Code
+
External Configuration
+
Secrets
The source code remains in Git, while production secrets are managed separately.
That separation is a simple idea, but it makes a big difference when thinking about production systems.
AWS Secrets Manager as a Central Place for Configuration
What I found useful about AWS Secrets Manager is that it isn’t limited to storing a single password.
A secret can contain multiple key-value pairs.
For example:
DB_SOURCE
DB_DRIVER
SERVER_ADDRESS
ACCESS_TOKEN_DURATION
TOKEN_SYMMETRIC_KEY
This means an application’s production configuration can be grouped together and retrieved as a single secret.
For this use case, the appropriate secret type was Other type of secrets, since the configuration wasn’t limited to database credentials.
This helped me understand the difference between configuration management and simply storing passwords.
A production application has many values that differ between environments. Some are sensitive and some aren’t, but having a centralized mechanism for managing them makes the deployment process much cleaner.
Generating Stronger Secrets
Another thing I learned was the importance of using genuinely random values for security-sensitive configuration.
For example, a token-signing key shouldn’t be something simple or predictable.
The approach demonstrated was to use OpenSSL:
openssl rand -hex 64 | head -c 32
This generates random data and produces a 32-character hexadecimal value suitable for use as a secret key in this particular setup.
What I took away from this wasn’t really the command itself.
It was the principle:
Production secrets should be generated as secrets, not invented as convenient strings.
That’s an important mindset shift when moving from development to production.
Encryption and Secret Rotation
AWS Secrets Manager also introduced me to two ideas that I hadn’t fully connected with secret management before: encryption and rotation.
Secrets can be encrypted using AWS KMS, either with a default encryption key or a custom key.
There is also support for automatic rotation, where secret values can be changed periodically through a configured process.
For example, credentials such as a database password could potentially be rotated instead of remaining unchanged indefinitely.
I found this particularly interesting because it shows that secret management isn’t only about hiding values.
It’s also about managing their lifecycle.
Understanding IAM Better
One of the most useful parts of the process for me was seeing what happens when an AWS identity doesn’t have the required permission.
The CI user already had permissions related to Amazon ECR, but when it attempted to retrieve a secret from Secrets Manager, AWS returned an authorization error.
That made the relationship between identity and permission much clearer to me.
Having AWS credentials doesn’t mean:
“This user can do anything in AWS.”
Instead, the model is:
Identity
↓
IAM permissions
↓
Allowed AWS operations
The deployment identity therefore needed additional permission to access Secrets Manager.
This was a good practical demonstration of IAM rather than just learning it as an abstract AWS concept.
It also reinforced the importance of least privilege when designing production infrastructure.
Using AWS CLI to Access Secrets
Once the permissions were in place, the secret could be retrieved through the AWS CLI using:
aws secretsmanager get-secret-value \
--secret-id production_app
The response contains more information than just the configuration values.
The part that matters for this use case is the:
SecretString
field.
Using the AWS CLI’s query and output options makes it possible to extract that value directly.
What I liked about this part was that the AWS console wasn’t required for every operation.
Once the necessary permissions and CLI configuration were in place, secrets could be accessed programmatically.
And that’s where this starts becoming useful for automation.
The Role of jq
The secret was stored as JSON, but the application configuration was expected in a traditional environment-file format:
KEY=value
For example:
DB_DRIVER=postgres
ACCESS_TOKEN_DURATION=15m
This is where I learned about jq.
jq is a command-line JSON processor that can transform JSON into other structures and formats.
The interesting part was learning how several small jq concepts can be combined.
First, to_entries turns an object into an array of key-value objects.
Then map allows each entry to be transformed.
String interpolation can combine the key and value:
KEY=value
And finally, the raw-output option removes the surrounding quotation marks.
The overall transformation is:
JSON
↓
to_entries
↓
map
↓
KEY=value
↓
raw output
The result can then be written into an environment file.
This was a nice example of how small command-line tools can become powerful when combined.
Connecting Secret Management to CI/CD
The next part that made the whole concept click for me was integrating it into a GitHub Actions workflow.
Instead of manually retrieving secrets before every deployment, the CI/CD pipeline can perform that operation automatically.
Conceptually:
Git Push
↓
GitHub Actions
↓
Authenticate with AWS
↓
Retrieve production secrets
↓
Transform configuration
↓
Build application image
↓
Push image to ECR
The important thing is that production configuration becomes part of an automated deployment process rather than a manual task.
The workflow added a dedicated step to retrieve the secrets and generate the environment configuration before the Docker image was built.
This made me see CI/CD differently.
It’s not simply:
build → test → deploy
There is also an important configuration-management layer around those steps.
The Docker Problem I Didn’t Expect
The most valuable part of the exercise for me came when the supposedly working image was actually run.
The image built successfully.
It was pushed to ECR successfully.
It could be pulled successfully.
But when the container started, the database migration failed with:
URL cannot be empty
At first glance, this was confusing because the app.env file already contained the database configuration.
The problem was the difference between an environment file and the shell’s environment.
The configuration existed here:
/app/app.env
but the migration command was trying to access something like:
$DB_SOURCE
Those are not automatically the same thing.
Simply having:
DB_SOURCE=...
inside a file doesn’t mean that $DB_SOURCE exists in the current shell environment.
That distinction became much clearer when the value was checked before and after loading the environment file.
source and the Shell Environment
The solution was to explicitly load the environment file:
source /app/app.env
before running the database migration.
Conceptually:
/app/app.env
↓
source
↓
Shell environment
↓
Database migration
After the file was sourced, variables such as DB_SOURCE became available to subsequent commands in that shell.
That resolved the migration failure.
For me, this was one of the most useful lessons because it’s easy to assume that:
.env file = environment variables
But operationally, they’re different things.
An environment file is just a file until something loads it into the process environment.
Why Testing the Actual Image Matters
Another lesson I took from this was the importance of testing the actual production artifact.
The CI pipeline had already completed successfully.
The Docker image had been built.
The image had been pushed to ECR.
Yet the application still failed when the container actually started.
The problem only became visible when the production image was pulled and executed.
After fixing the startup process, the container successfully:
- loaded the environment configuration,
- ran the database migration,
- started the application,
- accepted an API request,
- and communicated with the production database.
That reinforced an important engineering lesson for me:
A successful build is not the same thing as a successful deployment.
There can be problems that only appear at runtime.
The Bigger Picture
Looking back at the entire process, what I learned wasn’t really about one AWS service.
It was about how several pieces of infrastructure fit together:
Each component has a different responsibility.
Secrets Manager handles sensitive configuration.
IAM controls who can access it.
AWS CLI provides programmatic access.
jq transforms the configuration.
GitHub Actions automates the deployment process.
Docker packages the application.
ECR stores the resulting image.
And the application’s startup process determines how that configuration is ultimately made available to the running processes.
Seeing these pieces together gave me a much better understanding of what a production deployment actually involves.
What I Took Away
A few ideas stood out to me most from this exercise.
Secrets should have a lifecycle
It’s not enough to generate a secret once and forget about it. Production systems need mechanisms for storage, access control, encryption, and potentially rotation.
IAM is part of application architecture
Permissions aren’t just an AWS administration concern. They directly determine what a CI/CD system can do during deployment.
Configuration is part of deployment
Building an application isn’t enough. The correct environment-specific configuration must also reach the application securely.
Small tools can solve real infrastructure problems
Something as simple as jq becomes extremely useful when connecting cloud APIs with traditional application configuration.
Containers expose assumptions
The environment-variable issue was a good reminder that assumptions about local development environments don’t always hold inside containers.
Runtime testing matters
A successful pipeline doesn’t guarantee a working application. Running the actual artifact is an important part of validating a deployment.
Final Thoughts
This exploration changed the way I think about production configuration.
Previously, I mostly thought of environment variables as something that lived in a .env file.
Now I see a larger chain:
Secret creation
↓
Secure storage
↓
Access control
↓
CI/CD retrieval
↓
Configuration transformation
↓
Container startup
↓
Application runtime
Every step has its own security and operational considerations.
The next area I’m interested in exploring is how this model changes when applications move into Kubernetes, particularly how secrets and configuration can be injected at runtime rather than being handled entirely during the image-building process.