k8s, ci, etc are all really useful and solve a lot of hard problems, but god are they a bitch to set up and then monitor, fix when something weird happens, secure from nasty people outside, patch when some update becomes necessary, resist the urge to “improve”, etc. This doesn’t include the time it takes to work these tools into your project, like k8s for example really requires you grok its model of how stuff it runs is supposed to look. There’s a reason entire companies exist that simply offer managed versions of these services!

If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running) and honestly it’s pretty fun to just run a command and poof you’re deployed. Then you can just wait until it starts making sense to use fancier tooling (more load, more sites/DCs, more services, more people/teams, etc). Monitoring is always good to start with, for other stuff a good litmus test is that price points for hosted k8s & ops stuff will start to make sense, you’ll be paying in time in any case. Who knows, maybe when that happens something new will pop up that makes even more sense. Until then, reducing the amount of stuff you need to worry about is always a good strategy.

I never realised I was using spooky arcane oldhat stuff! I feel wizardly now.

My projects (for small clients and myself) basically use this.

- A "build.sh" script that does a local build of back end and front end

- A "deploy.sh" script that scp's everything to the server (either a digital ocean VPS or an EC2 instance), runs npm install, runs database migrations and restarts with pm2

So running my entire CI pipeline in terminal is: ./build.sh && ./deploy.sh

This also solves the roll-back problem mentioned elsewhere. Just checkout the previous version (as long as you don't make any side effects like DB updates that are incompatible).

I admit, this requires relatively simple software.

Then you could have any necessary restart or migration or whatever run in a post-receive hook.

I realise that's starting to get back in to tooling and config etc., but it's not that complex, and if you're already using git anyway...

> and probably very stupid) method is an Expect script that lets me supply a password to sudo.

The old school sysadmin way of doing this would be to have a dedicated deployment user (ssh pubkey auth only - possibly restricting only specific commands), with a sudoers[.d] configuration to allow that user to run an explicit group commands without a password (NOPASSWD)

Also you probably don't want your web under /var if its a partition and you fill it with lets say user content then you will lock up your server.

Alternatively, it's pretty easy to use git-archive (optionally, with an exclusions list to keep out files you don't want to serve publicly) to do something similar without ever cloning the remote repo. You can fetch a zip or tar.gz with a given commit's files, straight from a remote repo.

----------

Static website:

1. Setup NginX using a symlink to the current version

2. Copy the new files to a separate folder

3. Change the symlink to point to the new version

----------

REST service (behind NgninX reverse proxy):

1. Current version runs on port X.

2. Deploy new version and run it on port Y.

3. Update NginX config to use port Y and reload

4. If you need rollback, just reverse step 3.

This can be done using scripts or Ansible too if necessary.

https://ansistrano.com/

It's pretty fantastic.

I've been writing some Ansible playbooks recently for the first time in years, and came upon geerlingguy's work. That guy is a powerhouse when it comes to writing Ansible roles/modules!

    git checkout abcd1234
    ./build.sh && ./deploy.sh

Each schema migration should also come with its own rollback script.

The downside is that you might need three migrations for some operation, but at least you won't break stuff.

The assumption that you can do a schema migration while deploying new code is only valid when you have very limited database sizes. I've seen Flyway migrations break so many times, because developers assumed it was fine to just do complicated migrations on a 200GB database. Or a Django database migration just lock up everything for hours because no one cared to think about the difference between migrating 100MB and 100GB. And I've never seen anyone seriously considering rolling back a Flyway migration.

   rsync -rvzC --no-p --no-g --chmod=ugo=rwX --stats -e "ssh . ${remote}:${path}"

The sites are either on Linode or SSDNodes. Has been working for me for decades and don't need to change it for these hobby things. After all, the log files prove that these things get more traffic from exploitation probes than real people ;-)

Admittedly, these are mostly "local" drive-to-drive transfers over USB rather than server-to-server transfers over house-wide gigabit. But consider trying the transfer without compression.

I'll definitely have to read up on your other flags though, if your rsync works from just one machine rather than both that might solve a minor problem case I have where the destination machine lacks rsync...

https://unix.stackexchange.com/questions/188737/does-compres...

I should review those options, definitely.

> -C, --cvs-exclude auto-ignore files in the same way CVS does

Just goes to show you how long I've been using this stuff :-)

The additional options may not be relevant now that I am just deploying creating a new target directory each time.

If you want an intermediate step you can just make docker images of your application and deploy and run those manually (again with shell scripts).

You can get a long way that way without having to deal with kubernetes.

I cannot imagine not setting up at least a basic CI. Even if you remove all unit/integration test alarms, a CI is still very useful. The 20 minutes of build -> upload -> deploy script -> check prod vs a 1/2 day to set up a CI and forget it.

I'm not sure you save much time but you save a lot of headache

That being said, it would take a lot to convince me a standard CI/CD pipeline is not worth the investment. My only experience with non CI/CD deployment process was a very manual process at a startup, then when I saw my first CI/CD it felt like magic.

If you have multiple people on the team that have access to servers, this will shoot you in the foot a year or two later. Good luck replicating that server setup when no one has any clue whatsoever about all the small tweaks that were performed ad-hoc over the years.

It is easy when the server is regularly backuped.

You would be surprised at how many old enterprises still do this. Sure, it's ugly, but it can be simpler than k8s, and anyone with scripting chops can understand it.

The problems of a solo dev are very different than a dev on a team. Knowledge silos don't exist. Distributed expertise doesn't exist. There's no one to mentor, no shared vision to maintain, no intertia to combat.

I consult on big complicated team projects. I also manage multiple solo projects.

On solo projects, deployment is a script I run from my dev machine. I'm the only person who deploys; anything else would be solving a problem I don't have.

The only "CI" is running the tests before I run the deployment script. I'm the only one who needs to see the outcome of the tests. Anything more would be solving a problem I don't have.

Architecture is whatever makes the most sense to me personally -- which is often miles away from what I would recommend to a client, who needs something any new hire will recognize and be able to work with.

I pay a service to manage backups so I can walk away from a solo project for months and know it's ticking away.

The point is: solve problems you actually have. Don't try to run a "professional" operation by doing what big teams do all by yourself. Big teams have different problems.

This bit seems to hold true across any size team for me. If I have notes I can spiffy them up and make cheat sheets out of them and bootstrap other developers faster. Or if it is just me, my feeble brain will forget odd details that I needed for something.

[1] https://sambleckley.com/writing/lab-notebooks.html

Any recommendation of such service?

[1] https://github.com/spatie/laravel-backup

Yes, you'll pay more for a Heroku Postgres instance than you would for a VPS on Digital Ocean with Postgres running on it, likewise you'll pay more for Heroku Dynos than another VPS to run your application server. On the other side though, your backup, rollback, deploy, and scaling strategies can all be "Heroku does that", and you can focus on the much more valuable job of building a business.

  * Digital Ocean Apps -- somewhat finicky but works very similar to Heroku, it gets closer to bare metal so I prefer it
  * render.com
  * AWS Elastic Beanstalk -- though setting this up is non-trivial, it is very similar to Heroku with its "set it and forget it"
  * AWS Container Services -- if you're using Docker
  * Google App Engine
  * Supabase -- geared for SPAs but this may be the ticket depending on what you're building, is open source

1. Their server manager process got in an unresponsive state. It wouldn't update my service. It stopped reporting logs from the service that was running. I went to the AWS Loft and consulted with the AWS Support technician. They had me log into the machine with SSH and troubleshoot. I was using Elastic Beanstalk specifically so I would not have to log into machines.

2. The console frequently showed servers as running when they weren't. And vice-versa. I wasted a lot of time trying to figure out why my server didn't start when it actually did start and why my server isn't listening on its port when the console incorrectly showed that it started. Logs were also delayed randomly.

3. The final straw happened on a Saturday. Elastic Beanstalk suddenly started returning errors. I purchased an AWS Support subscription and filed a ticket. It turns out Elastic Beanstalk Team had deployed on a Saturday and then rolled back. Unfortunately, anyone who did API calls through the new API version got new values added to their Elastic Beanstalk API backing database. The rolled-back the server errored when it saw the new values. They refused to just fix the bad data in their database caused by their bad deployment. Instead, they asked me (their customer) to download special management tools and perform manual operations to fix it. And I had to pay them to find out these instructions.

Now I use Heroku.

Would be curious how it compares to some of these.

Looks interesting

> Create a backend

But you'd probably host your SPA elsewhere, eg Vercel, Netlify, GitHub pages

BUT: In the last year I've had some rather serious downtime issues that were 100% Heroku's fault. (Poor encapsulation of Postgres configuration via the connection string, and they mistakenly claimed that I violated their TOS when I didn't. They apologized and fixed the situation.)

I'd really like to see some good alternatives to Heroku. Not just because of their screwups, but because the languages that I want to work on have limited support in Heroku.

Edit: Yes, I know there are community supported ways to do this, But I want something that is as slick as the NodeJS ecosystem on Heroku.

Other competitors like render.com are starting to look like they'll be competitive on features and are way cheaper too.

They use self-signed certificates on Heroku Postgres servers without a certificate authority present in the chain. No chance to establish reliable peer verification. A shame, I would love to use Heroku more often without compromising security.

I see this all the time. Entrepreneurs trying to solve the people problem by avoidance. The problem with the super efficient player is twofold. First, it's not really scaleable. Second, what if the person gets sick or is in vacation?

That said, I would also recommend Heroku to many.

The context here is also that of a single entrepreneur building a business. At that point in the lifecycle of a product downtime really isn't that big of a deal, customers tend to be pretty understanding so long as you recover.

Here is what works for me:

CI: From the terminal, I run my tests and commit to git.

Deployments: rsync

Rollbacks: Never did one. If something breaks, I fix it and rsync the fix to production.

DB: MariaDB

k8s: I don't use it. Computers are very fast these days. A cheap single VPS will get you a long way.

Nightmare: Not really. I spend about 30 minutes per week on DevOps.

If you're solo, you probably know enough that fix-forward is the way to go. (This assumes the system isn't critical enough that it needs fixing even if you're asleep, but then you would probably not be solo anyway.)

1: Work on something that you yourself or your users want to exist. Don't think about what others might want. Make something that you consider pretty cool shit. Or something your users are asking for. Something where you or your users say "OMG when this is ready I will fricking love using it! I can't wait!".

2: Most founders, me included, often get stifled. They stop putting time into their project because they think it is just a dream. Remember that all amazing things have been built by people just like you. And to keep putting time into building something that you yourself or your users want.

Not immediately, but in the following weeks.

Thanks again for sharing.

PD: I follow a similar approach. I am glad you said it.

If I want to set it up on a new server, I start a fresh Debian machine, do "apt install mariadb-server" and I am good to go.

I don't like managed services (like Heroku) because:

1: I cannot easily replicate the setup locally.

2: It creates vendor lock-in which will cause me trouble when the vendors service changes in a way I don't like.

3: I would not be able to put the whole setup of my project in one setup script. It is so nice to just have a single script that contains all the steps that need to be done (Like the "apt install mariadb-server") to run the project on a fresh machine.

It's easier than you think if you stick to your distributions' packages. Just need to know how to 1, do auth right 2, enable SSL 3, run with SELinux 4, set a different location (for attached storage) 5, managed backups.

I think if you’re a solo founder you have to accept on day one you’re at a time disadvantage when it comes to the actual hours you’ll spend on moving the business forward vs admin, ops, legal stuff etc etc and so you have to really fight smart!

I.e K8S? Definitely don’t need it. Complex CI/Deployment workflow? Nope, just the minimal shell scripts to do clean deploys and rollbacks. Pay for a HA RDS so you never need to worry about DB replication etc.

If you just work with the minimal set of tools you’re comfortable with and relentlessly keep things simple it’s possible. I’ve built high availability stuff as a solo founder and still found (at least some...) time to work on sales and the business.

Unless you build a side project to learn a new technology, you should use something "boring" that you know well.

Here's my personal choices in preferred order:

1. Static web sites - Just upload to S3 behind a CDN and forget about it

2. Server rendered website - PHP. It may not be something you brag about, but in my experience it just "works" and has excellent uptime.

Note: If I had to recommend something to a larger organization, I would probably suggest Next.js to attract developers. PHP is often associated with "old school" tech

3. REST service - Spring Boot. I have so much experience with it and there are so many Spring projects that supports a huge range of technologies from monitoring to data

4. Scripts - Python

An example would be AWS' RDS

- CI on github actions

- Project management on post-its (short) and READMEs (long term)

- deployment is done by a github action trigger on the `main` branch

- Hosting is using AKS, GKE, or on-prem k3s on a raspberry pi. Want to restart a service ? just kill the pods.

- Devops took about 2 days of work initially and now shared by every project for less than 30 minutes by project.

- Deloying a test cluster (or a test k3s node) is reproducible on whatever hardware / cloud provider you get, sometimes I often create a full-blown dummy cluster for running test scenario

- Certificate renewal is automatic (cert-manager)

- DNS record creation to services/ingress is automatic (ExternalDNS)

- Authentication using OAuth/OIDC is also set up automatically on whatever identity provider you have

- Database backup is automatic

- Load Balancing is built-in

- Job scheduling is built-in (and -fuck- logs are much more accessible on a failed job container than in a custom cron + sh solution)

- Service discovery is not needed

- Monitoring and alerting is not, but cloud providers often have something for you

Note: this is highly effective because I already had significant K8s experience, so, if you're still learning about what Ingress-Controller to choose for your OIDC proxy, then don't go that route.

>> Job scheduling is built-in (and -fuck- logs are much more accessible on a failed job container than in a custom cron + sh solution)

Can you expand on this? How do you handle logs?

What would you recommend instead ?

You have a lot of freelance work and devops positions that will give you that experience. You shouldn't try to do this alone, you need guidance to understand how this ecosystem works.

- Use NFS (EFS), put builds in a /releases folder

- Bash scripts for build, deploy, release

- Bash scripts to wrap apps (write pidfile, start, stop, etc)

- Cron those scripts

- Check crontabs into a repo along with all other config

- Cron a bash script to pull from that repo and update crontabs on every box

- Have a staging environment, deploy stuff there (esp DB migrations) and do smoke tests.

- Have some kind of monitoring (I have a slackbot that sends me messages when things break)

I general I follow the principle of proportional response, and n=2 (or 3 depending on how trivial the task is to automate).

Proportional response means you invest in automation / tooling proportional to the pain you have suffered. Burning a day to build a bulletproof script only makes sense to solve a big problem. For a smaller issue, maybe adding a debug line or writing a line somewhere that you can copy-paste later is good enough.

N=2 means you don’t solve stuff until the second time you need it. This stops you from burning hours building clueless complex garbage to solve unnecessary problems - at least by the time you are automating, you have solved it once or twice by hand and you know where you are going.

Elon’s 5 principles have been very useful to me as a solo dev:

1. Make requirements less stupid

2. Delete the part/requirement

3. Simplify/Improve the design

4. Accelerate cycle time

5. Automate

So in particular you might try 1-4 to see if you can magic problems away before you invest in building devops automation you will then need to maintain.

GitHub Actions

> deployments/rollbacks

Docker. Scaleway offers a container registry that's ridiculously cheap[1]. Deployments are infrequent and executed manually.

> DBs

Again, Scaleway's managed RDS[2].

Outside these, we have setup Grafana + Loki cloud[3] for monitoring and alerting. They have a generous free plan.

For product analytics that can be derived from our database we've a self hosted instance of Metabase[4].

[1]: https://www.scaleway.com/en/container-registry/

[2]: https://www.scaleway.com/en/database/

[3]: https://grafana.com/

[4]: https://www.metabase.com/

P.S. We were a 1 person company when most of this was setup. We're 3 now, works just as well.

It sounds like you have a great set of intuitions, and you're right, all that infrastructure is a nightmare to set up and manage.

Step 1: Question every requirement

Step 2: Drop everything that is not critical

Step 3: Profit!

Git hooks and Makefiles are great!

I have an HP workstation I bought from eBay under my desk that has 32GB of RAM, 2x 12 core CPUs and a static IP address. It's probably 8 years old, but fast enough to serve whatever we need for a long while.

The machine (Old Blue), hosts our Git repos, web app, database and integration service (git hook that calls, 'make publish').

We're not serving Google scale traffic, so we don't need Google scale infrastructure.

Keep it simple whenever possible and don't let the modern stack complexity creep in until you absolutely need it.

Even going to the cloud, when you've done it 10 times and know how, is way more work than you need when just starting out.

Take on those costs and complexities only when your traffic requires it, and you may just find out that you never have to pay rent for compute.

In fact, it's super easy to back up as it's in a SQLite database (for now).

I used to rsync them, but when traffic picked up, I started to fear an inconsistent copy (rsync does verify at the end there are no new changes, and restarts - but i don’t know the exact strategy so I am wear if relying on it).

I switched to doing .backup and rsync that instead - but am considering switching to litestream instead.

If we're talking a plain saas type deal, I'd keep it simple, elastic bean stalk, or use a heroku or render.com like setup until you grow to the point of hiring a team. If it's just a basic saas, I don't see how a 1 man team could really out grow this setup. I've seen 100 person teams using heroku.

K8s is just way too much work. Even cloud formation is to much for my tiny show.

Use the automated backups setup by your host for your db. If you need to roll back, just redeploy your previous git hash. I typically use GitHub actions to deploy, so rolling back is just a matter of force pushing the prod branch to the sha I want to deploy

Skip micro services, they are too much work for a small time thing, and don't really provide much benefit.

I don't have dev ops experience (I still don't know much) so a lot of what I did was to exhaust the limits of the setup before implementing any new infrastructure, sort of learning as I go along, with many 'oh, so that's why they do this' moments. I've read about K8 but still can't see when I would need that though.

A lot of what saved effort for me was to use off-the-shelf component whenever I can. I was quoted $10~20k to build a Django website for user & API management to connect to my back-end but it was much easier to use Azure API Management service (~$400/month), which also came with simple, reasonable looking web front-end that I was able to launch within a week.

Before moving things to cloud (Digital Ocean), I've also exhausted what I was able to do with simple NAS servers (overall I processed about 200TB of raw data with ~1TB/day on-going using what I call a 'poor man's HPC' setup) - luckily I'm based out of Hong Kong so had access to fiber-optic internet at home.

I tried to slap together a simple weekend app based on historical weather data by zip code, and never even got to the coding. Just spent most the weekend trying to decipher those API's.

I’ve been told that my API is easy to use - feel free to let me know your thoughts if you get a chance.

DBs run side by side and are simply backed up regularly with whatever backup solution the VPS offers.

That's it.

Out of school I put way more effort in building my infrastructure but in reality a good VPS, something like Cloudflare and some app based caching can run millions of daily views over 20+ different apps without issues.

Edit:// it's mostly rails apps I host btw. For my wordpress installs i use dedicated VPSes because those scale horrible.

I read such messages (and also agree to it mostly) and then I encounter my peers wanting active-active clusters with K8s on top or something and I feel I like a dinosaur.

Refreshing to see KISS still being used. Also gitea. What an awesome piece of open source software.

> git init --bare

On your remote server is all you need to spawn a repo to push to, usually hook examples already included.

The hook can then directly checkout to my live directory, run migrations and restart the server without weird remote automations after every (or specified) push.

Gitea is just a little extra (setup once, touch never thingy) for quick edits and checks (it's a wonderful lightweight piece of software too)

For:

* Database: PostgreSQL installed through apt in the same server: https://github.com/sirodoht/mataroa/blob/master/docs/server-...

* Backups: MinIO-upload to an S3-compatible object storage: https://github.com/sirodoht/mataroa/blob/master/backup-datab...

* CI: Github Actions + sr.ht builds: https://github.com/sirodoht/mataroa/blob/master/.github/work... + https://github.com/sirodoht/mataroa/blob/master/.build.yml

* CD: (not exactly CD but...) ssh + git pull + uWSGI reload: https://github.com/sirodoht/mataroa/blob/master/deploy.sh

* Rollbacks: git revert HEAD + ./deploy.sh

* Architecture pattern: stick to the monolith; avoid to deploy another service at all costs; eg. we need to send multiple emails? not celery, that would mean hosting a redis/rabbitmq. We already have a database so let's use that. We can also use Django management commands and cron: https://github.com/sirodoht/mataroa/blob/5bb46e05524d99c346c... + https://github.com/sirodoht/mataroa/blob/master/main/managem...

I personally manage the CI/CD infra as well as the data infrastructure for my startup.

Here is my setup:

1. Self-hosted Drone.io instance connected to github(the instance is behind a firewall and can only be accessed via a self-hosted VPN. Github IPs are whitelisted so that auto build/deployment can be initiated via github hooks)

I have drone.io config that helps me manange the versioning, deployment to stage/production, run tests, do rollbacks etc.

for eg: merging to master on github auto builds the master branch on drone and deploys to staging along with pulling the latest db backup from S3 and adding to stage for testing

2. Self-hosted production and staging server that hosts my web/django app. It has a local postgres db and is encrypted and backed up on S3 every hour.

3. Self-hosted Elasticsearch cluster using Ansible playbooks with daily snapshot backups to S3.

beyond this, I self-host 3 VPNs to protect my entire infrastructure. read my old HN comment about it: https://news.ycombinator.com/item?id=28671753

All this is one time infrastructure setup and it all has been running smoothly for more than a year without any hiccups.

I would be happy to help you setup a similar setup if you want. hit me up at vikash@quantale.io

Did you have any instance where you needed to restore the DB / Have you done restoration tests to ensure this is a feasible solution? Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ?

Also, how many days of backup you keep on S3? Considering you create 24 backups a day.

Whenever a build is deployed to stage server, it automatically pulls the latest version backup in last 1 hour and restores it on the stage for testing with the latest data. I have config to do the same restore to the prod if needed.

I have been using the same setup for over 2 years to support an internal web app of a company.

> Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ?

This problem only arises when there is heavy load on the database, specifically high WRITE/UPDATE/DELETE operations. Quantale uses postgres to store only user info and all the other data is stored in Elasticsearch(twitter, reddit, news, SEC, Financial data on over 15000+ stocks). This is intentional because with the scalable server resources, my postgres setup can handle a large number of users before I see that problem. I can always scale up later when needed rather than over-engineering it right now.

>Also, how many days of backup you keep on S3? Considering you create 24 backups a day.

I have 7 days backups on S3 with versioning switched on for 60 days. So at anytime I can access the version backed up in last 60 days.

Safe to say you have a verified way of backup and restore (with worst case data loss of an 1 hour).

I am not exactly an expert on what stage does Transaction ID Wraparound comes in but still know that as one of the key criticisms of PostgreSQL so wanted to check.

Also, 7 days backup with versioning switched is a smart tactic! Thanks once again for this.

I've been running a Python-based, highly custom web store solo since 2007, and it supports multiple people doing fulfillment. I host on Opalstack so as to outsourcing patching, email config, database maintainence, etc. I run directly in a Git repo (actually Hg, it's that old) and releases are "git pull && ./restart.sh". Rollbacks are "git checkout ...".

I've had to migrate/rebuild the VM about every 5 years. Tech changes enough in that time that no automation will still work unmodified. So I just keep good notes about what I did last time, and figure out what the new equivalents are when I finally have to do it again (updating the notes, of course). Database and Conda are easy to port. It's usually DNS and email integrations that are a pain.

As others have said, KISS is key. Industry DevOps is for a work setting with a decent-sized team, where you can afford the overhead of maintaining it all in order to make the overall team more efficient.

Use Heroku (or any other PaaS) and throw in some serverless stuff here and there.

Again DO NOT geek out on the tech. Geek out on getting customers.

In short, I lean heavily on serverless and Github Actions.

Each story covers a different pattern in my stack and I apply these principles:

-I only want to pay for what I use -I don’t have a lot of time available for learning or building -I don’t have time for maintenance activities -I’m not a good UI designer or front end engineer

The DevOps side of things is trivial in the beginning and takes very little time compared to actual development.

Ubuntu server, Nginx, Mysql or Postgres, PHP, Go, Redis. Configured reasonably, it's a ridiculously reliable stack where things very rarely go wrong.

I prefer DigitalOcean these days. Takes <15 minutes to configure a base new setup - a template - from scratch, make a few adjustments and double check everything. From there I can pop up a lot of servers as needed. I usually tweak things based on the project, although that doesn't take a ton of time.

And that's it. Back-ups on eg a database focused droplet are automated by DigitalOcean. Occasionally I have specialized back-up needs, and I'll do a bit of custom work for that. Most of this could be offloaded by using DigitalOcean's database service, I just prefer to limit cost by not.

Under no circumstances would I use Kubernetes for a smaller to medium size service.

Both have a similar setup:

   - 1 droplet with docker pre-installed from DigitalOcean
   - clone directly the repo from github
   - together with the code, I have a folder with a bunch of docker images (caddy, mysql, php, redis, etc.) that I can easily spin up.
   - for any release, I manually ssh, git pull and run the migrations (if any) and manually rebuild any docker image if needed
   - I have daily jobs that dumps and zip the entire DB to S3
   - if I have some deployment that I know will break any functionality during deployment, I warn the users before and accept that downtime.
   - never had to handle a "hard" rollback till now.

I spend 20$ (10$ per droplet) + few cents on S3 per month with them.

I’ve been running my small company for 4 years so far. It’s a podcast search engine & api.

Product & infra & Devop & all kinds of manual processes all evolve over the past 4 years. Things were added / improved on demand.

4 years ago: 3 digital ocean instances + ssh to manually deploy code. No users. So no devops.

Now:

- ~20 EC2 instances, provisioned via ansible

- to deploy code: run a bash script on my MacBook, which is actually to run ansible. The script can also rollback to any specific git sha

- use rollbar + Datadog + PagerDuty for monitoring & alerting

- 3 Postgres database EC2 instances (1 master, 2 slaves). Daily cron job to automatically upload db dump to aws s3 and google cloud storage

- no CI. No docker / kubernatte

- a few interesting devop events are sent to slack, so I’m aware of what’s going on for the entire system on my phone. Eg, when a db dump was successfully uploaded to s3, when a Django request is slower than 5 seconds, when an important cron job fails…

Not very sophisticated, right? But in practice, we rarely have outages and spend very little time to do devops.

The most annoying thing was to upgrade Postgres to a new version. We had to prepare for an entire week and minimize downtime (in practice, disable db write for < 1 minute or so).

I’ve got a couple old blog posts on engineering for solo founder:

- https://www.listennotes.com/blog/the-boring-technology-behin...

- https://www.listennotes.com/blog/good-enough-engineering-to-...

Use tech that you understand and minimize the layers of complexity between deploying, starting and troubleshooting an instance.

For me, I like digital ocean droplet(s). Stupid simple IAM. SSH scripts for CI deployments. Everything sits behind a Cloudflare cache.

The backend sends key events like restarts and fatal errors into a dedicated Slack channel. Its my confirmation a deployment worked and first alert if something crashed.

My business partner and I are currently running a $1m/arr business with this setup on a single digital ocean instance, Python3 and Nuxtjs.

What I have now is terraform on Github which handles Networking (VPC, subnets), ECS/ECR, ALB for my backend and S3/CloudFront for frontend.

Everything is on AWS because I'm familiar with it, so it's faster. And I compensate for costs with AWS Activate + Free Tier. This combination is usually enough to understand whether the project will get traction or not.

Actually spending little time on IaC was one of the best recent investments. But I guess it holds more value when you have a couple of projects, still exploring and these tasks get repetitive

Later, I realized that Heroku + Netlify are good enough. Switching was easy. The hardest part was admitting to myself that I had wasted so much time on AWS. I felt relieved after I deleted the Terraform and Packer code.

I was thinking Digital Ocean. Or Heroku for the current project, but ended up with a design with quite a few serverless pieces using lambda. Which has a great free tier and is very useful for my use-case.

Would not say that AWS is too much complexity though.. When you go over things at least once and understand how thing fall together

I've done this since writing my first CGI app in 2003. And it hasn't changed other than the commands I call. Back then it was manual through GUIs (visual source safe and FTP) and now it is git and scp, but 80% of the process hasn't changed in almost 2 decades.

If it breaks (which it shouldn't, but inevitably does), I just git checkout the previous commit and scp again, and try and fix the issue.

If I have a "high availability" product (which I hate doing), I will release onto a clone of the production server, then test it, swap to that clone, deploy to the others, swap back and kill the clone. It is a much more involved process that I really don't like doing, so I've stopped making high availability applications.

But that is web, so it's super simple. Also to avoid issues regarding data schemas, I always try to make sure my changes are additive. And on startup, my web server checks the database version then runs any upgrade scripts if needed. If the deploy fails, the db changes stay, and the code just gets updated. This is sloppy, but it's simple.

When a real schema change needs to happen, I abandon the project (just kidding). I usually do it in multiple updates. New table with the schema changes, upgrade script transfers the data, and code to point to new table. Manually verify that all of the data copied. Backup the old table and drop it. Then another update to rename the new table to the old name (if needed) and it's done with minimal disruption in service.

Envoyer and Forge for PHP stuff.

Render/Heroku for node.js stuff. Netlify for static stuff.

For the more complex bits I have, I use Pulumi to manage everything through IaC. It's still complex but at least it's robust and I can sense-chech changes to infra through a PR to myself.

I create my small/bootstrap projects the following way:

1- create a free tier AWS account - can work with any VPS/server really, but with AWS you can get it 100% free :)

2- create some Ansible provision/setup/deploy scripts

3- create some bash scripts to wrap this all

Create the Ansible scripts:

1- Provision script, consisting of 1 EC2 instance, 1 RDS and the security groups. Store the IPs/adresses of the newly created instances.

2- Setup script, basic instance config, packages, app/project repo, you name it

3- Deploy script, to run every time I want to update the app, mainly just doing a git pull on the instance and some database backup/migrations (although on RDS backups aren't always necessary)

I get these Ansible scripts wrapped into basic bash scripts, provide my AWS credentials via the AWS cli to keep it safe, few extra creds with Ansible vaults, SSH keys, publish in a Github [private] repo and I'm all set.

It took me a couple of days to get fully operational and I was learning Ansible at the same time so it can really be done with basic features. Now it's done, I can reuse the same skeleton for every new project in a couple of hours!

I find this solution extremely resilient. That's basically free hosting for personal projects. Every year, just need to run the script again when free-tier expire, and can change hosting provider anytime or upgrade the instance type if I decide to get a project publicly released. A small extra Ansible task to write for migrating the data and that's it!

I keep it super simple. I have a git repo where I directly commit into master. When I want to take the changes live I ssh onto my VPS, I git pull the changes and restart the website service.

I use an SQLite DB that I automatically copy (backup) to my local machine daily via the Windows equivalent of a cron job. Once it's on my local machine it's also backed up to other places as part of my local backup process.

I run my tests manually when I think I've made significant changes.

Otherwise I just react to errors ASAP (I get an email notification from sentry when a new error occurs)

A lot of this doesnt work with a bigger team of course.

It's a balance between risk and time investment. The simpler your dev ops stuff, the higher your risk of introducing bugs into production. But it saves you a lot of time not dealing with that stuff.

We're building [1] something similar to solve this problem but just on the infrastructure end. I really believe in the power of templates. In our future release, we're planning on launching a "infrastructure" catalog of commonly used setups so that you can just get the foundation in place. Why re-invent the wheel every time when you can just spin up a boilerplate that gets you 90% of the way there?

[1] https://atomizedhq.com

Good luck!

It isn't that hard, for someone with decent experience, to setup TeamCity/Jenkins/Octopus/Whatever but why spend the time? If you change something, you then have to go and spend time changing the automatic deployment.

As someone else said, Devops is about velocity with quality, if velocity is OK and quality won't improve with tooling, don't bother.

services:

  certbot 

  haproxy

  db

  api

  client

build:

    stage: build

    script:

        - docker-compose -f docker-compose.yml -f docker-compose.release.yml up -d --remove-orphans --build

Unless you're doing something very slow / processor-intensive, you'll probably never even need an autoscaler before you can afford to hire an expert to do it for you. You'd be surprised how far you can get with a small VM running your whole stack.

I am not sure how it would work on a larger scale, but for my use is perfect.

Vercel first paid plan is $20/month and FaunaDB $25/month, so my guess is a business with some traction would have to pay at least these $45/month.

I’m a solo-founder, with many years working in IT, and I focused on DevOps for part of it; I know what best practice looks like and it would be easy to fall down that rabbit hole doing an unnecessarily complex buildout. Currently I’m doing a small private beta, I’m avoiding 99% of standard practice. I’ve got a single EC2 node running Redis as the only datastore and NodeJS + Nginx + certbot, a cron to do backups to S3. No CI (there are no other devs to integrate code with) I run all tests locally and push/rollback with rsync. All code, assets, and server config (except creds) are in a monorepo.

If the server goes offline, I will have a little downtime, that’s fine. If I run out of memory for Redis (not likely to happen soon), I’ll change to a different datastore or scale up the node. If I lose data, I can restore from S3, and additionally the architecture is such that clients will re-push their latest changes.

Do the bare minimum to support the business, stick with what you know, outsource what you can, and properly value your time.

Dedicated servers, specced to provide lots of capacity for spikes. A 20-node k8s cluster could fit on 2 beefy dedicated servers for about the same cost. Decreased infrastructure redundancy but massively increased operational stability through simplicity.

Everything runs in a docker-compose project: one for the API, another for the everything else monolith. I've worked for a couple of small companies that ran with docker-compose, so have a good sense of the weaknesses and footguns (breaking your firewall, log rotation, handling secrets, etc).

CI is running `make test` on my dev machine. Deployment is `git pull && docker-compose up --build`. Everything sits behind haproxy or nginx which is set to hold and retry requests while the backend is down, so there aren't any failed requests in the few seconds a deployment takes, just increased latency. I only deploy to the API once or so per week, that stability reduces headaches.

DB backups are done with cron: every hour a pgdump is encrypted then uploaded to backblaze. Customer subscription data is mirrored from stripe anyway so an out of date DB backup isn't the end of the world. Error if the backup fails after a few retries.

Sentry on everything for error alerting. All logs go into New Relic.

- Manage your own DBs for production. I used to manage my own but now I use IaaS for it. It's worth the extra cost even if your budget is low since it lowers your risk and lets you move faster. AWS Aurora is awesome but expensive. There's a SaaS/IaaS out there for most DBs. And they handle the backups!

- Use Kubernetes. k8s is awesome, I love it, but it is a beast and probably overkill for your solo ops team that is also multitasking.

- Manually SSH / rsync / scp directly onto prod.

Do:

- Version control (Git) everything

- Use containers where it makes sense (I use Docker when I'm not using Serverless). It keeps your dev and prod in sync and helps with immutable deployments (see later points)

- Use Infrastructure as Code

- Use Github actions to do your deploys

- Use immutable deployments. More time to setup but makes rollbacks easier and if you never hot patch production you can't create a unique state that can't be restored.

Finally, (and possibly controversially): use serverless where you can (I personally use AWS Lambda, S3, and Cloudfront).

My personal flow:

- All my infrastructure is in a parameterized CloudFormation template.

- The template is checked into Git with the service code.

- After CI runs the tests, CD builds my serverless functions and uploads the zip.

- Then it runs the cloudformation template.

The same flow also works for Docker except "uploads the zip" becomes "pushes the container"

Using this setup I can tear down and build up my setup in minutes, making it easy (and cheaper) to separate dev and production.

Why, though?

If you're really on a low budget, the extra $$ for a managed DB doesn't really pay for itself. I'd rather prefer spinning up a container running whatever DB I need and run daily backups to an external vendor.

In my case, the website is free, living on ads revenue, so every penny counts.

If you compare it to a single machine with multiple tenants and no replication, it doesn't make sense. But that comes with risks and given the OPs question in the context of a k8s cluster (which for HA is a minimum of 5 servers), I figured they are prioritizing time savings over money.

Losing customer data, downtime, and data breaches can kill some companies before they even take off.

If your particular business is not sensitive to that than you can run your own database. Or better yet, if you're a content business you may not even need a database at all.

But even so, if you run your own DB you need to worry about backups and such so it may even still be worth the money. And if you are going for High Availability (HA), running a properly replicated cluster can be time consuming.

The cost of a t3.medium RDS instance on AWS for instance is $0.068 / hour compared to a t3.medium instance on EC2 is $0.0416 per hour. That is an extra $19 per month. For my use case, paying an extra $19 a month is well worth the savings in time and the piece of mind I have knowing my backups are running and if the server crashes I can recover quickly. But you must consider use cases and your's might be different.

I might being naive here, still:

   - losing customer data: the backup must be done frequently enough. So I'm not risking losing data.
   - downtime: I'd expect that RDS could still go south as well, maybe less frequently, but still. In my case I start another VM, run again the container and apply the backup.
   - data breaches: Of course one can misconfigure something here, but not sure how it'd be different using a managed database. 

$50 total would be tough with fully managed AND high availability. I agree. But to digress from the tech side a bit and talk my own experiences, the question I ask myself is: as a solo-prenuer can I use that time to be doing something that will make me more than $X per month? If the answer is "yes, immediately!" I go with the managed server. If the answer is, "it would take me years to break even"... I may roll my own.

To address your points, thought: I think managed databases do help here. Data breaches and outages are usually configuration issues. Since they start with a good known configuration, managed services do help. Also, I see not being able to SSH in as a positive in this scenario.

For example: Say you misconfigured a database an break backups (happens all the time even to good DBs). That is much harder to do on a managed database.

If you're managing just one machine and being down for 10-30 minutes while your backup is being restored is fine, you probably don't gain much by using a managed service. In that case I'd take frequent backups, upload them immediately off the server (probably S3), and containerize the database. In my architectures I always run at least 3 servers.

With most database engines, if you have three servers you need to lose 2 to cause an outage. But a 3 server cluster is much harder to manage manually, tilting the scales more towards a managed database.

Also, to your point, would blow $50 out of the water.

If I can, I use a managed serverless database like DynamoDB (there is also others) that are pay as you go. You can easily run a service on just DynamoDB free tier if your use case is well suited to that kind of key/value stored based DB.

You can also, always start with your own and switch to managed later as you add machines to the cluster.

Just, whichever you choose... but "especially" when rolling your own... make sure you run fire drills. I.e test your backup and restore process regularly.

Until you decide to move away from AWS. ;-)

Nevermind, ignoring the potential lock-in there are many pros to this setup.

1. Vendor lockin when you are a startup should be a low priority in my opinion. Your priority should be moving fast and getting customers.

2. To expand on 1. It's not really vendor lockin, the only things vendor specific are your infrastructure templates and interface code. Rewriting the interface for another cloud provider takes a fraction of the time of rewriting your codebase (source: I've done it many times).

To come full circle to the question, one thing you should probably not do as a solo-prenuer is concern yourself pre-maturely with vendor lockin at the expense of getting a working product.

Edit/Example: The code for interfacing with AWS Aurora is the same as any other MySQL or PostgreSQL code except for authentication. Switching to a non-IaaS database WILL take time but no more time than rolling your own. So your choices are not "lock in" or "freedom".., it's, spend lots of time setting up databases now or later when you have proven your business model (and potentially never)

Edit 2: This is an anecdote certainly but... of the dozens of startups I've worked on and with, some of which have grown pretty large. The only vendor lock-ins I've ever found to be issues have been with software frameworks (especially ERPs). Never infrastructure. I used to be worried about infrastructure lockin, now I prioritize product market fit.

As for DBs / engines and such, I try to go for hosted services as long as they're not stupidly priced. I'm currently using Elastic cloud. If I need a relational DB I'll probably deploy Postgres on AWS and install some scripts to back up the data to an S3 bucket every day or so.

I chose NOT to use k8s or Docker because, like you said, it's basically a fulltime job to maintain. I don't think it's needed at all unless you're a large organization.

So basically: a mixture of roll-your-own, streamlining platforms, and hosted services. Do whatever works and is easiest, and don't worry so much about having ALL the goodies.

Also, I review how much time each product / feature consumed each year and then decide what to deprecate. Each April I then have my own little party where I ceremoniously turn off last year's time wasters.

I'd say it all boils down to "know your customer lifetime value"

Here are the list of DevOpss flow I've done:

* One project is a website. I have a pipeline, triggered when something is merged to master, that builds Docker images, pushes them to a Dokku instance and deploy them.

* One project is a JS library. I have a pipeline, triggered when a git tag is created, that packages the library and uploads it to npm. I use Netlify to host the documentation website of it.

* For this same project, I'm planning to have a pipeline triggered for each PR to run the linter and tests.

I work as a DevOps/MLOps, and I'm quite comfortable with CI tools in general. I hate doing things manually and I love automating, so it comes quite naturally for me to do it. Once it's set up, it rarely needs to be changed.

GiHub actions-> docker build->docker push to github package repo

then on vm a 2 line script to docker pull && docker-compose up -d

Database all by hand.

Extremely easy i just push, wait build and restart

New services like render.com make it really easy to get robust, cheap hosting up. Much cheaper than Heroku.

For simple monoliths it's a non-CI method plus scp, for infrastructure-only it's SaltStack and Terraform. For something where the DevOps part is the deliverable, it can go wild pretty fast: GitLab CI with various CI steps (static analysis, building, unit tests, integration tests, browser tests, blob storage of the result), then CD steps depend on what it is that was built; docker images just get pushed to a registry and another async job listens for pushes and does the deploy in Fargate or Kubernetes. For some jobs with larger needs you get into ArgoCD and Kubernetes resource deployments that way.

Essentially it depends on the zoom level:

Zoomed in to the max: just a build.sh or makefile, optionally called by a CI or IDE of your choice so the building is exactly the same no matter what invoked it.

Zoomed out a little: the results of the build need to end up somewhere, sometimes a step in between like a package registry, docker image registry etc.

Zoomed out to level 3: once the deployment gets actually done, the runtime itself is responsible for checking for database migrations, locks, maintenance mode, so that has to be part of the startup. If it's a multi-instance workload it depends on database migrations if there has to be downtime.

Zoomed out to level 4: if it also needs to deploy 'other things', like rules and records to Cloudflare, resources in AWS, resources in K8S and even update some stuff in buckets or DynamoDB, that requires some orchestration and DSL like Terraform, SaltStack, Ansible and application-specific CD like ArgoCD or Flux or even just the aws cli tools to trigger a refresh.

Customer decides how much money they want to spend and that scopes what zoom level they get. Usually depends on how many developers or ops people they have themselves and how involved they want to be and what continuity or guarantees they are looking for. The only thing that is not optional is that everything has to be in Git. If a customer doesn't want that, they cannot be my customer. I'm done living in the 90's.

This was also how things began at a previous startup I worked at until we grew and we could hire more resources to fix things.

I have a few old boxes that run VMs. Some of these VMs make up a virtual k8s cluster, but almost everything I need runs on bare VMs rather than on k8s.

When I find something particularly annoying to administer, I move it off a bare VM and onto k8s.

The main thing this gives me is a migration path off of "a bunch of custom shell scripts" toward something I believe will be useful in the long run.

A bunch of shell scripts and rsync will work for a long time until they don't. And when they stop working you'll be in for some pain.

I didn’t do CI/CD. I just reviewed my own code the next day, improved it and put it live. As the only dev I knew more or less how my code worked and what risk factors to prepare for in each deploy, and serious, business-impacting production issues were rare.

A simple stack of Meteor on Galaxy + Mongo on Atlas meant all my tools integrated well together and a whole lot of devops things were easy enough that I rarely had to think about them after they were set up once.

I can think of many cases where this would not work but it worked well for my context.

I have multiple projects, none of them making money currently, so $100/month for each basic app is not good.

I switched to using Dokku on Vultr, $12 a month. You can easily create Postgres databases and link them to Docker apps. I haven't bothered to setup CD yet but it looks like it should be simple, for now I just push to it when I want to deploy. Liking this setup so far.

For frontend I use Netlify, I use their redirects to proxy the backend.

SVN checkout development, meld into testing. Copy production db into testing. Run a db upgrade script in testing. Test the new function for a day. If all ok stop the production HAproxy at 3am. run production DB backup, Checkout testing into the production server. mount the new dir into the NGINX path. Upgrade+start the DB, start production HAproxy.

Except for the meld and testing part, this is all automated.

A rollback on the app servers is pretty simple, just remount to the previous location. Even though there is a backup of the DB, a rollback there is not really feasible.

Everything is running in LXC containers, which I treat as cattle. Creating new ones is either automatic based on time/demand/failover or a few clicks in a custom web interface. HAproxy automatically pics them all up via DNS.

SVN and production DB are always backed up nightly and copied to a S3 compatible storage. Everything older than 6 months gets deleted. All the free space on the servers is globbed up with glusterfs to give a decent amount of fast,free storage.

Using Bash as scripting language, there is literary no problem in devops which hasn't been solved with bash and the solution is somewhere in the internet.

1) Server is high performance native C++ application running on rented native hardware on Linux. It processes thousands of requests/s in sustained manner hence no real need to use any of that k8s / docker / etc. There is also separate rented standby server.

2) I maintain a reusable single script that can rebuild complete system from the scratch with a single command. I periodically test it on local VM. It installs all needed compilers, tools, PostgreSQL, restores database from the latest backup, checks out from VCS and builds that C++ server, registers it as a daemon and refreshes Javascript Frontend.

3) I also wrote small server that subscribes to webhooks from VCS and can build artifacts. I only use it to pull and build Javascript part. It could do main server as well but being super cautious I trigger that step manually fingers crossed.

3) for DB I use automatic scheduled backups.

All in all after I've debugged and tested new version locally the release takes seconds. Script is written years ago and does not require much maintenance other then updating artifacts to new versions (replace apt-get install XXX-31 with apt-get install XXX-32) and register and build ZZZ server instead of YYY server. Compared to insanity I saw in some orgs my setups are piece of cake.

Provose is an open source project written in pure Terraform, so all of the high level abstractions are computed locally, and in general do not raise your AWS bill.

I started working on Northflank to make it more simple for developers either solo or in a team to manage and automate DevOps complexity away.

We support CI for GitHub, Bitbucket & Gitlab (SaaS and self-hosted) with either Dockerfiles or Buildpacks. We have a robust and feature rich platform via UI, API and CLI. Out of the box you get end-to-end DevOps: CI/CD/CD, Horizontal and vertical scaling, persistent workloads with managed (MongoDB, Redis, Postgres, MySQL, Minio), backups, restores, real-time observability and metrics, DNS, TLS, mTLS, Teams & RBAC and more…

Let me know what you think of our offering + site!

Platform: https://northflank.com

Application Documentation: https://northflank.com/docs/v1/application/overview

API Documentation: https://northflank.com/docs/v1/api/introduction

On that it was a Django product with deploy scripts for staging and production, i.e. `./deploy_production` ran:

   #!/bin/bash
   ssh  web@[production ip] << 'ENDSSH'
   cd /home/web/
   ./pg.sh
   ENDSSH

Last summer I moved away from the spookies and now do all deployment using Github Actions.

Commits to either a feature branch or `develop` run tests and all necessary deployment steps to a staging instance. (my stuff so far does not require k8s) and all containers are put up using docker compose. Commits to `main` (or `master` depending on the project)

I blended a few guides written by Michael Herman to build my GA-based CI, and it took several weeks to work on the scripts and learn Github Actions. But it was time absolutely worth investing because my DevOps is modern enough now.

No more bare metal stuff. No more scripts. Nice visual display of deployment success, "free" VMs for the deployment and CI UI integrated alongside my issues and code all on Github.

Importantly, I still use git flow. I tried moving to trunk based, but the macros inherint in git flow (including the pycharm plugin) make this production / staging // master / devops or feature/ work very well (I'm not making PRs to myself)

On every stage (dev, test, prod) I can deploy it automatically up to two times (blue and green) by running a simple shell script. So when I update the application I just deploy it another time on prod, test it one last time by adding the IP of the load balancer in my local hosts file and if I'm satisfied with the result switching the DNS entry to the new version using a weighted DNS record with the ability to switch back until I shut down the old version.

Doing anything continuous doesn't feel worth it in such a small setup with one update every 1-3 months.

What I like most about the approach is that I'm free to change any aspect in the main template without any risk to break something in the live application. Only changes to the elements shared between versions need to be handled carefully.

The main template includes VPC, network, fargate service, loadbalancer, firewall, KMS, DNS records, access rights, database tables, queues, monitoring metrics, email alerts

It excludes everything that is shared from update to update which are defined separately and just referenced from the main template such as some database tables, persistent storage container registry, user groups (Only the groups, not the rights assigned to them)

If you aren’t already comfortable with K8 I wouldn’t use them, there is a learning curve there for sure and not necessary.

I wouldn’t even worry about containers at all unless you are planning on using a managed container service like elasticbeanstalk.

My CI pipeline is almost nothing, just run tests on main and dev commits. I have no deploy pipeline, I choose to actively deploy.

My deployment workflow is a powershell script since I’m on Azure, if it was AWS and I was using plain EC2 instead of a container service I would probably use ansible to avoid having to repeat commands 2-3 times when I deploy something behind a load balancer, but it would be super minimal.

DB backups are super easy to schedule with RDS or similar offerings. For rollback with database changes I would schedule downtime, let your customers know, replace your page with a maintenance banner, do upgrade, test upgrade then repoint site to your running instance.

Overall I echo what many have said, use familiar tools as much as you can, don’t worry about whether what you are doing will scale to even 2-3 people if it costs significant time. Use managed services where possible to save you time, usually the cost isn’t high vs doing it yourself.

Basically, building a docker image and server it using Docker compose. Of course you'll need NGINx as a reverse proxy running on the server.

If I do it again I'll use a service like render. Not worth managing this myself.

At the start of this year I joined hands with Chris on https://hatchbox.io to work on cost effective deployment service. We are putting together all the deployment best practices we've learned over the years into this product in a cost effective way.

As for rollback, I don't really use rollback. I have a roll-forward strategy. My DB deployments are migrations and so long as there's no data-loss caused by a migration (my migrations are always backwards compatible with the previous version) there's no need to roll back. Azure brings point-in-time restore which can be triggered if necessary.

I think the key is like with anything. There's a learning curve on your toolset. Once you've overcome that hurdle and you know to set up automation for everything from the outset. You build your pipeline shell. You never do anything manually. If you need infrastructure of any form, it's always code-first, included in source control and done the right way. Never do anything manually and rely on your automation to carry you. It seems labour intensive up front, but once you're there, it's sustainable.

It takes discipline not to cut corners. The minute you start to cut corners and get lazy is the path leading to your doom.

We are all in on AWS.

0. Tech stack: NodeJS, React, RDS Aurora, Redis, S3, Route 53, SES, Lambda and a few more.

1. 3 EC2 machines behind an ELB. Deployment is scp + cd <deployment-directory> + ./deploy.sh prod

2. Aurora takes care of everything around DB for us. Backup, read/write replica etc., etc.,

3. No CI; our development cycle isn't rapid enough to warrant investing in CI.

4. We recently moved some of long running tasks to Lambda. There's definitely a big value here so we'll invest further on it.

5. Also, recently, we experimented with an internal service on "ALB + AWS Lambda"; it's a sweet combination so we'll invest further.

6. Frontend is hosted on AWS Amplify. It's an underrated nifty service; highly recommend it.

7. Datadog is for monitoring. It's quite good but a bit expensive.

I can tell you from my experience that CI, auto-rollback etc., are overrated for smallish teams given their development velocity. Even if you have to do once or twice a deployments per week you can do it manually, don't need to wade into the CI territory. And unless you have prior experience steer clear of k8s.

There are a few rough edges you face once and then forget about but minimal resource impact and it was fairly simple. I have another project that is using hacky scripts and it's a pain when something is off or I need to run something else on top of the machine.

Some stuff is not terribly maintained but everything works.

If you're small, it's definitely not a full time job. I didn't pick k8s because I initially deployed on a single machine and didn't want overhead. Things kept working afterward and I'm reconsidering whether I need k8s at all.

Rollbacks are just deploying an old version + optional down migrations if needed. I try to minimise them and fix the code and redeploy anyway.

For migrations, I guess it's very application specific, but most tools in most languages are fine these days. I use typeorm, a node.js library.

There are some practices you can follow that you probably heard working in some normal company. Eg. I don't break the db structure immediately, I deprecate columns and then delete them after a while.

1. common (Google leap second smearing, increase max open connections, misc security and performance changes)

2. infra (install Nginx, install RabbitMQ, install Postgres, install SSDB)

3. product (run common stuff, then infra stuff, then any stuff specific to a single server)

Sometimes it breaks after upgrading Linux, but it's easy to fix. For CI, I use GitHub Actions. Deployments are mostly built with GH Actions. If deployment fails it just keeps the old server code, there is no need for rollbacks I just re-deploy with working code. If deployment passes but there's still a prod bug, I don't rollback. I just commit a revert and deploy that or commit a fix. LetsEncrypt renewal, log aggregation, db backups is all custom with combination of RabbitMQ and CRON + Shell scripts.

I miss out on autoscaling, so someday I'll invest the time to migrate web servers to hosted k8s. With limited time as a solo-founder I have to prioritize for impact meaning incremental improvements usually never get done.

Not having k8s and autoscaling isn't causing any problems, won't increase revenue, and won't save any meaningful expenses so it's currently only an incremental improvement for me.

Finally, I have a small Slack community for bootstrapped founders. Let me know if you want to chat in there about specifics.

Currently going with a container image as the minimal deployable unit that gets put on top of a clean up to date OS. For me that's created with a Dockerfile using Alpine image variants. In a way I could see someone's rsync as an ok equivalent, but I'd do versioned symlinked directories so I can easily roll back if necessary if I went with this method. Something like update-alternatives or UIUC Encap/Epk: https://www.ks.uiuc.edu/Development/Computers/docs/sysadmin/.... Anyone remember that? I guess the modern version of Epkg with dependencies these days is https://docs.brew.sh/Homebrew-on-Linux. :-) Or maybe Nixpkgs: https://github.com/NixOS/nixpkgs?

Deployment-wise I've already done the Bash script writing thing to help a friend automate his deployment to EC2 instance. For myself I was going to start using boto3, but just went ahead and learned Terraform instead. So now my scripts are just simple wrappers for Docker/Terraform that build, push, or deploy that work with AWS ECS Fargate or DigitalOcean Kubernetes.

No CI/CD yet. DBs/backups I'll tackle next as I want to make sure I can install or failover to a new datacenter without much difficulty.

1. Tricky and unfrequent changes: - mail server, load balancer, database etc: for this I just write bootstrap.sh, rsync to server and run. - It's literally just a for server in `foor bar`; scp ; ssh ` done

2. Deployment/Frequent changes: - docker-compose to spin up everything. It's super nice. Again, the deployment is done with a `rsync` then `docker-compose up -f docker-compose-prod.yml`

Eventually when deployment changes very frequent and need scale/ha I added in Kubernetes. K8S is way easiser to setup than you think and it handle all other suff(load balancer, environment variable etc).

And my deploy now become: `kubectl apply -f`

One trick I used is to use `sed` or `envsubst` to replace the image hash.

For backedup, I again, literally setup cronjob from an external server, `ssh` into database and run `pgdump`.

I also have a nice NFS server to centralize config and sync back to our git repo.

I used this whole setup to operate https://hanami.run an email forwarding service for the first 3 months before I added Kubernetes.

> I self-host three VPNs to protect my infrastructure heavy startup https://quantale.io

I am a big fan of Pritunl which is opensource and provides network security with a lot of ease. I am in no way affiliated with them, I am just a big fan of Pritunl. I use Pritunl to limit access to servers and web applications for my different teams. For each user, you can generate a profile and assign the servers and port they have access to on the server. For eg:

- Only dev team can access ssh port(22) on stage server and not open to internet.

- Any one in the team can access stage version(port 443) for testing purpose.(Not open to internet)

- Only I can access all ports on all Prod servers(only 443 open to public)

What hackers can't see, they can't attack. Especially the port 22 on your servers should only be accessible to you and not the internet.

I self-host one instance each of OpenVPN and Wireguard with Pi-Hole which is then used to access my Pritunl Server adding extra layer of security.

Each of these 3 servers can be hosted on Hetzner $2/month instance. With a mere $6, you can add an extra layer of security to your infrastructure. Pritunl itself also provides subscription so that is also an option.

https://news.ycombinator.com/item?id=28671753

If you want to discuss more about this or security in general, feel free to reach out to me on my email on profile.

Databases:

use AuroraDB on AWS. Pros: no server maintenance, no backups, no maintenance at all really. Automatic snapshopts set to 30 days. I back it up once a week to an S3 bucket for longer term storage. That S3 bucket is "aws s3 sync" to a local drive on a debian machine that has a tape drive in it. I do the tape backups just after the weekly DB backup.

Code deployment: For static assets I use GitHub actions. For the containerised servers I haven't migrated to GitHub actions yet (.NET Core app running in an Ubuntu container) but since the system is usually deployed with the AWS docker tools that should be easy to automate. Lambda functions just use the AWS dotnet command line tools.

This is all much easier when you use a debian machine as your main dev/integration box. You can use WSL2 as a reasonable substitute. The command line AWS and dotnet tools are much nicer in a unix shell as you can do things like use a Makefile or a shell script to capture the more complicated actions. PowerShell is an abomination.

Thankfully lots of tools do most of the heavy lifting. I use k8s, GKE does most of the work for me. It's very nice to have autoscaling for traffic spikes. Same with database (MongoDB Altas), dead simple autoscaling. I would never run my own k8s nor database.

I wrote more details about some of the Ops stuff I do here in a previous similar question: https://news.ycombinator.com/item?id=26204402

Coincidentally I also wrote some architecture notes about a new product last night: https://blog.c0nrad.io/posts/slack-latex/

I think everyone's milage will very, but as general principles, staging is nice, reading docs saves time overall, tests help you sleep at night and make it easier to make changes 6 months in the future, simple health checks (or anything on a critical path) help you catch the real issues that need immediate attention.

Good luck!

I tried to optimize for cost and very little devops at all. I still mull about it but so far, here is what my architecture would look like.

- no servers to manage - static website hosted on CDN, and even dynamic user-specific pages would be built statically every 8 minutes - all writes go to an SQS queue - every 8 minutes, a lambda is spun up to batch read the SQS queue and all writes go through a single writer process that writes to a SQLite file on EFS - every 8 minutes, a new static version of the site is built using the above hopefully updated SQLite file

The one thing I hate about this is that I can't in good conscious say to the user that their write request succeeded b/c no state has actually changed on the backend yet. but I might be okay with this tradeoff and just simply showing only them their most recent write.

The other tradeoff is when things go wrong, it'll probably go terribly wrong.

If it were a private repo I'd still shell out for Github Actions or Circle CI most likely. I'd also consider buying a chunky-enough minipc for ~$500 and an older mac mini and set up runners on them.

For the moment private runners isn't a problem. But soon I'll need to start integration-testing proprietary code paths like querying Oracle or MS SQL Server. In that case I probably need to set up a dedicated box with all the right licenses so I can run CI jobs on it.

[0] https://github.com/multiprocessio/datastation/blob/master/.g...

The server accepts .jars over HTTP with code (and files) so I can hotdeploy while developing on live on the entire cluster in real time. My turnaround is about 1 second.

The JSON database allows for schema-less simplicity, and it has all the features you need like indexes, relations, multi crossref and security (and then some, like global realtime distributed while still being performant) in 2000 lines of code.

I have zero pain developing the most scalable (and energy efficient) backend in the world, yet very few seem to care or use it: https://github.com/tinspin/rupy

It has been proven on a real project with 5 years uptime and 350.000 users: https://store.steampowered.com/app/486310/Meadow/

https://store.steampowered.com/app/1152340/Book_of_Travels/

I know it had some server issues on launch - too popular it seems.

BoT did not fail because it had too many players, you can clearly see how many are playing, maybe it has a complex backend... simplicity is your friend!

Meadow for reference launched with ~150 (on T2 Micro at 30%) concurrent players per machine and peaked at 450 (on T2 Medium at 25%). That's before you could buy more credits so we had to upgrade to M4 Large after a while.

And Meadow is an action game, not a point and click!

The rupy chain is solid and proven and has been running Meadow for 5 years without any write downtime except last week when one instance got nuked by AWS or somebody at M&D. That's a pretty good track record of 100% read uptime!

The Meadow launch probably was the only MMO to have a flawless launch with high pressure per server so far, and it did so over HTTP with a JSON file database! :D

It's also the only MMO with one database spanning 3 continents in real-time.

Try https://treblle.com/how-it-works

Treblle allows you to test API requests with one click. Because Treblle knows what data was sent and what was returned it can replicate any call that the user has made and quickly allow you to re-run specific requests. Treblle also provides you with a way of running manual tests from our platform.

It also has an auto-generated documentation feature. Treblle can generate the required documentation for each endpoint just after 1 API call. It understands your JSON responses, can detect various authentication methods, group variable URLs in a single endpoint, support different documentation versions, and similar.

https://www.youtube.com/watch?v=3_W7zHrsM7E&ab_channel=Trebl... Here is a quick overview.

Let me know what you think.

DevOps is not a synonym for "operations work" or "server work" or "devs doing operations work".

DevOps literally means "dev teams and ops teams working together". That's all. You don't "do DevOps" unless you are "doing the work of collaborating between two different teams of people".

[0] https://anthonynsimon.com/blog/one-man-saas-architecture/

[1] https://www.wyzetickets.com/

[2] https://fluxcd.io/

I simply run "make deploy" and a new production build is made and rsync'd to production. Then the script executes a "deploy.sh" on the production server which reloads the system process to pick up the new binary.

Poof.

I do daily backups and have the ability to deploy a "new" server if need be. It's a little bit of manual work but would be trivial.

With that said, I run a semi resource intensive operation, so I've invested a bit into dev ops to keep our costs down. My setup is currently on AWS, primarily using ECS, RDS, Elasticache. Infra is managed via Terraform.

I felt ECS was a nice balance vs K8s, as it's much simpler to manage, while getting the benefit of maximizing resource utilization.

For CI / deployment, I use Github Actions to build an image and push, then start a rolling refresh to update the containers to the new version. It was pretty easy to setup.

On DBs, RDS handles all the backups and maintenance. For migrations, I use https://github.com/amacneil/dbmate.

Happy to answer any other questions you have, as I've learned a lot through trial and error.

If it’s just you, you can afford to use a simple shell script to run tests locally before deploying or use a Travis/Circle/Cloudbees CI option too.

You have enough to focus on with the application itself.

I've got a big client who went from 6 infrastructure engineers to zero. They survived for half a year without any real problem. You either pay a company to manage your infrastructure for you or you pay a person. Devops people are expensive.

Things I've done to automate DevOps Make sure you pay attention to automatic testing and deploys with Ci/CD. I like automated deploys on Gitlab. Trigger staging release on merge into main, and deploy prod on merge from main into a deploy branch. Infrastructure as code with Terraform. K8S is more trouble than it is worth (though it is robust after you invest 3 months into getting it tuned right, although a lot of people make critical mistakes in their config, I've seen a half dozen insane mistakes.)

Monitoring - Loads of cron jobs pinging slack (monitoring disk, CPU, Network I/O etc), Healthchecks.io, TurboAPI, Nginx Amplify. Mostly built up over time

DB - Done entirely via migrations. Most of the time they are small migrations so I can do it by running the command directly on the instance

Deployments - GitHub actions to build a docker image (there was a prebuilt template). Watchtower then runs on my server to pick up the new image and that's then deployed automatically. There is no rollback system as I've not needed that so far (and 99% of the time you won't).

Backups - DB backup is done via a cron pushing to AWS S3 Glacier. The app code is stored on GitHub already so no need to back that up. Happy to share the script if you like :)

Lead time to change Deployment Frequency Mean time to restore (MTTR) Change Failure Rate.

Though it often is, I do not believe it should be used interchangeably to refer to a suite of tools designed to enable CI/CD and SDLC Lifecycles.

A cron job which saves a copy of the DB to S3.

That’s it.

If you’re focused on anything more with no employees you’re probably wasting your time.

I have no idea what you are building or planning it to be but for my projects I have a single VPS with Hetzner per project running linux and hosting the applications. Deployment is done by a powershell script that copies files from my laptop to the server - that's it.

If I were to think about clusters, cloud instances, load balancers and what not then I would be scared away before having delivered anything of value. Cross that bridge when you get there.