I started a new Big Project at work and just wanted to write out how the planning is going.
Table of Contents
How it started
So this current project, started from a desire at work to save money on a software product work is currently paying a lot of money yearly for. As I have already done a lot of programming work for the company it was brought up that they wanted to have our own in house version of the software made from the ground up to better match how the company does things.
This is a huge project so I was naturally anxious about taking it on but its the mandate and having a new project to work on is an interesting challenge. I ended up in a meeting with my boss, the head of IT and the CEO going over scope and deliverables.
That was about two weeks ago and since I have been working on research and planning work for the most part. I do get to learn a new programming language which is fun!
The Planning
Hardware / Infra
As work wants to eliminate as many ongoing costs as possible the main design from the beginning has been for a self hosted local hardware solution. Because of that and the limitations of the budget the current plan is for two identical server boxes (which are just pc towers as rackable solutions are more then we need and super expensive) along with our current networking hardware and one raspberry pi.
The server boxes will be running in a proxmox cluster along with the pi, with the pi handling quorum negotiation. We are currently looking at some custom build Ryzen 5 7600x pcs as the servers. These have both the case for storing 4 3.5“ drives and the motherboard to support that many or more sata connections along with multiple nvme drives. The ideal solution would be for a RAID 1 ssd storage for proxmox itself. Then the server vms would be stored on another RAID 1 ssd drive. Finally storage would run off of 4 3.5“ drives in RAID Z2. This would allow for failures at every part of the stack discounting the High availability from the mirrored servers.
Software wise the VMS running would be a Nginx server to handle tls/ssl, a storage server to handle postgresql and s3 storage(Garage) on the Z2 raid, and finally the application server. This all would be mirrored on the second server and then there would also be backups to an online s3 compatible storage. We are probably going to go with backblaze b2 because their pricing is good and we won’t be storing huge amounts of data.
I have also been working on sketching out the structure for the SQL tables as they are kind of the bedrock of an application’s design. The future application has to work around how the tables are setup, its not easy to change later on. Luckily we are not expecting to store any customer information in this app so most security requirements regarding that stuff we don’t have to figure out lol.
The Application
For the application itself, I was able to talk my boss out of using PHP or python thankfully! We are going to be going almost fully golang. Golang will handle communicating to the SQL server, watching the http webhooks we need and serving the web front end to the application. I have tested this a bit and was able to get a short piece of golang code that read in a 1.7m line csv into a database in less then a minute which is super fast! Go seems really fast and the standard library has everything we will need as far as I can tell.
For the front end I have been looking at using SSR(server side rendering) htusing HTMX and Alpine.js. This would allow for the majority of the work to be server-side in go which both myself and my boss are a lot more familiar with. I took a look at what a react / reactive front-end would require and I was not comfortable with jumping into that lol. 😄 HTMX seems like a much simpler solution where you make http calls defined in html and then re-render only the returned html at the described location. This is in contrast to react where the server sends json data that the front-end javascript then renders into html.
The intention going into this so far has been for the application to be a simple C.R.U.D. app and as such not a ton of fanciness is needed to begin with. I am sure the scope will eventually balloon outwards but to begin with something doable as described above should be enough.
I am planning on spinning up a super simple one page interface with a table layout displaying some data from the test SQL server and letting the user delete, add and edit items. This should be doable this week and act as a decent first demo of the Create Read Update and Delete paradigm.
Ahh I almost forgot! Logins. The current plan for letting users log into the front end is going to be passkey only. This negates all the issues with passwords and as every user that would need access is already part of the company and using the company 1Password accounts sharing a pre-made passkey and username for login is more then doable for rolling out access.
This approach prevents any chance of unauthorized access as if you don’t have a passkey there is no way to login and phishing a passkey to an internal only tool is not very likely. I am not saying its impossible to get into but this should limit our possible attack surface.
Final Thoughts
I am kind of excited to work on this project. I know it will be less fun later on, especially when it comes time to do UI work (which I hate a lot lol 😄) but for now I am having fun doing research. The current scopes and expectations seem manageable especially since the timeline is months out for a useable demo. We should be getting a crude staging server to work from later next week and once that is here I will be able to get going with a version 0.1.
If you have experience with this kind of thing / this scale of application please let me know what you think! You can email me any feedback I would really appreciate it! [email protected]