Finally, we’ve done it | Our own server

Introduction

Before us I haven’t seen a server setup in Inovus. Well, there have been some in the cloud AWS, Oracle, SSD nodes etc., but not a local server. I did try to set it up first since I had an interest in integrating, fixing, physical interaction etc. I’ve tried out many fields like Web dev, designing, automation, hardware, programming and others but the ones that truly peaked my interest are hardware and devops. Yes, I know web maybe interesting to people who are starting to learn since it provides output to your code easily. But don’t get your hopes up for web dev or programming from this.

Do you know “abstract factory singleton adapter decorator proxy” or “Microfrontend-Based Distributed User Interface Architecture” else you don’t know how to code and web dev is a lie.

Just kidding...

Anyways I should get to the story.

Note:
Everything is simplified here, the process was long and had many errors.

The Start

Well, I can’t say which is the correct starting point, we had couple of starting points. One day I thought of building a server and told about it. At that time, we had no servers but had a pc, I think. We needed a PC and another computer for setting it up as a server. We had no extra to spare and we even asked someone that might have a pc and we were rejected IYKYK. And, we didn’t have any working pc even to be assembled so we dropped the idea and thought to get back to it when we get a pc and carried on.

We got a first Toshiba Satellite Pro C650.

First Server

Yep, it became our server, the laptop which had an Intel Core 2 Duo T6570 @ 2.101GHz with 2GB of RAM(we’ve upgraded it to 4GB); happy. If you aren’t happy to hear it not my problem, we were happy at that time since we got a machine to ourselves to experiment with and no one to ask for. First question was which software is to be used for our server. There was a bunch of options like simply using server OS with docker and manage and setup every app (or service) or we could use a dashboard to manage it. I’ve seen many videos in youtube about homelabbing and private cloud and found out casaos is good for our use. It is not an OS itself first of all. Its just a web UI for managing apps installed using docker in more user-friendly way. For all the advanced geeks who are going to burst out about casaos about it not being advanced or industry standard wait, we are getting there.

Anyways it has inbuilt file manager, terminal emulator, app store etc. Which makes it easy to add apps.

Few days after showed interest and was very much interested at first after seeing casaos. We’ve installed uptimekuma (monitoring), crafty (Minecraft server), portainer (container manager (GUI)), tailscale (VPN) etc. It was fun but I got eventually bored after trying it.

One day was trying to fix janvi (not to be confused with a person, its just an ai lobby assistant). The problem was to run AI locally in RPI4. I suggested to use home assistant (HAOS) as the interface to connect and mange it. There is also a video to do it by networkchuck. But he proposed a direct method and used ai from a provider (it was fast). But I’ve to try it and it looks interesting. There are mainly four ways to install home assistant but I needed to install the method that provided me all the functionality of it specifically Addons not that I will use it every time but because I thought I need it. The container and core method don’t provide me with addons so skipped it. I needed casaos also to be on the same machine but HAOS directly don’t allow anything to be in it, so my option was supervised.

Here is a comparison among the installation methods of HAOS by chatgpt:

But in supervised method there shouldn’t be any other services installed or it won’t work. First, I decided to install the HAOS supervised. It is like installing in the system but in docker container something like that I don’t understand it. After watching a rare video on youtube with somewhat ok setup I’ve completed it by trial-and-error method. Home assistant is finally up🎉. They won’t allow me to install other things before setting up HAOS but now I will setup any software I want since I didn’t use HAOS directly but ran it supervised in Debian. I’ve installed casaos again. I’ve tested the AI feature and setup pi with Wyoming protocol. I’ve also setup n8n and this started my automation journey.

needed only one thing can it run immich specifically the face grouping feature. I’ve tried telling him it’s a dual core machine with 4gigs but he needed that. Ok go ahead I said but it didn’t work. Then we got:

The Upgrade

No, we didn’t receive a whole server. We got a CPU that has Intel Pentium G3220 @ 3.000GHz (dual core) with 4GB RAM. We got 4GB extra RAM from the one we asked before IYKYK. If you know basic math and computer you would know how much ram we have in our servers. So now we had two system with bare minimum standards according to this time. But with our current situation we couldn’t get a good system for a good price. So, we went with it. We’ve set it up as two servers.

told us to setup it up properly for our needs, so we set out to it. asked me again can it run immich.

Still, the answer was no. I’ve setup photoprism a light weight immich alternative even that can only categorize one face and that of . That’s when he realized it won’t work in it.

The Talks That Lead You

We had a get together or community chat or something like last week which include me, and

We had chat about cloud, work, projects and some random things. When it was server time we talked about cool stuff (cool to someone atleat) about different ways to set it up and the apps and services that helps in it. told about the server that we’ve set up in inovus. He learned that we’ve casaos in our server and asked us to set it up directly without UI and use Kubernetes. I’ve been meaning to study Kubernetes and now I’ve a reason to. And, other things we’ve learned in that session motivated me. was also interested in using Kubernetes in our server and asked us to wipe the server.

Behind The Scenes

and I’ve installed the OS in the servers. The first thing to test our system was to run an AI model. Well, it works that’s all I’ve to say about it. and us tested the AI for its knowledge. We asked it who is Tom, Amith, Sane etc., was. Tom was a millionaire with business and a developer. All others were biotechnologist and published many books.

So that’s that.

We needed to access it locally first but the people who have set up local servers or pc for access knows that it can be accessible with the ip or hostname. I can ssh it from my machine, but I can’t ssh the second machine from the first or first from the second. This was frustrating since I can directly access the server with hostname in my machine directly and not each other in the same network. So we researched (I mean used chatgpt) and found out avahi was useful and has mDNS. Whatever and install avahi on both server and it worked. But in local network if I need to access one server from another using hostname I needed to use:

   ssh username@hostname.local

Now for worldwide access we used tailscale and added both servers. But for access through hostname we enabled magic dns in dns setting in the admin panel. Now we can access the devices from anywhere if all connections are ok with

    ssh username@hostname

I’ve to set up Kubernetes in the server not because we need it but because we can. No, I just set it up so I can learn it. Our constraint was resources and multi-node support. We also wanted scaling and other Kubernetes features. The best that I’ve found out was k3s. It is used for low-end machines, IoT etc. And we were not building AWS in inovus so we don’t need others. After setting it up I borrowed (stole) the manifest file for nginx pods from someone I ran it and it worked. I chatgpted (if that’s a word) how to increase or decrease the pods (every container or app is run in a pod).

    kubectl scale deployment test-nginx --replicas=50   # starts 50 instances of same pod

I think the default for k3s is to use master and worker both to run the pods so they share load but the worker takes the majority. But when I check all the deployments only 38 was working even if how high I set the replicas. I looked into the resources but there was no throttling and I didn’t find the solution to it either. But our goal was not to set a million replicas but to understand Kubernetes. asked can’t we run AI models in our own server, yes its possible but you know how it will run in a Pentium machine. If not open llm.inovuslabs.org and find out.

Note:
Admins have to allow you even if you’ve signed up to chat. And, now we’ve connected external AI api so that users can chat with significantly faster models.

Overengineering

“Someone who is very much passionate about tech and know too much tends to overengineer things”

This is a normal tendency in people. Everything that we do can also be done simply. Anyways, and me did the same here. said we can use multiple servers, our local and cloud ones and use hyperspace ai (i think) and run a big model instead of running one small model in a single system. But then I told him it would be network bottleneck and it would be like the smaller model running on a single machine. He knew about it and agreed it. Then I suggested that we have k3s in our server and we should run it locally in a cluster. So, we agreed upon it. I knew I had to use nginx for routing so to check if I’m correct, I checked it in…

It said to me that I should use nginx ingress no matter how I tried to convince it otherwise. So, I decided to use ngnix ingress in k3s. I then used ollama as single docker container and open webui (web UI for ai chat). I thought to set multiple open webui deployments so that it would be like corporate style or at least use k3s for a project. But then asked me not to do it and even simplify by using only the open webui without k3s. At first I thought why and resisted and then I knew that we are dealing with Pentium here so I agreed. I installed open webui and ollama with docker. added a cloudflare tunnel into master and installed nginx. And, the server died. That may happen if the service or app goes overboard. Yah I didn’t set the resource limits of open webui container. After a while the master also died. We concluded that it was power issue even though the server was connected to a ups power supply. We were discouraged.

But after sometime both came back online ( did set up wake on lan). So, we jumped back to and while setting it up both servers went down.

So, we stopped it for the day. The next day I routed the traffic from llm.inovuslabs.org to the worker which has the webui. That day went to inovus and talked about the success and how we can improve it. But don’t dare to run the local model through the UI if you want the answer now. Then we talked about overengineering again.

Conclusion

There is no conclusion its just for the sake of a heading. We are continuing over journey in cloud and other things. We also discussed about setting up multiple pc, android phones as servers. And if you’re thinking can’t we set it up in iphones too…