I've always been curious about some videos that I come across on Facebook and YouTube that look a little strange. Like they were just generated by a robot. Think of a video playing with a little box of a guy reacting to it while their reaction is incredibly strange and appears utterly unrelated. They always feature stuff that either doesn't fit the caption or the content itself is unusual.

I finally understand why. These videos are all automated and performed by scripts. It also appears that "Youtube/Facebook automation" is popular right now.

I first became curious about "Youtube automation" after running across the word. I didn't know that some people were building scripts to scrape stuff from websites (like Facebook and Reddit, but largely Reddit, to be honest), process it, and then produce a video from it to put online.

Imagine this thing running, and having a cron-job to run and post a new video every day. Consider your potential earnings and the consistency of this. You simply need to worry about funding your server and maybe improving how the material is processed while being scraped.

There is a lot to say about this, but I'm here to explain how I used Golang to get several Tiktok videos from a subreddit, combine them, and post a video of the "Top 10 funny/cringiest Tiktok videos" on YouTube without knowing what the video's content was.

TLDR

Social media is really weird. Authentic content is becoming harder to find, and BTW! I managed to code a Golang script to scrape Tiktok videos from a Subreddit, combine them into a single video, and upload it to Youtube all automatically.

So how though?

So for starters, the initial idea is to split the big thing into smaller things and blah blah you know the boring engineering stuff. But I seriously wanted to make it to scale. So it made sense to have 3 Golang packages/modules:

1. Reddit Aggregator:
   • This would be responsible for scraping a subreddit and extracting Tiktok videos out of it.
   • Will save the results into a MongoDB collection - because schema is overrated -
2. Video Processor:
   • This would grab unpublished results from MongoDB collection:
     • Downloads all videos on the server
     • Process each single video
     • Combine all videos into a single video
   • Now obviously this seems a bit too much for a single service, but ... meh
3. Uploader:
   • This will upload the final video to Youtube

TLDR

This is a side project, and I wanted it to be complex enough that Youtube would actually use it. However, spoiler alert: they won't. That is why I divided the code into three Golang packages to run them as microservices serverless functions. Spoiler alert again: they are not.

Reddit Aggregator

The aggregator is straightforward; all it does is use the Reddit API to obtain posts from a subreddit and store them in a MongoDB collection.

https://www.reddit.com/r/<SUBREDDIT>/<POSTS_TYPE>.json?limit=<LIMIT>
https://www.reddit.com/r/linux/hot.json?limit=10

Now not to keep it too simple, every single thing between < > is a variable, that can be provided before the run to make it more robust.

The aggregator will save each post on a DB collection that will look like this:

{
    "_id" : ObjectId("633c37a1740ff26f2433f99c"),
    "hash" : "78a0428f87a9adf7d692e5435466bc5b",
    "title" : "<POST_TITLE>",
    "video" : "<VIDEO_LINK>",
    "created_at" : ISODate("2022-10-04T15:39:45.808+02:00"),
    "updated_at" : ISODate("2022-10-04T15:39:45.808+02:00"),
    "published" : false
}

What are the hash and published fields for you wonder?

Well, hash is a simple MD5 hash of the video link, and on DB this field has a unique index so that we're sure that the aggregator never inserts duplicates. While published is for determining to not return a post/video that was already published to our Youtube channel, this way it will not slip into the process.

Video processor

So, this package will begin by getting a list of unpublished posts from the database, downloading them all to a specific directory, and then starting the processor to process them all.

Downloader

Downloader is using youtube-dl behind the scenes to download every single video, now why not just a simple HTTP GET request and write the file?

• youtube-dl provide great support for downloading videos, from various sources.
• The goal is to be scalable and to be able to even have multiple aggregators ( facebook aggregator, youtube aggregator, Vimeo... etc )
• I already had a wrapper to run a CLI command safely that I did (sniff.go#L118)

Goroutines were required since downloading 10 videos one at a time seemed incredibly slow. I had to run 10 goroutines and wait for them to complete before going on to the next step.

With Goroutines in place, there comes a challenge, more on that will be on here

So to download a single video is just that:

const ydl_command = "youtube-dl %s -q -o %s.%%(ext)s"

func download(url, base, filename string) (string, string, error) {
    dest := path.Join(base, "downloads", filename)
    cmdString := fmt.Sprintf(ydl_command, url, dest)
    args := strings.Split(cmdString, " ")

    cmd := exec.Command(args[0], args[1:]...)
    out, err := helpers.RunCmd(cmd)

    return args[len(args)-1], out, err
}

Processor

ffmpeg is pure evil

The processor is identical to the downloader; it will run through each video and, given that all Tiktok videos are vertical and have a black background on the left and right sides, its sole function will be to create blurry background for the video.

So to achieve that goal, you will have to use ffmpeg which seems more evil than Regex.

Anyway for processing a single video this is the function to do it:

const (
    ffmpeg_quality = "1080k"
    ffmpeg_command = `ffmpeg -i %s -lavfi %s -vb %s -c:v libx264 -crf 20 %s.mp4 -n`
    ffmpeg_filters = `[0:v]scale=ih*16/9:-1,boxblur=luma_radius=min(h\,w)/20:luma_power=1:chroma_radius=min(cw\,ch)/20:chroma_power=1[bg];[bg][0:v]overlay=(W-w)/2:(H-h)/2,crop=h=iw*9/16`
)

func process(video downloader.Video, base string) (string, string, error) {
    dest := path.Join(base, "blurry", video.Post.Hash)
    cmdString := fmt.Sprintf(ffmpeg_command, video.Path, ffmpeg_filters, ffmpeg_quality, dest)
    args := strings.Split(cmdString, " ")

    cmd := exec.Command(args[0], args[1:]...)
    out, err := helpers.RunCmd(cmd)
    if strings.Contains(out, "already exists. Exiting") {
        err = nil
    }

    return args[len(args)-2], out, err
}

For the record, it took me ~1hr to fix an issue running ffmpeg command to learn that I had to change this:

`ffmpeg -i %s -lavfi '%s' -vb %s -c:v libx264 -crf 20 %s.mp4 -n`

to this:

`ffmpeg -i %s -lavfi %s -vb %s -c:v libx264 -crf 20 %s.mp4 -n`

Now of course since the script will be processing 10 videos, this needs to be done using Goroutines to benefit the concurrency. As with the downloader, this will be later explained in the concurrency challenge section

Merger

The merger is the processor's last step, where after adding the blurry background to all of the videos, it will merge them all into a single video. But it does that in steps:

1. Create a text file that has all of the videos paths in this format

file /some/path/blurry-file1.mp4
file /some/path/blurry-file2.mp4

1. Then pass that file to ffmpeg so that it combines all videos

ffmpeg -f concat -safe 0 -i all_videos.txt final.mp4

1. Then remove the temporary all_videos.txt file

func MergeAll(videos []ProcessedVideo, base, output string) error {
    temp, err := createVideosFile(videos, base)
    if err != nil {
        return err
    }

    out, err := runMerge(output, temp)
    if err != nil {
        log.Printf("error running command, output: %q", out)
        return err
    }

    return os.Remove(temp)
}

Uploader

This is the last piece of the puzzle where the final video gets uploaded to youtube, but to avoid having the same title always, there is an option to have a random title deduced from the posts we already have on the database.

Since the goal is to be able to publish a video every single day, you can just get all posts of today and get a single random post, and then make it the video's title.

type Config struct {
    VideoInfoType    string
    VideoTitle       string
    VideoDescription string
    DatabaseURI      string
}

func (c *Config) GetTitle() (string, error) {
    switch c.VideoInfoType {
    case VideoTypeManual:
        return c.VideoTitle, nil
    case VideoTypeRandom:
        db, cancel, err := getDbClient(c)
        if err != nil {
            return "", err
        }
        defer cancel()

        var posts []types.Post
        err = db.FindRandomOfToday(1, &posts)
        if err != nil {
            return "", err
        }
        p := posts[0]
        return fmt.Sprintf("Top 10 best/cringiest Tiktoks today: %s", p.Title), nil
    default:
        return "", fmt.Errorf("unknown video type %v", c.VideoInfoType)
    }
}

Besides that script of course will need client_credentials and API token to be able to use youtube API to upload the video.

upload := &youtube.Video{
  Snippet: &youtube.VideoSnippet{
    Title:       title,
    Description: description,
  },
  Status: &youtube.VideoStatus{
    PrivacyStatus: c.PrivacyStatus,
  },
}

call := service.Videos.Insert([]string{"snippet", "status"}, upload)

file, err := os.Open(c.OutputFile)
check(err)
defer file.Close()

r, err := call.Media(file).Do()
check(err)

Concurrency challenge

The main challenge for me was to be able to just concurrently run tasks I want to run, of which i have 2:

• Downloading videos
• Processing videos

These are the most expensive tasks I have in terms of time because if they run one by one it will take so much time obviously.

So I wanted to have a way of running a tasks pool, where I pass the number of needed threads, and the pool will just split the tasks across these threads. So I ended up implementing this:

type ThreadElement struct {
    Element interface{}
}

func Threadify(numOfThreads int, elements []ThreadElement, f func(args ...interface{})) {
    length := len(elements)
    each := length / numOfThreads
    acc := length - (numOfThreads * each)

    var wg sync.WaitGroup

    wg.Add(numOfThreads)

    start := 0

    for i := 0; i < numOfThreads; i++ {
        running := each

        if acc > 0 {
            running++
            acc--
        }

        go func(start, i int) {
            for j := 0; j < running; j++ {
                e := elements[start]
                f(e.Element)
                start++
            }

            wg.Done()
        }(start, i)

        start += running
    }

    wg.Wait()
}

Pretty basic right? and for sure there is a better way to do it, but hey it actually worked, and now I can download videos by calling it like this:

var (
  videos   []Video
  elements []helpers.ThreadElement
)

for _, p := range posts {
  elements = append(elements, ThreadElement{
    Element: p,
  })
}

Threadify(3, elements, func(args ...interface{}) {
  p := args[0].(types.Post)
  log.Printf("calling download %q", p.Hash)
  downloadedPath, out, err := download(p.Video, base, p.Hash)
  if err != nil {
    log.Fatalf("error downloading video: %q of this post: %q::: %q\ncommand output: %s", p.Video, p.Hash, err.Error(), out)
  }

  log.Printf("Downloaded video: %q with hash %q on %q\n", p.Title, p.Hash, downloadedPath)
  videos = append(videos, Video{
    Post: p,
    Path: strings.ReplaceAll(downloadedPath, "%(ext)s", "mp4"),
  })
})

fmt.Println("Videos downloaded", videos)

I loved how generic this function is, and that is I just can use it for anything I want to concurrently run, and the way you can pass elements to it, by just kind of mapping it into ThreadElement so that it understands what it will later pass to the callback function.

But still, that looked hard to maintain, and I asked myself there has to be a better way to do this, just out of curiosity i googled and came across ants which seemed exactly right for this kind of functionality I wanted, so I converted the same function to use ants, and it became this:

func Threadify(numOfThreads int, elements []ThreadElement, f func(args ...interface{})) error {
    var wg sync.WaitGroup

    p, err := ants.NewPoolWithFunc(numOfThreads, func(e interface{}) {
        f(e)
        wg.Done()
    })
    if err != nil {
        return err
    }

    defer p.Release()

    for _, e := range elements {
        wg.Add(1)
        err = p.Invoke(e.Element)
        if err != nil {
            return err
        }
    }

    wg.Wait()
    return nil
}

Now it even looks cleaner, and I just don't care anymore about how it works as long as it works, and the most beautiful thing is that the function signature is still the same, which means I could still use it in the same way I did and have it generic just as I wanted initially.

Lessons learned

Finally now after the project is complete, here are the top lessons I learned from:

• I shouldn't be surprised to know about "Youtube automation", literally everything is possible nowadays.
• Golang is just super easy, Goroutines are easy when you just think of them logically, but also they're easy to produce bugs.
• Don't use any dotenv package in Golang, IDK why I did!! it just makes no sense at least in this kind of project.

Finally

I got the idea from this Reddit post huge kudos to the OP, he made it simply in bash, but I just had to be a nerd to do it in Golang :/

Here is the repository link: https://mrg.sh/github/automato

There is huge room for improvement indeed, but I'm happy with the results, mission is achieved now to the new side project.

Well in this article i'm not going to go deep into what aggregation query is, since i think MongoDB Docs explained this perfectly, but instead i will go through a kind of seeming-complex aggregation query, which is you probably going to face if you're currently with MongoDB and i will use this example here to just show you how aggregation pipelines are just logically easy, and it's always depends on how you think while building it

The example

Well for the sake of our example, let's consider that you have this silly products collection which has these 2 documents:

{
  "_id": {
    "$oid": "55c30ff62cfa09af198b465a"
  },
  "name": "Awesome Tshirt",
  "currency": "usd",
  "one_size": "xl",
  "variants": [
    {
      "type": "color",
      "base_sku": 132145,
      "items": [
        {
          "color": "Grey Melange",
          "price": 80,
          "sku": 1243252369
        },
        {
          "color": "Bottle Green",
          "price": 90,
          "sku": 1243252368
        },
        {
          "color": "Deep Charcoal Grey",
          "price": 80,
          "sku": 1243252376
        },
        {
          "color": "White",
          "price": 80,
          "sku": 1243252363
        },
        {
          "color": "Black",
          "price": 80,
          "sku": 1243252362
        }
      ]
    }
  ]
},
{
  "_id": {
    "$oid": "55c30ff62cfa09af198b465c"
  },
  "name": "Hacker Tshirt",
  "currency": "usd",
  "one_size": false,
  "variants": [
    {
      "type": "color",
      "base_sku": 132155,
      "items": [
        {
          "color": "Black",
          "price": 100,
          "sku": 87987963
        }
      ]
    },
    {
      "type": "size",
      "base_sku": 342434,
      "items": [
        {
          "size": "sm",
          "price": 100,
          "sku": 97896796
        },
        {
          "size": "xl",
          "price": 100,
          "sku": 43534534
        },
        {
          "size": "xxl",
          "price": 100,
          "sku": 76576532
        }
      ]
    }
  ]
}

and now our goal is to get the price of all grey T-shirts, so since as you can see that each product has it's price vary based on the variant itself, so we have to get both from variants items item object. Yes in this collection its so silly that somehow you have to choose of either having black T-shirt or XXl tshirt, but not both :D

Stage #1

The very first step when you create a pipeline is to always match your query, this to narrow down querying the whole collection to just a limited number of documents that match your criteria

You always want to narrow down the number of documents you're searching within at the very beginning of the aggregation pipeline, this will lead to faster queries

So let's do this, first we want to use only the documents that has color variant, and it has also grey color inside it's variant items. So this is how we're translating this:

    {
        '$match': {
            'variants': {
                '$elemMatch': {
                    'type': 'color', 
                    'items': {
                        '$elemMatch': {
                            'color': /grey/i
                        }
                    }
                }
            }
        }
    }

$elemMatch operator matches documents that contain an array field with at least one element that matches all the specified query criteria.

We're using $elemMatch here to find a variant of type color first, and then we're using it again to find an element of color that contains grey and its case insensitive - notice the regex /grey/i -

So this was our first step, notice that only 1 item will be returned in this case that has ID 55c30ff62cfa09af198b465a, since it is the only one with variants of type color that has grey color

Document 55c30ff62cfa09af198b465c it has variants of type color, but it has only black color

Stage #2

Now we still need to query a nested object that is inside variants (array) and also inside items (array), so its more like this variants -> items -> {color, price}

This seems complicated, and since we're dealing with array of objects here, what could make it easier? .. to just deal with this array as an object instead, so we can just leverage the dot notation syntax in Mongo

Well we can do that by just unwinding the array! simple as that, just flatten this array into objects by using $unwind

$unwind Deconstructs an array field from the input documents to output a document for each element

$unwind is so simple that the only required parameter to pass is path which is the path of the array you want to flatten

  {
    '$unwind': {
      'path': '$variants'
    }
  }

Notice the dollar sign before variants, we have to prefix the field name with it so Mongo can interpret it, it just tells Mongo to inject the actual value of variants

Now by then we will have also 1 document still, since variants array has only 1 element

Notice that variants is now object instead of array

Stage #3

Now what? we still need to query variants.items elements which is the same case as variants before we unwind it. So i guess we will have to flatten variants.items too, so next stage will be

  {
    '$unwind': {
      'path': '$variants.items'
    }
  }

Notice now that we can easily access items with dot notation, since variants is an object and not an array anymore, now these are the the new documents returned after this stage

items is an object now with 1 document per items element which is exactly what we need, but did you notice something strange?

Stage #4

Now we have documents with items.color value that does not contains grey we have Black, White, and Bottle Green as well, why is that?

Well that would be because our first $match stage was only getting the documents that have items with grey color, having this does not necessary means that it will magically just filter other colors from items, this our job to do now

So now we will need to get only the documents that has variants.items.color with greyish color, looks like another $match query, right?

  {
    '$match': {
    'variants.type': 'color', 
    'variants.items.color': /grey/i
  }

It is so simple now, here we're just saying, out of the result of stage 3 we just want the documents which has variants of type color and any item that has grey with case insensitive, that will return us these documents:

Stage #5

Now we have the documents, but we don't really care about all of these properties right? We only care about the color and the price

So since we're having a couple levels of nesting here variants -> items and we only care about price and color properties, then we need to focus/project these properties only and ignore any other stuff we don't need

$project Passes along the documents with the requested fields to the next stage in the pipeline.

$project is so simple also, you'll probably just need to know that:

• It takes field name as a key and 1 or 0 as value (1 to show, 0 to hide)
• _id is shown by default, unless you specify to hide it
• You have to specify at least 1 output field

So our stage implementation will be simple as

  {
    '_id': 0,
    'color': '$variants.items.color',
    'price': '$variants.items.price'
  }

Notice that we passed '_id': 0 because we don't really care about the document ID - at least not in this example, normally you'll need it though - so we just hid it

So now this will be the final result

The aggregation query

db.getCollection('products').aggregate([
  {
    '$match': {
      'variants': {
        '$elemMatch': {
          'type': 'color', 
          'items': {
            '$elemMatch': {
              'color': new RegExp('grey', 'i')
            }
          }
        }
      }
    }
  },
  {
    '$unwind': {
      'path': '$variants'
    }
  },
  {
    '$unwind': {
      'path': '$variants.items'
    }
  },
  {
    '$match': {
      'variants.type': 'color', 
      'variants.items.color': new RegExp('grey', 'i')
    }
  },
  {
    $project:  {
      '_id': 0,
      'color': '$variants.items.color',
      'price': '$variants.items.price'
    }
  }
])

Final words

As you can see it's pretty straight forward, and it's quite easy and seems quite logical too, building aggregation pipelines stages is just like actually talking to rubber duck while coding, instead this time you're just talking to Mongo shell

Meetings are indeed time-consuming but only if they were with the wrong people, or if they were just useless, thank god that is not my case.

The day started with an ~1hr long meeting, discussing some challenges that are facing us related to our upcoming features, and oh these challenges are indeed complex and a brand new type for me probably I can write about them whenever I get the chance.

Having challenges are quite good training and experience booster, but do you like having constant challenges? Do I like that?

Then my day went on to exploring some emails, checking snd reviewing some PRs, till I was interrupted with some issue, that I kinda didn’t have the time to check it and it was quite important. I checked the card, did some quick investigation (~1hr) and hey, it was not replicating! Smooth eh? Typical bug, the support team will say it exists but you can’t replicate it, no surprise there.

I checked again, went deep this time by checking some corner cases, read the code again tried to be at least sure before pulling out my big gun, but it was in vain. So I had to do it... I called one of our support guys, the one reported it, we hoped in a meeting, that lasted for ~1hr and we come up with this besides exchanging some laughs

• Issue is not an issue, and it was something wrong with the client configuration.
• There is another different type of issue happening (nice trade eh?)

so that was it, now I had to eat something and get some fresh air, but oh wait... it’s hot outside, so just eat something while watching some YouTube videos.

I finished and went back to a task that I planned on doing, we have an issue and we need to check for a memory leak. So ideas were thrown at my first meeting to try flame graphs, and heap dumps, so here I go trying.

I tried to use perf command to monitor a Nodejs running server, got nothing, just weird ASCII characters on the output, tried parsing them, but it just didn’t look right, so now I just have to try heap… oh message from DevOps what they want? They’re asking about something related to the database, I have to give them a query now, and it’s a kinda complex one. Okay, let’s first start the Vagrant machine, hop into Robo3t to access my local database, and start building a 5 stages aggregation pipeline to return a field value from a nested array of objects.

Now where we’re we? Oh, heap dump, yeah let’s try to use heapdump package to generate a heap dump for my running Nodejs server, as usual pasting some code from the documentation, tweaking it a bit, and now .. a message from scrum master? I need to review these 2 PRs ASAP that belong to another team but they’re touching our team functionality so we can do the release. Ding it, I didn’t notice these PRs so I gotta check them.

Now, what is this variable for? Let me open the repository, look for it, I get it now.. wait.. no I let’s understand the code, oh what’s that? Let me call the developer he knows better, I message him we hop into a meeting and he explains the issue perfectly and we find that this code has a problem and it wouldn’t properly work with a specific corner case. Okay great now I just jest changes on the PR, type a comment summarizing our meeting discussion, and back to where I was.

Now what? Where? Oh, heap dump I left the server running, so I just have to hit a specific endpoint and the heap snapshot will be written to the disk for me to analyze it, let’s do that. Endpoint called, no errors, but the server is restarting … hmmm is this expected? Does this package somehow affect the runtime? IDK but hey doesn’t matter now, let me just see the heap dump.

cat ./heaptest.heapsnapshot

It’s empty! Hmm weird, let me do it again. After 4 times repeating the file is still empty now I’m pissed off it should work, right? Okay, let just google that, I found tutorials, articles, and StackOverflow questions but they’re all the same, maybe I’m missing something.

In the meanwhile, I’m replying to messages and answering questions, but what got me mentally blocked is another PR, but this time a comment I made a while ago and the developer is asking about it. The comment was about some clean code stuff and how it’s better not to pollute a function with custom conditions and instead think of another way of doing it. We discussed and he suggested a perfect approach, so now it was time to continue.

But I lost it already, I’m not as focused as before, so I just did a couple of other tries, and called it day at 6 pm already.

Now what?

This was indeed a busy day, but I’m kinda used to it by now, but Is this okay? I do not know.

This is definitely not bad to make you practice multitasking and make you available at all times, but it comes with a cost on your concentration. I’m not here to propose a solution though, I’m just here to share and for the hope that I will find the solution by just writing this kind of a daily routine.

Do I like being busy? Yeah, it makes my mind spin and thinks, and mostly shifting from thinking on a certain problem to another one, helps me find a solution for the first problem. I got a weird mind IKR?

So wargames bandit was something that John Hammond actually recommended to get familiar with Linux for anyone that wanted to start hacking and become a hacker. I thought oh what is this wargames and started googling

I found it, wargames bandit and i read it and thought, well this is going to be easy, i know Linux i been working on it for about 6 years now, managing and deploying servers and use it everyday since then for my daily life, i even play games on it for god sake

What is wargames bandit?

Well this game is simple, its consists of levels, you start at level 0 and make your way through by finishing the level, basically finishing a level means that you get the password for the next level, and so on

Bandit is made for absolute wargames beginners and it aims to get players familiar with Linux and the wargames style

Why wargames?

Well first because it's fun, and i mean it, it is actually fun, and no matter how you're using Linux, mostly you'll learn a lot by playing it

You will know more about some commands that you might be using everyday like grep, ssh, cat, or find. And even new commands (at least for me) like strings, uniq, bzip2, s_client, and xxd

You will write a simple bash scripts, which is super fun especially if you're like me, someone that loves to automate some tedious work by writing scripts

You will understand some Unix concepts and even interact with text editors like vi for example, and even know how to deal with different shells

Also you will interact with git and running some git commands so you can look for some secret credentials in git branches, commits, or tags

And of course you will know some hacking concepts, and how to protect your server by avoiding some silly mistakes, that we all may be doing

As a developer you'll find this also important, especially if you're interacting with servers on any environment, you'll learn essential security concepts and it will open your eyes to a whole new world

The gist

You should give a try , i started it on a weekend and i found myself finishing the 33 levels in 1 day that's how entertaining this is, and i will probably check other wargame as well maybe natas? who knows!

Why not socket.io?

Well first it has all you might need, and i mean all like all of it, which is kinda a lot since you end up using a lot of stuff you don't actually need, while on the other hand ws provides you with the base functionality and the ability to customize it as per needs

Besides i remember times when socket.io repository was abandoned, and they had a lot of issues, and they it was kinda deprecated in favor of engine.io and made a great role also in choosing ws over it.

HTTP server setup

Now let's setup our express HTTP server a typical implementation with nothing fancy

const express = require('express')
const app = express()
const PORT = 3000 || process.env.PORT

app.get('/', (_, res) => {
  res.send('Welcome to main page')
})

app.listen(PORT, () => {
  console.log(`Server is working on http://localhost:${PORT}`)
})

Nothing special here, we defined our index endpoint which is going to return Welcome to main page if we went to http://localhost:3000/

So to setup our websocket server we need to import ws package and then tweak this a little bit

const express = require('express')
const WsServer = require('ws')
const { createServer } = require('http')

const app = express()
const server = createServer(app)

function initWs() {
  const options = {
    noServer: true
  }

  return new WsServer.Server(options)
}

function initHttpServer(port) {
  app.set('view engine', 'ejs')

  app.get('/', (_, res) => {
    res.render('index')
  })

  server.listen(port, () => {
    console.log(`Server is working on http://localhost:${port}`)
  })

  return app
}

function initWebSocketServer(port = 3000) {
  initHttpServer(port)
  const wss = initWs()

  server.on('upgrade', async(req, socket, head) => {
    try {
      wss.handleUpgrade(req, socket, head, (ws) => {
        // Do something before firing the connected event

        wss.emit('connection', ws, req)
      })
    } catch(err) {
      // Socket uprade failed
      // Close socket and clean
      console.log('Socket upgrade failed', err)
      socket.destroy()
    }
  })

  return wss
}

const wss = initWebSocketServer()

wss.on('connection', (ws) => {
  ws.on('message', (data) => {
    ws.send(data)
  })
})

Now we splitted the code into 3 functions, 1 to handle preparing our websocket headless server by just passing noServer: true to ws server constructor, beside a function to setup our http server and notice that we added new route

  app.get('/', (_, res) => {
    res.render('index')
  })

This to handle our client that is going to be connected to our server and thats where our client code will live on index view, where it exists under views directory here

websocket-server/
├─ views/
│  ├─ index.ejs
├─ server.js

Client side setup

Now on the client side the code is pretty simple

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>WS Test</title>
  </head>
  <body class="container">
    <script>
      let keepAliveId = null
      function connect({ url, keepAlive, keepAlivePeriod }) {
        const ws = new WebSocket(url)

        if (keepAlive) {
          clearInterval(keepAliveId)
          keepAliveId = setInterval(() => {
            ws.send('ping')
          }, keepAlivePeriod || 5000)
        }

        ws.onopen = () => {
          console.log('Websocket is open')
        }

        ws.onclose = () => {
          console.log('Websocket closed')
        }

        ws.onerror = (err) => {
          console.log('Websocket error', err)
        }

        ws.onmessage = (message) => {
          console.log('Websocket data', message.data)
        }
      }

      connect({
        url: 'ws://localhost:3000',
        keepAlive: true,
        keepAlivePeriod: 5000
      })
    </script>
  </body>
</html>

We have our connect function here that expects the following options { url, keepAlive, keepAlivePeriod }

keepAlive property is to keep the websocket connection alive by sending message after a period of milliseconds which what in the keepAlivePeriod for

The way we're keeping the websocket alive is in this snippet

        if (keepAlive) {
          clearInterval(keepAliveId)
          keepAliveId = setInterval(() => {
            ws.send('ping')
          }, keepAlivePeriod || 5000)
        }

So every period of time we're sending this ping message to server and thus server is going to echo back what we're sending and this is how simply the socket connection is alive

That’s basically how we created a simple echo server using just ws package with even the ability to keep the socket connection alive. Later on I will explain how to trigger events from client side and even subscribe to these events so whenever server is emitting an event it will be received by the client

The compose file

version: "3.2"
services:
  mongo:
    container_name: mg_mongo
    build:
      context: ./config/mongo/
    restart: always
    networks:
      default:
        aliases:
          - mrgeek_mongo
    ports:
      - "27017:27017"
    volumes:
      # Mongo config
      - ./config/mongo/mongod.conf:/etc/mongod.conf
      # Mongo data to be persistent between reboots
      - mongo-data:/data/db


  rabbitmq:
    container_name: mg_rabbitmq
    build:
      context: ./config/rabbitmq/
    restart: always
    networks:
      default:
        aliases:
          - mrgeek_rabbitmq
    ports:
      - "5672:5672"
      - "15672:15672"
    volumes:
      # RabbitMQ configuration
      - ./config/rabbitmq/:/etc/rabbitmq/

  elasticsearch:
    container_name: mg_elasticsearch
    build:
      context: ./config/elasticsearch/
    restart: unless-stopped
    networks:
      default:
        aliases:
          - mrgeek_elasticsearch
    ports:
      - "9200:9200"
    volumes:
      # Elasticsearch data
      - elastic-data:/var/lib/elasticsearch
      # Elasticsearch configuration files
      - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./config/elasticsearch/jvm.options:/usr/share/elasticsearch/config/jvm.options

  kibana:
    container_name: mg_kibana
    build:
      context: ./config/kibana/
    restart: unless-stopped
    depends_on:
      - elasticsearch
    environment:
      ELASTICSEARCH_URL: http://mrgeek_elasticsearch:9200
      ELASTICSEARCH_HOSTS: http://mrgeek_elasticsearch:9200
    ports:
      - "5601:5601"
    volumes:
      # Kibana configuration files
      - ./config/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
      - ./config/kibana/node.options:/usr/share/kibana/config/node.options
      - ./config/nginx/ssl/:/usr/share/kibana/config/ssl/

  main:
    container_name: mg_main
    build:
      # define parent directory as context so we can get things from other
      # repositories
      context: ./../
      dockerfile: ./mg_docker/Dockerfile
    depends_on:
      - mongo
      - rabbitmq
      - elasticsearch
    ports:
      - "443:443"
      - "80:80" 
      - "5222:5222" 
      - "22:22"
    extra_hosts:
      # extra hosts
      - "mrgeek_api:127.0.0.1"
      - "mrgeek_xmpp:127.0.0.1"
      - "mrgeek_file_upload:127.0.0.1"
      - "mrgeek_reporting:127.0.0.1"
      - "host.docker.internal:host-gateway"
    volumes:
      # Bashrc
      - ./config/.bashrc:/root/.bashrc

      # dnsmasq.conf
      - ./config/dnsmasq/dnsmasq.conf:/etc/dnsmasq.conf

      # Nginx configuration from our nginx-edge repo
      - ./config/nginx/ssl/:/etc/nginx/ssl/
      - ./../mg-nginx/src/conf.d/:/etc/nginx/conf.d/

      # Kibana configuration
      - ./config/kibana/:/etc/kibana/

      # Ejabberd
      - ./config/ejabberd/conf/:/opt/ejabberd/conf/
      - ejabberd-data:/opt/ejabberd/database/
      - ./config/ejabberd/ssl/:/opt/ejabberd/ssl/

      # PHP fpm conf
      - ./config/api/php-fpm.conf:/etc/php/fpm/php-fpm.conf
      - ./config/api/www.conf:/etc/php/fpm/pool.d/www.conf
      - ./config/api/opcache.ini:/etc/php/fpm/conf.d/05-opcache.ini
      - ./config/api/php.ini:/etc/php/fpm/php.ini
      - ./config/api/php.ini:/etc/php/cli/php.ini

      #
      # Main services
      #

      # api
      - ./../mg_api/:/var/www
      - ./../mg_api/composer.lock:/var/www/composer.lock

      # mg_xmpp_node
      - ./../mg_xmpp/:/var/app/mg_xmpp/

      # mg_reporting
      - ./../mg_reporting/:/var/app/mg_reporting/

      # mg_file_upload
      - ./../mg_file_upload/:/var/app/mg_file_upload/

volumes:
  mongo-data:
  elastic-data:
  ejabberd-data:

The first 4 services have common attributes as you can see all of their config we get from a ./config directory were it has sub-directories for each service it has its own configuration files plus its own Dockerfile as you can see here:

mg_docker/
├─ config/
│  ├─ api/
│  │  ├─ php-fpm.conf
│  │  ├─ www.conf
│  │  ├─ opcache.ini
│  │  ├─ php.ini
│  ├─ dnsmasq/
│  │  ├─ dnsmasq.conf
│  ├─ elasticsearch/
│  │  ├─ elasticsearch.yml
│  │  ├─ jvm.options
│  │  ├─ Dockerfile
│  ├─ kibana/
│  │  ├─ Dockerfile
│  │  ├─ kibana.yml
│  │  ├─ node.options
│  ├─ ejabberd/
│  │  ├─ conf/
│  │  │  ├─ ejabberd.yml
│  │  │  ├─ ejabberdctl.cfg
│  ├─ rabbitmq/
│  │  ├─ Dockerfile
│  │  ├─ enabled_plugins
│  ├─ mongo/
│  │  ├─ Dockerfile
│  │  ├─ mongod.conf
├─ Dockerfile
├─ docker-compose.yml
├─ docker-entrypoint.sh
mg_api/
mg_xmpp/
mg_reporting/
mg_file_upload/

Important thing to note here is the networks aliases for each service because that's how the services are going to communicate between each other, a good example for this is elasticsearch container aliases:

    networks:
      default:
        aliases:
          - mrgeek_elasticsearch

and we're using this alias in kibana container passing it to it's configuration as environment variables

    environment:
      ELASTICSEARCH_URL: http://mrgeek_elasticsearch:9200
      ELASTICSEARCH_HOSTS: http://mrgeek_elasticsearch:9200

Now for the main service it has more configuration as you can see, but good thing to notice is that the context is set with context: ./../ and since the context is changed we're referring to the docker file with this dockerfile: ./mg_docker/Dockerfile

Context is changed so we can refer to other services which exist on the parent directory of this one ../

And we're also setting the extra hosts so that services can communicate to each other inside the same container

    extra_hosts:
      # extra hosts
      - "mrgeek_api:127.0.0.1"
      - "mrgeek_xmpp:127.0.0.1"
      - "mrgeek_file_upload:127.0.0.1"
      - "mrgeek_reporting:127.0.0.1"
      - "host.docker.internal:host-gateway"

Also we're setting the volumes as well where you can see we're mounting other services

dnsmasq is used to act as a DHCP server inside the container so we can reference these extra hosts inside the container

The docker files

Docker files for our services are really simple and they actually might be redundant and we can even use docker compose directly for this, but for better and clear control we went with docker files

The one hell of a Dockerfile that we rely on is the main docker file for our main service which has:

• Nginx
• PHP & PHP-FPM
• Ejabberd
• NodeJs

It is a big one i know, and later on this one will be splitted into multiple services of course, so here is it:

FROM debian:10

###############################################################################
# Install basic software
#
RUN set -x \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
        systemd \
        sudo \
        build-essential \
        nano \
        openssh-server \
        apt-transport-https \
        ca-certificates \
        gnupg \
        curl \
        software-properties-common \
        locales \
        python-pip \
        python-setuptools \
        python \
        unzip

###############################################################################
# Install nginx
#
RUN curl -fsSL https://nginx.org/keys/nginx_signing.key | apt-key add -

RUN add-apt-repository 'deb https://nginx.org/packages/mainline/debian/ buster nginx'

RUN set -x \
    && apt-get update \
    && apt-get install -y nginx \
    && rm -rf /var/lib/apt/lists/*

###############################################################################
# Install ejabberd
#
ENV EJABBERD_HOME=/opt/ejabberd
ENV EJABBERD_CTL=/opt/ejabberd/bin/ejabberdctl
ENV NGINX_INTERNAL_SERVICE_HOST=localhost

COPY ./mg_ejabberd/ejabberd.deb /root/ejabberd.deb
RUN dpkg -i /root/ejabberd.deb

RUN mkdir -p /var/log/ejabberd
RUN touch /var/log/ejabberd/extauth.log
RUN chown -R ejabberd:ejabberd /var/log/ejabberd
RUN chown -R ejabberd:ejabberd ${EJABBERD_HOME}/

###############################################################################
# Install dnsmasq for nginx resolver
#
RUN apt-get update
RUN apt-get install -y dnsmasq dnsutils net-tools


###############################################################################
# Install php, php-fpm and composer
#
RUN apt-get update
RUN apt-get install -y \
      php-cli \
      php-fpm \
      php-xml \
      php-bcmath \
      php-mbstring \
      php-gd \
      php-curl \
      php-mongodb \
      php-pear \
      php-dev \
      php-xdebug \
      php-apcu

# Setup xdebug
COPY ./mg_docker/config/api/xdebug.ini /etc/php/mods-available/xdebug.ini

# Install mongo extenstion
RUN pecl install mongodb

RUN curl -sS https://getcomposer.org/installer -o /root/composer-setup.php
RUN php /root/composer-setup.php --install-dir=/usr/local/bin --filename=composer

RUN echo "extension=mongodb.so" > /etc/php/cli/conf.d/20-mongodb.ini
RUN echo "extension=mongodb.so" >/etc/php/fpm/conf.d/20-mongodb.ini

###############################################################################
# Some cleanup to reduce space
#
RUN apt-get clean && apt-get autoremove
RUN rm -rf /root/ejabberd.deb
RUN rm -rf /root/composer-setup.php

###############################################################################
# Nginx main configuration
#
COPY ./mg-nginx/src/nginx.conf /etc/nginx/nginx.conf

ARG USER_ID=1000
ARG GROUP_ID=1000

# Remove www-data user
RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi

# Generate a new www-data user with IDs 1000
# Will be used by API to create cache & logs
RUN groupadd -g ${GROUP_ID} www-data
RUN useradd -l -u ${USER_ID} -g www-data www-data
RUN usermod -G sudo www-data
RUN install -d -m 0755 -o www-data -g www-data /home/www-data /var/run/php /var/run/nginx
RUN chown --changes --no-dereference --recursive \
        ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /var/lib/php/sessions \
        /etc/nginx /var/run/nginx /var/run/php \
        /var/cache/nginx \
        /var/log/nginx \
        /var/run/php \
        /var/run/nginx \
        /etc/php/
RUN sed  -i 's/^\%sudo.*/%sudo   ALL=(ALL) NOPASSWD:ALL/g' /etc/sudoers

###############################################################################
# Entrypoint
#
COPY ./mg_docker/docker-entrypoint.sh /tmp/docker-entrypoint.sh
RUN chmod +x /tmp/docker-entrypoint.sh
USER www-data

###############################################################################
# Install nvm and node v14.15.4 and npm v6.14.10
#
ENV NODE_VERSION=14.15.4
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
ENV NVM_DIR=/home/www-data/.nvm
RUN . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION}
RUN . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION}
ENV PATH="/home/www-data/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"
RUN node --version
RUN npm --version


###############################################################################
# Install pm2
#
RUN npm install -g pm2

ENTRYPOINT ["./tmp/docker-entrypoint.sh"]

###############################################################################
# Expose needed ports
#
EXPOSE 433 80 5222 22

STOPSIGNAL SIGQUIT

CMD ["nginx"]

It's pretty straight forward docker file the only thing to note is this part

ARG USER_ID=1000
ARG GROUP_ID=1000

# Remove www-data user
RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi

# Generate a new www-data user with IDs 1000
# Will be used by API to create cache & logs
RUN groupadd -g ${GROUP_ID} www-data
RUN useradd -l -u ${USER_ID} -g www-data www-data
RUN usermod -G sudo www-data
RUN install -d -m 0755 -o www-data -g www-data /home/www-data /var/run/php /var/run/nginx
RUN chown --changes --no-dereference --recursive \
        ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /var/lib/php/sessions \
        /etc/nginx /var/run/nginx /var/run/php \
        /var/cache/nginx \
        /var/log/nginx \
        /var/run/php \
        /var/run/nginx \
        /etc/php/
RUN sed  -i 's/^\%sudo.*/%sudo   ALL=(ALL) NOPASSWD:ALL/g' /etc/sudoers

Here we're creating new www-data user with the same UID and GID as the host user which are both 1000 then we're changing the owner of most of services' directories to the new user, so whenever for example PHP is generating logs, which are mounted on the machine, host user can still control them without using sudo

This is also used to run our docker entrypoint as you can see

COPY ./mg_docker/docker-entrypoint.sh /tmp/docker-entrypoint.sh
RUN chmod +x /tmp/docker-entrypoint.sh
ENTRYPOINT ["./tmp/docker-entrypoint.sh"]
`

In which we're running all the services of this container.

This way it was easier to mount files, and even the mounted volumes will have the same exact user as on the host machine. So we will never have to use sudo to remove a directory for example or change a file.

Final thoughts

It was quite a journey, which is still ongoing BTW but to reach this point where at least the software is up & running and is development friendly is quite amazing

The directory structure we have plays an important role to have this working, where each service is a repository on its own, and exists on the parent directory of this docker project, that's why it was easier to reuse them in docker containers

DevOps indeed helped and fixed some hard issues, and i learned a lot by just touching this bare topic, i was exposed to so many technologies so deep more than i would ever dreamed.

So the real work started, i had to read more about k8s since now i'm not gonna simply use it like i did before by just running some ambiguous commands that i barely know, but i will now need to setup a whole environment using k8s and booooi that was shocking!

The research

I knew something about k8s but know i know even more, kubernetes has yaml files configurations where you just setup your environment into services, applications, secrets and more. This was quite interesting for me because now i need to do the exact same for our tiny local development.

Minikube was the first thing i did install, i already had virtualbox to run vagrant so this dependency is checked already. So i started playing a bit first of course i looked at some examples, explore some already written code on github and started to connect some dots, beside of course actually looking at our kubernetes code that we already using on production, our DevOps have written some good documentation for that and this was actually my starting point.

It is worth to mention that one of the main resources i had was kubernetes documentation , they have written an excellent docs that helped me getting up and running and even understand some concepts, especially for learning the basics.

At this stage i was kinda ready to actually slowly build our environment, but first you need to know about the tech stack itself.

The tech stack

Well we're obviously have a microservices architecture and that's why we're using kubernetes on production to orchestrate these services for us, but what is the actual tech stack we're using?

It's a SaaS web application so our tech stack consists of:

• Nginx
• NodeJs
• AngularJs
• PHP
• Ejabberd
• RabbitMQ
• Elasticsearch
• Kibana
• MongoDB
• Python

Now i need to write some yaml configuration files for kubernetes to have our local development environment ready, but wait .. we already have all the yaml files written for production so i can re-use these but tweak a bit for development environment.

Tweaking the configurations

It was not easy as it seems though, turns out kubernetes is designed to orchestrate a production ready pods not some development applications. But what's the difference though?

Well in development environment what developers really care about is the ability to immediately see their code changes by a simple page refresh if its a web app, or a just by running the same command again if it is a CLI app. But for production app you'll have to rebuild the whole pod, and then maybe clear some cache and then you'll be able to see these changes, and that is of course time consuming.

So hot-reloading is required, and sure to do this in docker you must have some volumes mounted to the host machine, so that whenever you change a file on your host machine, the file is also going to be changed on the container itself, and hence on the pod.

So now you understand the main issue, that given this main requirement k8s is going to be a pain to configure since it's by design not ready for this.

Let's try anyway

So who cares, we knew that it won't be easy, so let's try anyway! and let me walk you through the whole process.

I started writing some yaml files of course that will define how i wanted our apps, services and secrets to communicate, here is an example of one of our NodeJs services deployment file that i wrote.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: super-doper-nodejs-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: super-doper-nodejs-service
      group: super-doper
  template:
    metadata:
      labels:
        app: super-doper-nodejs-service
        group: super-doper
    spec:
      volumes:
        - hostPath:
            path: /host/path/super-doper/
          name: super-doper-persistent-storage
      containers:
      - command:
        - /var/app/run-super-doper.sh
        volumeMounts:
        - mountPath: /var/app/super-doper
          name: super-doper-persistent-storage
        env:
        - name: LOG_LEVEL
          value: debug
        - name: PROCESSORS
          value: '2'
        - name: DOMAIN_NAME
          value: super-doper.default.svc.cluster.local
        - name: MONGO_MAIN_DB
          value: awesome-db
        - name: QUEUE_SERVER
          value: rabbitmq-svc.default.svc.cluster.local
        - name: QUEUE_USER
          value: test
        - name: QUEUE_PWD
          value: test
        - name: ENCRYPT_KEY
          valueFrom:
            secretKeyRef:
              key: encrypt_key
              name: common-secret
        image: mrgeek/super-doper:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /health-check
            port: 3002
            scheme: HTTP
          initialDelaySeconds: 300
          periodSeconds: 60
          timeoutSeconds: 58
        name: super-doper

No crazy stuff of course, looking at this you deduce the approach i'm taking to actually start this service. First by filling some metadata like the deployment name super-doper-nodejs-service and having this under a group super-doper and also passing some ENV variables to the pod that will be later used by our NodeJs application like for example our RabbitMQ server will be taken out of this ENV variable QUEUE_SERVER which has the value of rabbitmq-svc.default.svc.cluster.local which is our RabbitMQ service FQDN inside our kubernetes cluster.

Notice also volumes and volumeMounts as this is how we're telling kubernetes to mount the host path to the pod, so whenever we change some files, they will be reflected into the pod and hence a simple refresh is needed.

And this is the service file for this deployment, since we need to define also how are we going to communicate with this NodeJs app

apiVersion: v1
kind: Service
metadata:
  labels:
    app: super-doper
  name: super-doper-svc
spec:
  ports:
    - port: 3002
      protocol: TCP
  selector:
    app: super-doper

So now the app will run and we will start experiencing the issues with hot reloading not actually working as expected along with other issues due to this matter, mostly in services not able to communicate between each other, or changing some files will cause pod to crash or even to change some internal configurations entirely.

Its good to mention i tried this long time ago, that's why i'm not so sure about the issues but that is the glance of it. But i'm sure that i'm going to try it again especially after i fixed most of the issues using docker compose as we're going to see in the next part, and maybe kubernetes is not the problem after all and we can actually use it as a development cluster.

This is fine and mostly even be the average daily routine for all software engineers, but to get to the gist we'll have to understand what is Vagrant first.

How are we ab/using Vagrant?

Well, I mentioned VM above, and that's basically what Vagrant is. Simply by writing a small text file Vagrantfile in which you'll configure what VM you'll use, and what open ports you need along with much other stuff, Vagrant will then transform these text configurations into a VM box which you can just ssh to and manually install services just like you do on any Linux VM.

Now, what makes it good is that after you're happy with your box, and you got it prepared to contain all necessary services/dependencies you can then ship this box and distribute it among all of you developers so they can all have the same environment and by just simple commands each one can just get the software up & running.

So why bother moving then!?

Well, it has a major downside, which is the box itself! because now each developer will have to clone not only the Vagrantfile they will need to also download the box itself which is huge (our vagrant box was ~25Gb) and then developers will need to use git-lfs to download and this is an extra dependency besides git.

But the main downside is that our Vagrant box becomes hard to maintain and then sure outdated because it's just hard to upload then download ~25Gb whenever you change a line or something in the box.

This of course led to errors and issues that each developer was facing, and then he will have to inform all the team about the solution in case someone is facing the same problem again.

So obviously this is the exact kind of problem that Docker is solving, and giving that we're using Docker on production and Vagrant only for local environments it even made more sense to do it.

But wait! why docker? why not K8S?

We're using Docker on the production of course, but we're using Kubernetes to orchestrate everything, so hmmm... why not have the same infrastructure but tweak it as a local development environment for a variety of reasons and the most important one is to make developers familiar with K8s and to make them get used to it since each developer have to touch some Kubernetes files or even manage his own cloud development environment that is deployed specifically for each developer.

This was the actual beginning of the journey and of a developer who knows little about Vagrant trying to migrate it to K8s which he knows very very very very little about.

In the next post, I will try to explain and remember what I did and what issues I faced. Till then, keep yourself ALWAYS posted ;)