While preparing to release a new version of the Vladhog Security API, we discovered that we could no longer push images to our registry. In this post, we will explain what happened behind the scenes and why the outage occurred.

We were using Harbor registry. Despite the issues it has caused us in the past, we remained a fan for a while due to its strong access control, OAuth2 support, vulnerability scanning, and its ability to function as a proxy for other registries. Unfortunately, for reasons that were unclear at that time, Harbor suddenly began to break in ways we could no longer ignore.

A Harbor deployment consists of several key components:

• Registry service, responsible for storing and returning Docker images, as well as performing garbage collection.

• Core service, acting as both proxy server and API server, handling authorization requests and forwarding image pushes to the registry.

• PostgreSQL database.

• Redis cache.

• Job service, responsible for tasks like garbage collection and vulnerability scanning.

When we push a Docker image, the request flows:
Load balancer → Harbor core → Harbor registry

Harbor core started resetting connections, even after changing timeout values and related settings. Redis and PostgreSQL were both stable, so the problem was clearly inside Harbor itself. We increased the number of Harbor core replicas, but those replicas also began losing connection to the registry. Eventually even registry replicas began failing.

After exhausting all options and unable to find a reliable fix, we decided that relying on Harbor was no longer viable. The resets continued, slowing our updates and deployment pipeline significantly. We began the search for alternative.

Searching for a Replacement

We needed only a few things from a new solution: Kubernetes-ready deployment, self-hosted, free and reliable.

Docker Registry, Github Registry, and Others

The official Docker Registry and Github Registry were not suitable for us. Docker allows only one private repository on the free tier, which is impossible for our workflow. Github Registry limits packages to 500 MB, which is barely fits a single Vladhog Security Bot build, let alone our API server.

Gitlab Registry (Cloud and Self-hosted)

Gitlab’s registry was a possible option. However, the free cloud tier offers only 10 GB of storage, which is nearly at our limit. The self-hosted version is solid and can be easily deployed with Helm, but we kept looking.

Nexus Repository

A popular solution, but with one major issue: Nexus does not provide licenses for self-hosted use inside the Russia Federation, which immediately ruled it our for us.

JFrog Artifactory

Also widely used, but entirely paid, even for self-hosted versions. Not an option.

Project Quay (Red Hat Quay, Quay.io)

A strong candidate, but Kubernetes deployment is complicated. It requires installing through OpenShift operator, bringing additional management components we did not need.

The Solution: Gitea (Self-Hosted)

A long time ago, we attempted a project called Cargo, where we used a customized Gitea instance for public use. Although that project didn’t succeed, we decided to revisit Gitea for our internal registry needs.

Gitea ended up having everything we required:

• good documentation

• solid access control

• lightweight deployment

• simple registry support

• smooth Kubernetes setup

After deploying Gitea self-hosted, our registry problems disappeared. After nearly a week of Harbor troubleshooting and testing multiple alternatives, the solution turned out to be surprisingly lightweight and reliable.

Thank you for being with us!

Most important features,

• Its built using Chrome’s native blocking engine which making it fast

• It have no ads, trackers or anything not related to blocking malicious links

• Working quietly in background and not affecting your system performance

• Easy to install, technical knowledge not required and it have step by step installation guide

• Urls block list updates often and trying to be up to date with current attacks

Its compatible with all chromium based browsers such as Chrome, Brave, Edge and others

We, Vladhog Security trust them and we sure it will make your search safer not only in discord, but outside of discord as well.

Thank you for being with us!

Phish Blocker repository on GitHub - https://github.com/AceCombatx/phishing-blocker/releases/tag/v1.0.1

At this moment they already providing NSFW detection service for us and users of discord to better protect everyone, and together by sharing resources we will be able to protect better, together.

Thank you for being with us!

This update means:

• More stable bot work, because when we will update security checks, bot itself won't be affected

• Free api for everyone, every check will be free and documented

• Expanding bot will be easier and we won't have to worry about syncing security checks between many different servers

• Better and faster results, bot won't do caching on its side anymore, all caching will be done on api side and store results in more effective redis database.

Thank you for being with us!

Before:

• Session cookies was fully stored on users side

Now:

• Session cookies now fully stored on server side, we send to user only user token

• For safety, user token being reset and randomly generated again on every user login to protect users from session token hijack

On Radar main page you can find 2 maps that are showing locations of websites that we was getting. Anyone can use our statistics anywhere where you want, just make sure to leave statistics source.

Thank you for being with us!

How cache working

When someone send to us url that was already checked, we still pass it with each check that we usually do, each of modules just check if url is in cache and result source was not current module, then we skip module and continue until result source module found.

How cache poisoning was working

When you using /link_check command, you can choose order in which bot will check urls. If you choose all modules that you sure url will bypass, bot will end up adding safe result to cache, and in next check with all modules bot will return cached result without checking with modules that could possibly block url.

What actions was taken

We updated bot to force /link_check command to not use cache and not add any results to cache.

Thank you for being with us!

Vladhog Security will do full incident investigation with as many information as Maw Studio would like to provide, and start internal investigation against RHO-9, in which we will investigate if RHO-9 was related to Maw Studio incident. We hope conflict between our partners will be solved, and Maw Studio will continue their work.

Thank you for being with us.

We have considered the appeal (input dated April 22, 2024 No. 02-11-17910) on the issue of blocking the resources vladhog.ru, disforge.com, emoji.gg and report.
 On 04/01/2024, the Public Communications Network Monitoring and Management Center recorded a problem related to an attempt to mask the traffic of the AdGuard VPN service, blocked in accordance with paragraphs.  c) paragraph 5 of the Rules for centralized management of a public communications network, approved by a resolution of the Government of the Russian Federation
 dated 02/12/2020 No. 127 “On approval of the rules for centralized management of a public communications network” - threats to counter (difficulty) limiting access to information or information resources on the Internet, access to which is subject to restriction in accordance with
 with the legislation of the Russian Federation, under the operating mode of the TLS protocol, which led to a partial blocking of the TLS protocol on certain network segments.
 In order to quickly resolve the problem, a complete update of the equipment software was carried out.  Currently the problem is not urgent, resources are available.

Original text on Russian:

Рассмотрели обращение (вх. от 22.04.2024 № 02-11-17910) по вопросу блокировки ресурсов vladhog.ru, disforge.com, emoji.gg и сообщаем.
Центром мониторинга и управления сетью связи общего пользования 01.04.2024 зафиксирована проблема, связанная с попыткой маскировки трафика VPN-сервиса AdGuard, заблокированного в соответствии с пп. в) пункта 5 Правил централизованного управления сетью связи общего пользования, утвержденных постановлением Правительства Российской Федерации
от 12.02.2020 № 127 «Об утверждении правил централизованного управления сетью связи общего пользования» - угрозы противодействия (затруднения) ограничению доступа к информации или информационным ресурсам в сети "Интернет", доступ к которым подлежит ограничению в соответствии
с законодательством Российской Федерации, под режим работы протокола TLS, что привело к частичной блокировке протокола TLS на отдельных сегментах сети. 
В целях оперативного решения проблемы произведено полное обновление программного обеспечения оборудования. В настоящее время проблема не актуальна, ресурсы доступны.

We monitoring activities and will make announcement if we detect any other network anomaly.

Thank you for being with us!

In current V3 release we limited bot just with 2 things - experimental features not available. Its because we honestly not satisfied with their work and we don't think on current state any other discord server should use them.

As well as security bot V3 release, we have lots of other news, like our new team structure. Now and all team products working not just Vlad, but all other programmers with our wonderful director of research and technology department, thanks to our HR for this new team structure.

As well as new team structure, just a few small changes to dashboard and radar pages, because today Cloudflare announced of removing "auto minify" feature, we added it directly to our dashboard and radar pages. They are pretty useful and minify all pages html, css and js files so they will load faster.

In V3 version we added better support for our partner, LinkShield, features - their new tag system that can classify website brands by how they look like. Of how each link check works you can find and test by yourself better with /link_check command at our bot.

We working on more and better services in feature, join our support server where we answering on any of your questions: https://discord.gg/25Strv2Pq5

Thank you for being with us!

Thank you for partnership, and thank you everyone for being with us!

"A New Group is Working to Make Discord a Safer Platform" - from Paradigm Intel group by Aaron C.Z. Arnold

Prologue

Discord is one of the most popular messaging and social media applications globally, with around 150 million monthly active users. Popular messaging and social media platforms have become an important part of many people’s lives; whether it’s a group of friends wanting to talk to each other while playing video games or it’s a company trying to increase their public outreach, Discord has a lot of opportunities. However, like many social media platforms, there are a lot of online safety issues and concerns.

Vladhog Security (VHS) is a Discord bot that was created to scan all links sent in Discord servers that it’s in to make sure the link is not malicious or corrupt. The bot is managed and maintained by a team of developers and programmers who operate under the same name as the bot. As VHS continued to improve itself and expand its public outreach, the owner of VHS, Vlad, started thinking about other groups and individuals that have developed Discord bots to improve the online safety of Discord users. After much thought and consideration, Vlad decided to create a united group, or network, of like-minded organizations and individuals to work together to make Discord a safer platform.

Discord Safety Cooperation

The Discord Safety Cooperation (DSC), which was founded by Vlad on April 18, 2024, was created to make a united effort towards making Discord a safer platform.

The founding members of the DSC are Vladhog Security, LinkShieldAI, and Rho-9. LinkShieldAI and Rho-9 are also teams of individuals who manage and maintain Discord bots that scan malicious and suspicious links, similar to VHS. Vlad contacted the owners of these other two groups to ask if they wanted to join the DSC and both owners agreed to join.

When we asked Vlad about why he decided to create the DSC and what his goals were, he said “Every security bot and people who were working on making Discord safer had their own discord server, so I was thinking, why not make one server for all of them? [My] final goal [is to] get Discord’s attention to the problem, Discord [is] currently not a very safe place, they’re trying to do something but that’s not enough”. Vlad acknowledges that Discord has taken some actions to improve its online safety, however, it admits that there are still many issues and concerns.

When we asked the owner of LinkShieldAI about the DSC, he said “Vlad proposed that we create a server that gathers all the people that are interested in making Discord safer and their communities into one place”. He goes on to say “My main goal is to bring together people who care about making Discord safer. I want us to work as a team, sharing what we know and helping each other out. Creating this kind of network is vital for making a real difference and encouraging a culture of support and responsibility on Discord.”

Conclusion

The DSC will begin working towards making Discord a safer platform by sharing information with its member organizations and raising awareness about online safety issues. As DSC begins working on different projects and initiatives, Paradigm Intel looks forward to what they can accomplish, and we will support them in any way we can.

CREDITS

Aaron C.Z. Arnold, Board Member

For any questions or inquiries, please contact Paradigm Intel.

First, our security partner LinkShield asked as to review their example bot before publishing, it was nice to help you and we always ready to help you in future.

Second, we working on Vladhog Security bot V3. Currently already ready basic logging, database manipulations, some events and tasks processing. Today we working on adding commands, and after that we will work on hardest part - security modules.

And third, our team helping our another security partner RHO-9 with reviewing and recoding their security bot.

We hope to end with these stuff very soon, we have lots to do and lots in plans to do.

Thank you for being with us!

Thanks to everyone who supporting our work and helping us to keep working.

See you soon in V3.

Thank you for being with us!