FreeBSD

I recently purchased a second-hand HP Proliant ML350p Gen8 in order to be used as a home server and to my disappointment realized I won’t be able to disable the hardware RAID that comes with this model, at least at first glance. Well, there is a way to do that, which is supported by HP themselves. And, this is how to do it the easy way!

But, before we proceed any further, you might ask why do I need to disable the hardware RAID? The answer is because I need to install ZFS. There is already plenty of documentation why installing ZFS on a hardware RAID is a terrible idea. Thus, I won’t go over that since it’s not the focus of this article.

Again, before we go any further there’s a catch you should know about. If you’d enable HBA-mode, the server won’t be able to boot from any disk connected in HBA mode to the controller! You should consider this before converting to HBA mode. In case you need to perform a FreeBSD/Linux root on ZFS installation through this controller there are two solutions:

1. Installing in hardware RAID mode, but making each disk a RAID-0 array consisting of only one disk. For example, if you’ve got 8 disks, you’ll end up with 8 RAID-0 arrays. Then you’d perform a ZFS installation and your operating system boots as expected. Though this is not recommended and if you’d proceed with this approach, it renders the rest of this post useless.

2. HP Proliant ML350p provides an SD-Card slot, which can be used to install a full system, which is not recommended due to the wear and tear effect of SDCards with each write-operation on them. In addition to that, this storage type is costly and slow. For example, a SanDisk SDXC Extreme Pro 256GB, which provides a write speed of 90MB/s and read speed of 170MB/s, costs around €99,99 where I live. With today’s standards, this is not fast at all or even good enough, especially on a server. I was also able to find a Sandisk CF Express Extreme Pro 512GB type B, with a write speed of 800MB/s and a read speed of 1500MB/s. It only costs €629,-, which costs an arm and leg to buy!

So, then! What’s the solution one might ask? We are going to install only /boot partition on the internal SD Card. Note, that placing only a bootloader such as GRUB, rEFInd or other similar tools won’t work as they won’t be able to see the boot drive anyway! So, the only solution is to put the boot partition on the SDCard. This way, the system only reads it one time at boot and it does not even have to be an expensive SD-Card.

So, let’s begin!

Well, anyone who has ever developed a C++ game or application with Gregorian to Jalali date conversion (or, vice versa) requirement is well aware of the hurdles of doing such a task. I, for one, have been maintaining my own cross-platform C++ library for almost two decades now, with occasional bugs coming up from time to time.

Recently, I have been playing with boost::locale (developed by Artyom Beilis and contributed to Boost project) a bit more in order to utilize it in a personal project called Barandazstorm, a home-grown social media analysis tool. Browsing the docs, I noticed the following example which does not even compile on my compiler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

using namespace boost::locale;
using namespace boost::locale::period;
generator gen;
// Create locales with Hebrew and Gregorian (default) calendars.
std::locale l_hebrew=gen("en_US.UTF-8@calendar=hebrew");
std::locale l_gregorian=gen("en_US.UTF-8");
// Create a Gregorian date from fields
date_time greg(year(2010) + february() + day(5),l_gregorian);
// Assign a time point taken from the Gregorian date to date_time with
// the Hebrew calendar
date_time heb(greg.time(),l_hebrew);
// Now we can query the year.
std::cout << "Hebrew year is " << heb / year << std::endl;

So, I tried to make a guess and replaced the en_US.UTF-8@calendar=hebrew part with en_US.UTF-8@calendar=jalali which didn’t work. But, on the second try replacing that with en_US.UTF-8@calendar=persian actually worked! Which is sheer delight; due to the fact that now I found a solution to convert dates between both calendars as efortless as techonologies such as .NET in C++:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

/**
* @file
* @author  Mamadou Babaei <info@babaei.net>
* @version 0.1.0
*
* @section LICENSE
*
* (The MIT License)
*
* Copyright (c) 2020 Mamadou Babaei
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* @section DESCRIPTION
*
* Two-way Gregorian / Jalali date conversion using boost::locale example
*/

#include <iomanip>
#include <iostream>
#include <cstdlib>
#include <boost/locale.hpp>

void from_gregorian_to_jalali()
{
    boost::locale::generator generator;

    std::locale locale_gregorian = generator("en_US.UTF-8");
    std::locale locale_jalali = generator("en_US.UTF-8@calendar=persian");

    boost::locale::date_time gregorian(
            boost::locale::period::year(2020)
            + boost::locale::period::february()
            + boost::locale::period::day(25),
            locale_gregorian);

    boost::locale::date_time jalali(gregorian.time(), locale_jalali);

    std::cout << "Persian date is " << jalali / boost::locale::period::year()
            << "/" << std::setfill('0') << std::setw(2)
            << (jalali / boost::locale::period::month()) + 1
            << "/" << std::setfill('0') << std::setw(2)
            << jalali / boost::locale::period::day()
            << "." << std::endl;
}

void from_jalali_to_gregorian()
{
    boost::locale::generator generator;

    std::locale locale_gregorian = generator("en_US.UTF-8");
    std::locale locale_jalali = generator("en_US.UTF-8@calendar=persian");

    boost::locale::date_time jalali(
            boost::locale::period::year(1398)
            + boost::locale::period::month(11)
            + boost::locale::period::day(06),
            locale_jalali);

    boost::locale::date_time gregorian(jalali.time(), locale_gregorian);

    std::cout << "Gregorian date is " << gregorian / boost::locale::period::year()
            << "/" << std::setfill('0') << std::setw(2)
            << (gregorian / boost::locale::period::month()) + 1
            << "/" << std::setfill('0') << std::setw(2)
            << gregorian / boost::locale::period::day()
            << "." << std::endl;
}

int main()
{
    from_gregorian_to_jalali();
    from_jalali_to_gregorian();

    return EXIT_SUCCESS;
}

On a side note, for the above code to work your boost::locale libraries has to be built with ICU support; otherwise boost::locale throws an exception. On most Linux or BSD distros this is the default when you install Boost libraries from your distro’s package manager. On Windows, it requires a bit of effort if you are trying to build Boost binaries yourself, which is another story for another time.

I did test the above code on FreeBSD, Linux, and Microsoft Windows, using MSVC, GCC, LLVM/Clang, and MinGW, and it’s working as expected on all of these platforms.

For those who don’t know, I maintain various official FreeBSD ports and for almost 18 months my submitted updates to www/wt has been stuck due to lack of a review. Hopefully, tonight it has been committed to the official FreeBSD Ports tree and has been divided into two ports:

1. www/wt for following the 4.x releases
2. www/wt3 for following the 3.x releases

Although Wt Webtoolkit 4.x is a significant update which brings in more modern C++ and performance improvements (read more about the changes between releases on their archive), I haven’t migrated any project to the 4.x release yet. I guess the 3.x releases will live on for the time being and any project using them should be fine for some time.

UPDATE 1 [2019/05/11]: Thanks to @mirrorbox’s suggestion, I refactored the script to use service status instead of ps aux | grep which makes the script even more simple. As a result, the syntax has changed. Since I keep the article untouched, for the updated code visit either the GitHub or GitLab repositories. The new syntax is as follows:

# Syntax
$ /path/to/daemon-keeper.sh

Correct usage:

    daemon-keeper.sh -d {daemon} -e {extra daemon to (re)start} [-e {another extra daemon to (re)start}] [... even more -e and extra daemons to (re)start]

# Example
$ /path/to/daemon-keeper.sh -d "clamav-clamd" -e "dovecot"

# Crontab
$ sudo -u root -g wheel crontab -l

# At every minute
*   *   *   *   *   /usr/local/cron-scripts/daemon-keeper.sh -d "clamav-clamd" -e "dovecot"

UPDATE 2 [2019/05/11]: Another thanks to @mirrorbox for mentioning sysutils/daemontools which seems a proven solution for restarting a crashing daemon. It makes this hack redundant.

Daemontools is a small set of /very/ useful utilities, from Dan
Bernstein.  They are mainly used for controlling processes, and
maintaining logfiles.

WWW: http://cr.yp.to/daemontools.html

UPDATE 3 [2019/05/11]: Thanks to @dlangille for mentioning sysutils/py-supervisor, which seems to be a viable alternative to sysutils/daemontools.

Supervisor is a client/server system that allows its users
to monitor and control a number of processes on UNIX-like
operating systems.

It shares some of the same goals of programs like launchd,
daemontools, and runit. Unlike some of these programs, it is
not meant to be run as a substitute for init as "process id 1".
Instead it is meant to be used to control processes related to
a project or a customer, and is meant to start like any
other program at boot time.

WWW: http://supervisord.org/

UPDATE 4 [2019/05/13]: Thanks to @olevole for mentioning sysutils/fsc. It is minimalistic, dependency free and designed for FreeBSD:

The FreeBSD Services Control software provides service
monitoring, restarting, and event logging for FreeBSD
servers.  The core functionality is a daemon (fscd)
which is interfaced with using fscadm.  See manual pages
for more information.

UPDATE 5 [2019/05/13]: Thanks to @jcigar for bringing daemon(8) to my attention, which is available in the base system and it seems perfectly capable of doing what I was going to achieve in my script and more.

Amidst all the chaos in the current stage of my life, I don’t know exactly what got into me that I thought it was a good idea to perform a major upgrade on a production FreeBSD server from 11.2-RELENG to 12.0-RELENG, when I even did not have enough time to go through /usr/src/UPDATING thoroughly or consult the Release Notes or the Errata properly; let alone hitting some esoteric changes which technically crippled my mail server, when I realized it has been over a week that I haven’t been receiving any new emails.

At first, I did not take it seriously. Just rebooted the server and prayed to the gods that it won’t happen again. It was a quick fix and it seemed to work. Until after a few days, I noticed that it happened again. This time I prayed to the gods even harder - both the old ones and the new ones ¯\_(ツ)_/¯ - and rebuilt every installed ports all over again in order to make sure I did not miss anything. I went for another reboot and, ops! There it was again laughing at me. Thus, losing all faith in the gods, which led me to take up responsibility and investigate more on this issue or ask the experts on the FreeBSD forums.

After messing around with it, it turned out that the culprit is clamav-clamd service crashing without any apparent reason at first. I fired up htop after restarting clamav-clamd and figured even at idle times it devours around ~ 30% of the available memory. According to this Stack Exchange answer:

ClamAV holds the search strings using the classic string (Boyer Moore) and regular expression (Aho Corasick) algorithms. Being algorithms from the 1970s they are extemely memory efficient.

The problem is the huge number of virus signatures. This leads to the algorithms’ datastructures growing quite large.

You can’t send those datastructures to swap, as there are no parts of the algorithms’ datastructures accessed less often than other parts. If you do force pages of them to swap disk, then they’ll be referenced moments later and just swap straight back in. (Technically we say “the random access of the datastructure forces the entire datastructure to be in the process’s working set of memory”.)

The datastructures are needed if you are scanning from the command line or scanning from a daemon.

You can’t use just a portion of the virus signatures, as you don’t get to choose which viruses you will be sent, and thus can’t tell which signatures you will need.

I guess due to some arcane changes in 12.0-RELEASE, FreeBSD kills memory hogs such as clamav-clamd daemon (don’t take my word for it; it is just a poor man’s guess). I even tried to lower the memory usage without much of a success. At the end, there were not too many choices or workarounds around the corner:

A. Pray to the gods that it go away by itself, which I deemed impractical

B. Put aside laziness, and replace security/clamsmtp with security/amavisd-new in order to be able to run ClamAV on-demand which has its own pros and cons

C. Write a quick POSIX-shell script to scan for a running clamav-clamd process using ps aux | grep clamd, set it up as a cron job with X-minute(s) interval, and then start the server if it cannot be found running, and be done with it for the time being.

For the sake of slothfulness, I opted to go with option C. As a consequence, I came up with a generic simple script that is able to not only monitor and restart the clamav-clamd service but also is able to keep any other crashing services running on FreeBSD.

Update [2019/05/08]: Many people have been asking for the wallpaper in the above screenshot. It is from System Failure II, oil on canvas, 31x43” on r/Art.

Well, I am really fascinated by Reddit art and genuine creative ideas such as Scrolller which was made possible thanks to gazillions of art pieces scattered throughout various art subreddits. I am also fascinated by Unix philosophy and have been a *nix enthusiast for as long as I can remember. In addition to all this, the discovery of r/unixporn - realizing I am not the only one who cares about aesthetics of their Unix box - was a huge blow for me; to the point that studying the GitHub dotfiles posted along the screenshots on r/unixporn by fellow nix-enthusiast redditors felt like a day to day hubby for me.

All the while, I had a successful experiment with writing a complex piece of real-world software in pure Bash with an amazingly wide range of features for around 3.5K lines of code. The real excitement came when it made to the official FreeBSD Ports Tree. In spite of the fact that many people find Bash syntax annoyingly ugly and unmaintainable and often wonder why do people still write shell scripts by asking it on Quora, since MS-DOS 6.22 era, I did develop a certain love–hate relationship with shell scripting languages such as Batch Files, Bash, etc. Thus, still I do automate almost everything with these ancient technologies.

So, here is my fully-configurable wallpaper changer software written in bash which automagically fetches and display wallpapers from your favorite subs. It has been powering and brightening up my i3wm setup for the past eight months which led me to the conclusion that it deserves a proper introduction.

Update 1 [2016/09/30]: A section has been added to the end of the article for dealing with major FreeBSD upgrades.

Update 2 [2016/09/30]: A section has been added to the end of the article for upgrading current installation of Discourse to newer versions.

Update 3 [2016/09/30]: A section has been added to the end of the article for installing Discourse under Ruby version managers which is required for dealing with newer versions of Discourse since the current system-wide version of Ruby on FreeBSD is 2.2.5p319.

Update 4 [2016/10/06]: I decided to get rid of Discourse on this blog for various reasons including negative feedback from my readers, performance issues, being a memory hog and not so easy on memory, difficult maintenance, dealing with building Ruby Gems which is a tedious task in case they fail to build and a bug that duplicates my posts and creates a new thread for each post which means it won’t show previous comments. Last but not least, in my estimation it’s too heavy for such a small task such as a comment system. As a result, this guide won’t be maintained anymore.

Well, when it comes to blog comment hosting services for static blogs, you will have a plethora of options such as SolidOpinion, Disqus, Livefyre, Google+ or Facebook comments, and many more. Unfortunately, such services has never been an option for me and I resisted them like forever. Yes, I’ve got one million reasons to believe and do so which demands another post of its own. So, I’ll avoid that argument for now.

This left me with two choices:

• Operate my blogs without the comment section as I did for years
• Running an open source self-host blog comment system such as HashOver, Isso, Juvia and talkatv

Since people asked me for a way to discuss their feedback on the website, this made the former choice a no go, anymore. So, in a search for the promised commenting system, I spent hours installing and trying every single FLOSS commenting system on GitHub. And everyone of them has had a big flaw that I could not tolerate. Finally, I came to the conclusion that I’m out of luck with that. Of course, I always had the option to write my own commenting system according to my own needs. As, I did with my own blog subscription system. But, due to the lack of time that was not an option either.

It happens that I randomly visit Coding Horror by Jeff Atwood the co-founder of Stack Overflow and Stack Exchange Network. I’ve always read the comments on Coding Horror but never had an urge to write a comment their. Since I’ve been desperately looking for a commenting system, this time the different look of its comment system catched my attention. Bingo! He wrote an open source discussion platform software named Discourse. Not necessarily a blog comment system, but could be used as one. It even supports the Farsi language and RTL out of the box which allows me to use it on my Farsi blog, too. In addition to that it supports multi-site which means I have to only run one instance for all my blogs. Just perfect!

Despite many nice feature it provides, it has a few rough edges, too. I was able to conquer some and some couldn’t, yet. But it did not stop me from integrating it into my Hexo-based blogs after two weeks of testing it. So, there it is, from now on both this blog and my Farsi blog are discussion aware :).

OK, the main issue that many people face when they want to install Discourse in a platform other than GNU/Linux, is they get hit in the face by the reading this line in the official documentation:

Why do you only officially support Docker?

Hosting Rails applications is complicated. Even if you already have Postgres, Redis and Ruby installed on your server, you still need to worry about running and monitoring your Sidekiq and Rails processes. Additionally, our Docker install comes bundled with a web-based GUI that makes upgrading to new versions of Discourse as easy as clicking a button.

Yes, I know Ruby and Rails are crap and deploying Ruby on Rails apps are pain in the Butt. Do not worry! This was the first struggle with Discourse that I had since this blog runs on FreeBSD. And, FreeBSD support for Docker is experimental, yet. Fortunately, I used to deploy GitLab instances on FreeBSD for three years which was also RoR before I migrated to Gogs which kicks butt, anyway! That made it easy to figure out a simple way to deploy Discourse without Docker on FreeBSD.

There once was a time when I did the following inside my home directory:

$ wget "some-url" -O "output-file.mp4"

I clearly remember copying the output file name from a web page. Unfortunately, the copied text has a new line at the beginning of it and I didn’t notice that. That’s because the newline or carriage return characters are control characters and have no visual representation. Anyway, when I listed files inside my home directory I noticed a strange file name on my list:

$ ls
?output-file.mp4

Undeniably, flaming has been around since the dawn of the Internet age. And even worse, it has been on the rise ever since. Gazillions of flame wars on websites such as Slashdot or Reddit surely is enough proof, the greatest of which is the perennial debates between users of competing operating systems.

No doubt I’m a FLOSS enthusiast and advocate who considers free/libre/azad software superior to the proprietary ones, by all means possible. But like all software, the FLOSS ones have some shortcomings, too. So, the point is no one-hundred percent reliable software exists on this planet which means open source software may also suffer from bugs or security flaws as proprietary software do.

Today is Linux kernel birthday and many FLOSS enthusiast and advocates around the globe are going to celebrate it. I’m a diehard FreeBSD fan, but it won’t stop me from celebrating Linux birthday. In fact, I still use GNU/Linux for building native C/C++ libraries using Android NDK on a Slackware machine. In addition to that, I used to be a heavy Funtoo / Gentoo user. The thing is, I appreciate the tools I use, no matter what. So, I’m going to celebrate Linux birthday by telling you a story. A story that made an unforgettable trip even more memorable. Here’s the story of a GNU/Linux system in production which crashed in its hometown, Helsinki, Finland.