MSVC

UPDATE 1 [2021/07/25]: It seems that Git LFS is able to resume your pushes after a network failure. At least it’s like that on Microsoft Azure DevOPS. So, it should be totally redundant to divide huge commits into smaller ones. How have I noticed this? Today, I pushed a huge single commit (around 53GBs) and it failed at 39GB due to a connection error without me noticing it for some time. A few hours later, when I made another attempt by issuing the push command again, it picked up and resumed the push at 39GB, which was really exciting.

UPDATE 2 [2021/07/25]: After pushing the repository to Azure DevOPS, if you find your self stuck in git pull without doing anything, the following command will fix the consecutive pulls:

$ git pull origin master

Or, alternatively:

$ git fetch origin master
$ git reset --hard FETCH_HEAD

UPDATE 3 [2021/07/28]: I’ve noticed due to the fact that the files modification times affect how Rsync and Git work by default, my approach in writing the original script was totally wrong, which in turn caused a bug where on each update it committed all tracked files over again causing huge bloat in the repository, despite the fact that the content of the files was unchanged. Thus, it led me to completely rewrite the script. Hopefully, the new script has been extensively tested with two repositories/projects and works as expected. In addition to that, the script now shows progress for every step, which is a nice addition in order to keep you informed and give an estimation of the time it is going to take to get the job done. And, last but not least, I have edited and improved the blog post a bit.

UPDATE 4 [2021/08/04]: Due to nested .gitignore files inside the Unreal Engine dependencies, I noticed tiny bits of dependencies for building UE4/UE5 on Microsoft Windows are not getting copied over to the repository. As a result, I fixed the script in order to also take care of that.

UPDATE 5 [2021/11/30]: Sometimes it’s possible that the amount of renamed Unreal Engine files surpass the Git’s optimal rename limit inside the Sync repository (the intermediary local git repository that we are going to use for syncing the engine source code with upstream):

warning: exhaustive rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 13453 and retry the command.

So, you could set that to a really large number in order to keep track of file renames:

$ cd ~/dev/MamadouArchives-Sync
$ git config diff.renameLimit 999999
$ git config merge.renameLimit 999999

Note: You will get this warning only when the Git option diff.renames is set to true (default behavior). Likewise, the above settings does not have any effects when the copy/rename detection is turned off. Nonetheless, you can always check your settings with:

$ git config -l

UPDATE 6 [2021/12/18]: I’ve added a step regarding EngineAssociation in the project’s .uproject file, which I forgot to mention in the original post.

UPDATE 7 [2023/03/03]: In UE5 UE4Games.uprojectdirs file shas been renamed to Default.uprojectdirs. Though the syntax and the contents of the file has remained the same.

UPDATE 8 [2023/03/04]: After upgrading my project to Unreal Engin 5.1 despite the fact that I’ve already set the git configuration http.version to HTTP/1.1 as instructed in this article, despite the commit size of no bigger than 166.30 MB and the acceptable upload bandwidth I’ve got, I was getting HTTP 413 Request Entity Too Large error:

Enumerating objects: 190058, done.
Counting objects: 100% (164439/164439), done.
Delta compression using up to 16 threads
Compressing objects: 100% (113439/113439), done.
Writing objects: 100% (138834/138834), 166.30 MiB | 47.32 MiB/s, done.
Total 138834 (delta 35613), reused 121343 (delta 22206), pack-reused 0
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date

I tried every suggestion that I came across in order to debug and resolve the issue to no avail. Including enabling git verbose logging:

$ export GIT_TRACE_PACKET=1                      
$ export GIT_TRACE=1
$ export GIT_CURL_VERBOSE=1

And then maxing out on all the size limits, buffer, packet sizes, and other hints:

[core]
	compression = 0
    packedGitLimit = 512m
    packedGitWindowSize = 512m
[http]
    postBuffer = 2147483648
[https]
    postBuffer = 2147483648
[init]
    defaultBranch = master
[pack] 
    deltaCacheSize = 2047m
    packSizeLimit = 2047m
	window = 1
    windowMemory = 2047m

Then I tried to change the origin URL to SSH:

[remote "origin"]
	url = https://SOME-ORGANIZATION@dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives
	#url = git@ssh.dev.azure.com:v3/SOME-ORGANIZATION/MamadouArchives/MamadouArchives

And then pushing without LFS:

$ git push --set-upstream origin 5.1 --no-verify

This attempt was futile as well, that made me revert back to https. Then I tried to push commit by commit since I had made a few commits using (5.1 which has been repeated twice in the following command, is the name of the new local branch intended to be pushed):

$ git rev-list --reverse 5.1 | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 1 == 0' | xargs -I{} git push origin +{}:refs/heads/5.1 --no-verify

And sadly, the approach of pushing one commit at a time was frutiless as well :/

Thus, for the time being I’m stuck pushing the updated project from my Linux machine and pulling it from my Windows machine. I’ll do another updated once I’ve figured what’s going wrong.

UPDATE 9 [2023/03/04]: As an experiment, I did create a new organiation and a new repository inside it. Then prior to changing the origin URL, I fetched all LFS objects issuing:

$ git lfs fetch --all
fetch: 78618 objects found, done.                                                                      
fetch: Fetching all references...

Then, I decided to first push the Git comnmits without the LFS objects, so after updating the origin URL inside the .git/config:

$ git push --set-upstream origin 5.1 --no-verify
Enumerating objects: 296578, done.
Counting objects: 100% (296578/296578), done.
Delta compression using up to 16 threads
Compressing objects: 100% (210384/210384), done.
Writing objects: 100% (296578/296578), 360.11 MiB | 5.78 MiB/s, done.
Total 296578 (delta 78143), reused 296578 (delta 78143), pack-reused 0
remote: Analyzing objects... (296578/296578) (77738 ms)
remote: Storing packfile... done (10303 ms)
remote: Storing index... done (3740 ms)
To https://dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives
 * [new branch]            5.1 -> 5.1
branch '5.1' set up to track 'origin/5.1'

¯\_(ツ)/¯ as unexpected as it seems, it worked! As you can see my actual Git objects without the LFS objects on this repository are in no way near the 10 GB size limit:_

$ git count-objects -vH                         
count: 0               
size: 0 bytes
in-pack: 296583
packs: 1
size-pack: 366.76 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

The issue might be that I’ve reached some kind of limit on the main organization that I’m not aware of.

Anyways, then I pushed the master and checked out back the new brnach for continuation on the upgrade:

$ git checkout master
$ git push origin master --no-verify
$ git checkout 5.1

And, then proceeded to pushing all LFS objects:

$ git lfs push origin --all

UPDATE 10 [2023/03/05]: Yesterday, I removed a large redundant repository from the previous organization, in order to see if I could still push my updates and the error I am getting was not due to hitting some kind of ceiling limit. It didn’t work. I did also cleanup the limit hacks I’ve added to my ~/.gitconfig in UPDATE 8. Then, after successfully pushing to the new organization/repository, I’ve decided to revert back the URL section for the orgin inside my .git/config inside the local repository and try to push once more to the old repository and guess what? It worked! Weird Microsoft/Azure! Not sure what fixed the issue. It could be even I had to wait for Microsoft to clean up the repository’s space I’ve deleted if the organization size limit was the issue. Don’t really know.

Among the gamedev industry, it’s a well-known fact that Unreal Engine projects sizes have always been huge and a pain to manage properly. And it becomes more painful by the day as your project moves forward and grows in size. Some even keep the Engine source and its monstrous binary dependencies inside their source control management software. In case you are a AAA game development company or you are working for one, there’s probably some system in place with an unlimited quota to take care of that. But, for most of us indie devs, or individual hobbyists, it seems there are not lots of affordable options, especially that your team is scattered across the globe.

There are plenty of costly solutions to keep UE4 projects under source control; ranging from maintaining a local physical server or renting a VPS with plenty of space on the cloud, equipped with a self-hosted Git, SVN, or Perforce, to use cloud SCM providers such as GitHub, GitLab, BitBucket, or Perforce. Since I prefer cloud SCM providers and Git + Git LFS (which also supports file locking), let’s take a look at some popular ones such as GitHub and GitLab.

GitHub for one, provides data packs, but the free offering is far from enough for collaborative UE4 projects:

Every account using Git Large File Storage receives 1 GB of free storage and 1 GB a month of free bandwidth. If the bandwidth and storage quotas are not enough, you can choose to purchase an additional quota for Git LFS. Unused bandwidth doesn’t roll over month-to-month.

…

Additional storage and bandwidth is offered in a single data pack. One data pack costs $5 per month, and provides a monthly quota of 50 GB for bandwidth and 50 GB for storage. You can purchase as many data packs as you need. For example, if you need 150 GB of storage, you’d buy three data packs.

For GitLab, although the initial generous 10GB repository size is way beyond the 1GB repository size offer by GitHub, the LFS pricing is insanely high:

1. Additional repository storage for a namespace (group or personal) is sold in annual subscriptions of $60 USD/year in increments of 10GB. This storage accounts for the size calculated from Repositories, which includes the git repository itself and any LFS objects.

2. When adding storage to an existing subscription, you will be charged the prorated amount for the remaining term of your subscription. (ex. If your subscription ends in 6 months and you buy storage, you will be charge for 6 months of the storage subscription, i.e. $30 USD)

Well, before this all get you disappointed, let’s hear the good news from the Microsoft Azure DevOPS team:

In uncommon circumstances, repositories may be larger than 10GB. For instance, the Windows repository is at least 300GB. For that reason, we do not have a hard block in place. If your repository grows beyond 10GB, consider using Git-LFS, Scalar, or Azure Artifacts to refactor your development artifacts.

Before we proceed any further, there are some catches to consider about Microsoft Azure DevOPS:

1. The maximum Git repository size is 10GB, which considering that we keep binary assets and huge files in LFS, is way beyond any project’s actual needs. For Git LFS it seems that Microsoft since at least 2015 has been providing unlimited free storage. For comparison, the engine source code for 4.27 is 1.4GB, which in turn when it’s getting committed to the git repo becomes less than 230MB:

$ cd /path/to/ue4.27/source

$ du -h
1.4G	.

$ git init
$ git add .
$ git commit -m "add unreal engine 4.27 source code"

$ git count-objects -vH

count: 97545
size: 900.25 MiB
in-pack: 110815
packs: 1
size-pack: 227.80 MiB
prune-packable: 97545
garbage: 0
size-garbage: 0 bytes

2. The maximum push size is limited to 5GB at a time. The 5GB limit is only for files in the actual repository and it won’t affect LFS objects. Thus, there are no limits for LFS objects’ pushes. Despite that, if your internet connection is not stable, you could divide your files into multiple commits and push them separately. For example, the initial git dependencies for UE 4.27 is around 40GB spanned across ~70,000 files. Instead of committing and pushing a 40GB chunk all at once, one could divide that into multiple commits and push those commits one by one using the following command:

$ git rev-list --reverse master \
    | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 1 == 0' \
    | xargs -I{} git push origin +{}:refs/heads/master

3. Sadly, at the moment Azure DevOPS does not support LFS over SSH. So, you are bound to git push/pull over https, which for some might be annoying. Especially, that it keeps asking for the https token 3 consecutive times on any push or pull!

Q: I’m using Git LFS with Azure DevOps Services and I get errors when pulling files tracked by Git LFS.

A: Azure DevOps Services currently doesn’t support LFS over SSH. Use HTTPS to connect to repos with Git LFS tracked files.

4. Last but not least, there is an issue with the Microsoft implementation of LFS, which rejects large LFS objects and spits out a bunch of HTTP 413 and 503 errors at the end of your git push. It happened to me when I was pushing 40GB of UE4 binary dependencies. The weird thing was I tried twice and both times it took a few good hours till the end of the push operation and based on measuring the bandwidth usage, the LFS upload size appeared to be more than the actual upload size. According to some answers on this GitHub issue and this Microsoft developer community question, it seems the solution is running the following command inside the root of your local repository, before any git pull/push operations:

$ git config http.version HTTP/1.1

Well, not only it did the trick and worked like a charm, but also the push time on the following git push dropped dramatically to 30 minutes for that hefty 40GB UE4 binary dependencies.

OK, after getting ourselves familiarized with all the limits, if you deem this solution a worthy one for managing UE4 projects along with the engine source in the same repository, in the rest of this blog post I’m going to share my experiences and a script to keep the engine updated with ease using a Git + LFS setup.

Well, anyone who has ever developed a C++ game or application with Gregorian to Jalali date conversion (or, vice versa) requirement is well aware of the hurdles of doing such a task. I, for one, have been maintaining my own cross-platform C++ library for almost two decades now, with occasional bugs coming up from time to time.

Recently, I have been playing with boost::locale (developed by Artyom Beilis and contributed to Boost project) a bit more in order to utilize it in a personal project called Barandazstorm, a home-grown social media analysis tool. Browsing the docs, I noticed the following example which does not even compile on my compiler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

using namespace boost::locale;
using namespace boost::locale::period;
generator gen;
// Create locales with Hebrew and Gregorian (default) calendars.
std::locale l_hebrew=gen("en_US.UTF-8@calendar=hebrew");
std::locale l_gregorian=gen("en_US.UTF-8");
// Create a Gregorian date from fields
date_time greg(year(2010) + february() + day(5),l_gregorian);
// Assign a time point taken from the Gregorian date to date_time with
// the Hebrew calendar
date_time heb(greg.time(),l_hebrew);
// Now we can query the year.
std::cout << "Hebrew year is " << heb / year << std::endl;

So, I tried to make a guess and replaced the en_US.UTF-8@calendar=hebrew part with en_US.UTF-8@calendar=jalali which didn’t work. But, on the second try replacing that with en_US.UTF-8@calendar=persian actually worked! Which is sheer delight; due to the fact that now I found a solution to convert dates between both calendars as efortless as techonologies such as .NET in C++:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

/**
* @file
* @author  Mamadou Babaei <info@babaei.net>
* @version 0.1.0
*
* @section LICENSE
*
* (The MIT License)
*
* Copyright (c) 2020 Mamadou Babaei
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* @section DESCRIPTION
*
* Two-way Gregorian / Jalali date conversion using boost::locale example
*/

#include <iomanip>
#include <iostream>
#include <cstdlib>
#include <boost/locale.hpp>

void from_gregorian_to_jalali()
{
    boost::locale::generator generator;

    std::locale locale_gregorian = generator("en_US.UTF-8");
    std::locale locale_jalali = generator("en_US.UTF-8@calendar=persian");

    boost::locale::date_time gregorian(
            boost::locale::period::year(2020)
            + boost::locale::period::february()
            + boost::locale::period::day(25),
            locale_gregorian);

    boost::locale::date_time jalali(gregorian.time(), locale_jalali);

    std::cout << "Persian date is " << jalali / boost::locale::period::year()
            << "/" << std::setfill('0') << std::setw(2)
            << (jalali / boost::locale::period::month()) + 1
            << "/" << std::setfill('0') << std::setw(2)
            << jalali / boost::locale::period::day()
            << "." << std::endl;
}

void from_jalali_to_gregorian()
{
    boost::locale::generator generator;

    std::locale locale_gregorian = generator("en_US.UTF-8");
    std::locale locale_jalali = generator("en_US.UTF-8@calendar=persian");

    boost::locale::date_time jalali(
            boost::locale::period::year(1398)
            + boost::locale::period::month(11)
            + boost::locale::period::day(06),
            locale_jalali);

    boost::locale::date_time gregorian(jalali.time(), locale_gregorian);

    std::cout << "Gregorian date is " << gregorian / boost::locale::period::year()
            << "/" << std::setfill('0') << std::setw(2)
            << (gregorian / boost::locale::period::month()) + 1
            << "/" << std::setfill('0') << std::setw(2)
            << gregorian / boost::locale::period::day()
            << "." << std::endl;
}

int main()
{
    from_gregorian_to_jalali();
    from_jalali_to_gregorian();

    return EXIT_SUCCESS;
}

On a side note, for the above code to work your boost::locale libraries has to be built with ICU support; otherwise boost::locale throws an exception. On most Linux or BSD distros this is the default when you install Boost libraries from your distro’s package manager. On Windows, it requires a bit of effort if you are trying to build Boost binaries yourself, which is another story for another time.

I did test the above code on FreeBSD, Linux, and Microsoft Windows, using MSVC, GCC, LLVM/Clang, and MinGW, and it’s working as expected on all of these platforms.

Almost three weeks ago I received a review request from one of the Packt Publishing staffs to review Application Development with Qt Creator, 2nd Edition written by Ray Rischpater which has been recently published by Packt Publishing. Since I’ve been developing cross platform Qt (cute, often mispronounced as Q-T cue-tee) applications from Qt 4 era back in 2008 – when Qt Creator was not around yet and the project was running by Trolltech at the time – and a handful of Qt Quick mobile applications over the past two years, I consider myself eligible enough to write a brief review on it. So, I was provided with a review copy and after reading it cover to cover, my thoughts on the book are as follows.

Previously I’ve described the process of building Crypto++ on both FreeBSD and Windows using the GCC, MinGW and VC++ compilers.

Now, we want to develop our own cross-platform cryptographic wrapper library around Crypto++. I’ve already uploaded the full source code to GitHub. You can find the link to the code on GitHub at the end of this article.

Before you proceed, you have to build the Crypto++ library as I mentioned earlier here.

Crypto++ is an awesome free and open source C++ class library of cryptographic algorithms and schemes which fully supports 32-bit and 64-bit architectures for many major operating systems, including FreeBSD, Linux, Solaris, Windows, Mac OS X and iOS. Currently, Crypto++ officially supports the following compilers:

• MSVC 6.0 - 2010
• GCC 3.3 - 4.5
• C++Builder 2010
• Intel C++ Compiler 9 - 11.1
• Sun Studio 12u1, Express 11/08, Express 06/10

The latest version at the time of this writing is 5.6.1.

In spite of the power that Crypto++ offers, building and using it can be a little bit tricky. In the following we will describe the process of building Crypto++ on both FreeBSD and Windows using the GCC, MinGW and VC++ compilers.

Most of the time I see some C++ programmers who check if a pointer is NULL before deleting it.

if (ptr != NULL) {
    delete ptr;
    ptr = NULL;
}

Well, according to C++03 [ISO/IEC IS 14882:2003] §5.3.5/2 which explicitly states:

…if the value of the operand of delete is the null pointer the operation has no effect.

Therefore, deleting a NULL pointer has no effect (if the deallocation function is one supplied in the standard library), so it is not necessary to check for a NULL pointer before calling delete.

delete ptr;
ptr = NULL;

• Note: Keep in mind that deleting a void* pointer results in undefined behavior whether it’s NULL or not.