yoy.be "Why-o-Why"

E-mail: Intensive Delphi 2016 and DMongoBR

2016-12-13 10:48  20161211d  delphi  [permalink]

I recently received an e-mail from Brazil where an event is about to take place: IntenseDelphi and one of the speakers will present a suite of components for MongoDB based in part on TMongoWire. As a starter I decided to respond with a write-up of a bit of history and background story of why TMongoWire came to be. It touches on a lot of other things as well, so I've made a slightly revised version to put on my website:
 
I can give you some background on TMongoWire, and if you want to know more, please ask.
 
First a bit about me, a little CV:
I started at a young age with a little BASIC on a CP/M (on a Kaypro II!),
later GW-BASIC but soon enough Turbo Pascal versions 3, 5 and 7. Almost by chance, as my father bought Turbo Pascal and Turbo C++ on a whim without really knowing what it was. My brother who is 4 years older also did some Pascal, but picked-up on C++ and later did heavy work in assembler as well.
Later, when I studied for what is now equivalent to Bachelor Computer Science, I was introduced to Delphi 3. Again almost by chance as someone I showed old Pascal DOS programs to, told me there was something new for Windows. After that I moved to Delphi versions 5 and then 7.
By then I attained my degree and started with a web-development firm. I first did PHP and later Cold Fusion, but did a lot of work of my own on Delphi-projects as a hobby.
But as hobbies go I did not put any money in new Delphi versions. Also I did not know any other Delphi developers professionally. I later heard about the schism in Delphi-world as the first versions after Delphi 7 had put off a lot of people because the IDE and editor was less good.
So I'm afraid I'm also one of those people that sticks with Delphi 7, but in part by consequence and only in part by choice. Still, thanks to the careful design of the Object Pascal language, code written for/with Delphi 7 still works really well in modern Delphi versions (just with taking care about a few little things like using AnsiString/WideString where possible.)
 
A funny fact is that I only learned about SQL and databases later on, around when I started to work on websites. I have done a lot of work in Delphi and not used the data-aware components! Also by doing PHP, ASP and Cold Fusion and developing websites without an IDE (just a source editor), I really disliked the different options you had to develop websites in Delphi. I'm conviced the IDE/RAD component eco-sphere is toxic to the performance-centric build-up of a website and will always make important choices for you that will cause problems later on when the project gets bigger.
 
So I started xxm. The Delphi compiler is so fast, it is possible to have a DLL that runs a website, when source-files change and you hit Refresh, to unload the DLL, compile a new one, load it and have it serve the page with updated logic. This way, developing a website without IDE or components would be almost the same as PHP and Cold Fusion, except in Object Pascal, and the website would perform as fast as a native service.
 
Programming a website like this reduces the work to creating a stream of HTML, and any visual components you would use offer extra overhead which takes a bit of your performance away. So that's where my search for 'really thin wrappers' comes from. When using Microsoft Access MDB's at first, and later MS SQL Server, Microsoft already developed the ADO components to do the heavy lifting, and thanks to the integration of ActiveX interfaces (IUnknown) and the type library importer (to generate ADODB_TLB.pas and others like MSXML2_TLB.pas) it's easy to use the ADO components directly (mainly Connection, Command and RecordSet, see xxmData.pas).
 
Later I also checked out other databases, but was displeased with the performance of the ODBC connector to SQLite databases. Other existing Delphi SQLite connectors were for modern Delphi versions and had long lists of 'overload' properties, which I really dislike. Using ADO much of the data-transport is based on OleVariant and I like that. Then I discovered the C API to SQLite is relatively straight-forward, and created TSQLite. Also PostgreSQL and MySQL/MariaDB have a C API DLL ready to use to connect almost directly. Since their SQL dialect is very similar, it should be easy to switch between the two (e.g. start a project off on SQLite, and later switch to PostgreSQL to serve a growing user base) so I created DataLank to take a lot of the work of switching away.
 
After that I also wanted to check out some of the new 'NoSQL' databases. I had no experience, and thought if I can write a connector I would understand more about how these databases worked.
With Redis I learned it had a human readable protocol, but was very basic and very much removed from the SQL world.
With CouchDB I learned some 'NoSQL' databases concentrate on storing (JSON) documents and can perhaps do more server-side but can do it fully separate from the front-end.
I tried RethinkDB several times: first I learned about Protocol Buffers and made Delphi support myself because I couldn't find any. By the time I got that working (remember, I'm a hobbyist programmer that only puts a few hours a week in my own projects), RethinkDB was moving away from 'ql2.proto' but was not ready with their new binary-level protocol. And the next time I checked, they were in some kind of trouble.
With MongoDB, I think a nice balance exists between storage of documents, and other features you need to do specific manipulations or queries. I was lucky to find my way through the layers of documentation to what I needed to know about the binary protcol. First BSON and then the wire protocol, both show careful design and are not needlessly complicated.
 
There are a few more on my to-do-list, but I kind of lost interest. I'm using TSQLite the most for the moment, and will probably start on a new connector if I'm sure there's the drive for it from a new project that's bound to have to run on this or that newfangled database. One big change I consider making somewhat soon, is reworking everything BSON but based on my own JSON parser I created because I needed it for a project that otherwise didn't need the bits specific for BSON...