yoy.be "Why-o-Why"

MurMurHash3

2017-05-19 20:09  murmur3  delphi freeware  [permalink]

@stijnsanders do you have MurMurHash3 code in pascal aswell ?
i truly liked your optimized code for md5 and all those

— _pusher_ (@_pusher_0x90) 12 mei 2017

Why, thank you. Eeuh, is it at all optimized? I took some decisions that may perform a little better than the reference implementation, but I haven't taken any time to compare to see if it actually performs any better or worse...

So MurMurHash3... Let's have a look, it's on wikipedia, so it's a thing. And the reference C implementation is in the public domain, great! Looks straight-forward enough, could boil down to a translation-job...

Rougly two hours later, got triple zero (0 errors, 0 warnings, 0 hints). Now for checking if I got it all right. Hmm, not much there, but those x32_64 match, so barring any typo's, assuming this pretty straight translate job will result in the expected behaviour for the other two functions, this should be it:

md5.zip (25KB)

Checking xxm for PHP's vulnerabilities

2017-05-12 07:56  xxmphp1  delphi internet freeware  [permalink]

If I read about a newly discovered vulnerability related to PHP, for example this one here, I try to find out if it would apply to xxm as well. 

In this case I guess there's nothing more than sending out the message, again and again, to sanitize your inputs, and poperly encode your output. Strings are never just strings. They are always an internal representation of a bit of textual data. So always think about that taking string values in, and preparing strings for output. A few weeks back I had to speak up to someone who wrote OutJSON:='{"field1":"'+value1+'"}';Little Bobby Tables comes to mind, though I'm not sure 'JSON injection' could be so devastating as SQL injection. (And OutJSON:=JSON(['field1',value1]); is shorter!)

The other time I found out it's a really good idea to strip nastiness like EOL's (CRLF) from headers added to a response, just in case a malicious script is up to no good. Come to think of it, that's also just another case of properly sanitizing your inputs...

Delphi, the shrinking island.

2017-05-09 21:48  delphitrend  coding delphi  [permalink]

Is there a language that has a single word for 'the feeling of being on a shrinking island'? Anyway, this is somewhat sad to see, especially that the few most recent Delphi versions gave the impression there was a new uptake with more people getting persuaded, but it doesn't show in the curve.

(via)

'Relax' scripting vs xxm

2017-04-22 13:12  relaxxxm  coding delphi internet  [permalink]

Delphi Relax Web Scripting (Marco Tech Blog)

I'm sorry but I feel I must react. In general I keep silent, in the hope people by themselves will know better, but as I'm getting no input what-so-ever that that is the case, I feel tempted to write something about this.

First about what's at hand. I see this bit of code:

<h2>Employees</h2>
<ul>
@foreach (var emp in employee) {
  <li>@emp.FirstName @emp.LastName (@emp.PhoneExt)</li>
}
</ul>

and it looks kind-of OK. To the untrained eye it looks good and may even look tempting to write more in this syntax. This is a straight-forward example of a template that works with a templating engine that no doubt has many more capabilities and features. And then I thought, learned from practice, what I typically would get asked is to not show " ()" when the PhoneExt field is empty. I would not know how to make that happen in that template syntax. That's mainly because I know nothing about the template syntax. If I look into the documentation, I might find an @if predicate to make it happen, but let's move on:

This is what it could look like in xxm:

[[!var emp:TEmployee;
<<h2>Employees</h2>
<ul>>
foreach emp in FDMemTable1 do
begin
<<li>>=[emp.FirstName,' ',emp.LastName,' (',emp.PhoneExt,')']<</li>>
end;
<</ul>

Looks roughly simlar. A little more like Delphi syntax. And in fact it is. If you know [[, ]], << and >> get translated into Context.SendHTML() and Context.Send() calls behind the scenes (full details are here),  you know this code will result in the same output. Without templating engine! Streamed to the user's client! Perhaps even while the data is streaming in from the database server, in case it's a longer list, and in case there is a database server, Marco uses a memory-table for his example.

What I find important is that there's less going on between the native compiled logic and getting the data to the user launching a request. Not only a templating engine looks superfluous, this entire ORM thing is something I don't get. If it's a gigantic database model with so much tables that you clearly benefit from code-completion, then I agree, but I haven't come across something remotely close to that in web projects.

Also the HTTP-server itself is something I think that values extra attention. I've seen platforms and frameworks that offer you a wealth of capabilities and features, but hastily slapped on something that listens on TCP port for basic HTTP requests, in some cases on port 80, but more often 8080 or something else in the thousands. In real web environments, the server(s) has/have a lot more going on: load-balancing, reverse proxies, firewalling, authentication. Since we're in a Post-Snowden-era nowadays, we're all responsible to think about protecting privacy and get that HTTPS in order with the proper certification and encryption... Not to mention HTTP version 2 that's heading full steam towards being generally accepted/expected.

I can image the web-admin responsible for all that, isn't happy with your request to add this newfangled separate thing that's doing its own handling of HTTP requests. ISAPI DLL's or Apache modules play much nicer with existing IIS or Apache installations. (FastCGI is on the table, but for now xxm has SCGI available for other servers.) Even if your 'Delphi HTTP framework' of choice is specifically designed to tap your ORM of choice and offer a REST-API for your data-layer needs, it will still be one more stop along the way between the user's browser, and the delivery-setup, and the front-end, and the page-template, and the data-layer, and the database, and what the user actually needs or wants. I think of this in the postal office when there's twelve people in the queue in front of me.

I don't expect to convince much people of this way of working, but it works great for me. I remember the days with early PHP and ASP and how simple and straight-forward everything was. Knowing these work on scripting engines, I kept worrying about lost performance. This was the core reason to start xxm: employ the speed and power of the Delphi compiler to have a native library serve my websites. And it turns out that Delphi code looks quite nice between HTML to handle server-side logic, if I may say so. It took me a few years to make this happen, but I couldn't do without it any more. And people kind-of appreciate that for using this new application, all they need is their trusted browser and a URL.

TMongoWire v1.1, now on jsonDoc!

2017-04-04 19:46  mw_1_1  delphi freeware  [permalink]

→ TMongoWire

This took more effort than I anticipated, but I'm glad I saw it through. A tiny bit of history: when I started creating a Delphi connector to access a MongoDB instance, I had to create the tools to manipulate BSON, and since I really (really!) hate long lists of overloaded methods, I created something based on Variant values. It works great for ADO, so why not. Some time later I had to manipulate JSON, unrelated to MongoDB, so I took what I got, stripped BSON-specifics, into jsonDoc. There it evolved on its own and gained some useful features.

So it was time to re-work TMongoWire, making it use jsonDoc just like any other project. There's still a bit of BSON-specific code, but it deserves a unit of its own.  The downside is there's an extra unit to include in the project, but there were multiple already (jsonDoc, bsonTools, mongoWire and if needed mongoID, mongoAuth3 and mongoStream); the upside is the improved performance and features of jsonDoc, both of the current version and those the future could bring. Enough change to bump that version number:

https://github.com/stijnsanders/TMongoWire v1.1.0

Please let me know if the migration effort this generates is surmountable, and if I can do anything to help. In theory any references to bsonDoc, the BSON function and the IBSONDocument interface should keep working once replaced with references to respectively jsonDoc, JSON and IJSONDocument. The document object no longer implements IPersistStream, but this may have been stretching abstraction just a bit too far. We're on Delphi so plain old TStream should do, and apparently IPersistStream's Seek signature changed a tiny bit over Delphi versions, making it harder to make the code cross-version-ready.

Boxer

2017-03-24 21:57  boxer  delphi werk freeware  [permalink]

Drats. I thought I'll try something like Clover does, start with SetParent on any top-level window you could find with GetAncestor(GetForegroundWindow,GA_ROOT) and see what it gives. There's this old default MDI child mechanic that starts working, so I thought this might just work. But it looks like over at Microsoft the Windows 10 team decided to cut some corners. The theming doesn't do it's modern stuff and reverts to Vista-style borders, and the newfangled calulator, for example, won't even take the SetParent nicely any more. Oh well. For the time I'm still on Windows 7 at work, I guess it'll come in handy there for the time being. Probably in about a week I'll know if this could work where other desktop-management tools failed or were too cumbersome or too invasive...

How does it work? Boxer starts with an empty window, and will show an icon over the top-right corner of other windows. Click that to 'box' the window. The top bar shows a tab for each boxed window. Right-click to open a menu with options to unbox or close, and drag to reorder. Multiple instances play nice with each-other and show 'boxing handles'  next to eachother, the second instance labeled "A", the third "B", etc. To disable the boxing handles, right-click on the top bar to the right of all tabs.

Anyway, if you feel like having a try if you can do better, have a look at the code here. Download a binary here: Boxer.zip (209KB)

A story about two task-keeping-applications.

2017-03-18 17:22  s2tka  coding delphi werk freeware  [permalink]

Let me tell you a story of two task-keeping-applications. Once there was a team that was struggling to keep track of the work it was doing, had done, and still had to do. It was frustrating to work on the team, hard to schedule things, almost impossible to estimate when things where going to get done. And since clients are what they are, changing requirements and additional requests would cause much more disruption than expected of a team that calmly and firmly is determined to deliver the best of market solution.

An internal brainstorming-session about what to do about it, ended in two opposing visions about what decent issue-tracking should be about, and how a system that (co-!)operates with the team members should be structured internally, should behave, and which duties it should perform either by itself, or by effect of using its rules and restrictions correctly as designed.

In an attempt to get to the best solution, two teams were formed to develop each a project based on one of the two views. An evaluation would follow determining a winner, or if would it be possible to synergize the best parts of both into a single system.

A side note: from a managerial point of view this is a very tough decision. Allocating a lot of resources to work on internal structure, takes resources away of the work that brings in revenue, and such undertakings typically risk getting frivolous or spinning out of control. But I guess it's normal that all stop rowing to help keep the boat from sinking. So, apart from being intrinsically aware of the exceptionality of this opportunity, coordinators were given instructions to keep to a strict schedule in these projects and a determined focus on delivering a workable proof-of-concept quick. Sounds like a good work-ethic to apply generally, if you ask me.

A few weeks later progress was made, and the prototypes were already being used to keep track of the issues of these new task-keeping-applications and other projects. The main difference between design visions became apparent soon enough.

One application centered around the list of work items. Care was taken the entry form was extensive enough to have fields for all of the details about a work item, its categorization, its relation to the project, an outline of the projected outcome. An overview would show the current list, possibly filtered for those items assigned to you.

The other application was centered around getting information out of the system, especially structure and relation between items. Users would enter small specific reports, and add them to the best suitable node in a tree-structure, optionally marking relations with other items over branches. The overview showed an expandable structure starting at the root items. It also could have a filter applied, but would potentially show you an entirely different structure of the same data, when using a different relation type.

Having two working prototypes, attention gradually reverted back to the serious work and some of the frustrations of before were abated. After a few months an evaluation was undertaken.

One application had rendered itself useless. The first weeks of usage, a lot of entry happened, but without consensus about categorization, the list was enormous without a clear way of grouping relevant items. Duplicate entries and ambiguous task-descriptions were unresolved, causing confusion.

The other application was doing better. It needed work, but offered a good view of what had to get done.


Enough about the story. Reality, of course, is much more bleak. As no sane manager would make such a shift in resources of a troubled organization, the 'two teams' actually stand for the existing team using an off-the-shelf something badly administered on one side, and on the other side, well, just me, toiling on something potentially better, in my off-time.

I started — like many — with a plain text-file, then a spreadsheet. Then I took the step to design a database for it, but wanted something to do entry and retrieval with roughly the same ease-of-use that a spreadsheet would offer. Working with tree-structures for some other projects, I wanted a single structure to serve as the basis to store data in. Specifically with projects and tasks — as projects tend to have sub-projects, and tasks split into sub-tasks — using branches of a tree would enable to keep this distinction conveniently vague. If you add representation for entities like users and clients into this tree-structure, and relation between nodes over branches, you've got a richness to model much more of the world, and keep track of its changes, past and future.

From there it grew slowly, over years, into what it is now: tx. First versions suffered a notoriously bad interface, but I hope that has improved. For long I was its only user, but some cooperation features have been added since. What's left for me is to keep improving tx where possible, and demonstrating it to people what it's about.

Though entry and structure is important, where a task-keeping-system can really shine is helping to keep an overview. A cleverly designed filter can limit your view to exactly these items that are relevant to you, but in tx you get the additional option of having these items display with exactly those sections of the tree-structure they have in common. Also, I personally feel the most important items on any list, as long as it may be, should be on top. So when tx displays a list of items, they get ordered using their weight, determined by a combination of factors such as task-type and current status. As a task progresses from an active state to a final state, it may move down the list or even fade from view into an archival state.

And there's much, much more I could talk about, but it's all created out of necessity and designed adhering closely to a central vision of what a task-keeping-system should be and what it should be doing for you in order to be able to depend on it.

In conclusion, and for those people that — like me — skip long stories to the last paragraph, it's so very important that information systems succeed at keeping an up-to-date model of reality, and offer you the freedom and easy to update it's view of the world. Especially so with task-keeping software you depend on to keep track of the progress you make with projects, and to help stay on top of what's important. All of that served as the basis for developing tx.

How to import MSXML2_TLB

2017-03-13 15:07  msxml2  delphi  [permalink]

O my god I can't believe this isn't somewhere on the net already! I come across this question and ofcourse there's the default discussion about Indy and OpenSSL/LibreSSL, and Arnaud is very (very!) right about proposing to use WinHTTP. I personally would see if I could do it with MSXML2's XMLHTTP object. Contrary to what it looks like, it doesn't really have to do much about XML, and you can do really almost all of what you need out of a HTTP request with open, setRequestHeader, send and responseText or responseStream, with a much more straight-forward syntax. (If you can live with COM.)

Plus there's another pretty sound reason: back in the days, when using WinHTTP and/or VBScript (for the RegExp component...) you would make it very clear it would only work if you got Internet Explorer version 5.5 or newer installed. "How much back was this?" I hear you ask? It depends, but it's so far back that pretty much all of XML things came after, and XML being to closely related to data-processing, Microsoft needed to put MSXML*.dll deeper into the system than its web-browser. If it's tied to the MDAC or the actual core system I'm, not sure, but years of use and the success of AJAX caused it to get very many security updates. I suspect a far bit more than the DLL that handles your WinHTTP calls...

So, without further ado, do as I do and import the "Microsoft XML" type library in the most recent version you can find on your system:

On old Delphi versions (e.g. the venerable version 7) open 'Projects' on the main menu, then 'Import Type Library...', then select "Microsoft XML, v?.0 (Version ?.0)" from the list. Your system may have multiple versions installed, pick the highest version number (in my case 6.0). Uncheck "Generate Component Wrapper" as usually you don't need this, and actually have the added benefit of automatic reference counting when you use interface references to keep track of COM object instances.

On newer Delphi versions, I've noticed the import functionality has centralised, and is now under the 'Components' menu, the 'Import Component...' option and you need to select "Import a Type Library" from the first list, and then search for "Microsoft XML".

This will result in a automatically generated MSXML2_TLB.pas that you otherwise don't need to worry about. Except perhaps include it in all of your projects you'd launch HTTP requests from.
Use the CoXMLHTTP.Create class function to get a new XMLHTTP instance, and it's open and send methods to make magic happen.

uses MSXML2_TLB;

procedure TForm1.Button1Click(Sender: TObject);
var
r:XMLHTTP;
begin
r:=CoXMLHTTP.Create;
r.open('GET','http://www.google.com/search?q=hello+world',false,EmptyParam,EmptyParam);
r.send(EmptyParam);
Memo1.Text:=r.responseText;
end;

There's more ofcourse. This is just the basics. Depending on what you need to do there's HTTP POST and ways to correctly encode values into the request body you pass to send, using setRequestHeader. And if you need to handle larger chuncks of data, you should check out responseBody and perhaps TOleStream declared in AxCtrls.pas.

jsonDoc, jsonV 1.1.0

2017-03-09 01:22  jsonDoc110  delphi freeware  [permalink]

jsonDoc, jsonV v1.1.0

While working on TRethinkDB, I noticed something was wrong with document re-use. My original idea was to re-use a single IJSONDocument instance to process a list of similar documents, but keep the set of keys allocated, only overwriting the values from the new document. (If you're interested about the code, search jsonDoc.pas for FLoadIndex.) Nothing an extra internal-use interface can't fix. Then I noticed that IJSONDocument.ToString could do with a revision because it wasn't using the new IJSONEnumerator, though it should.

Then I noticed the feature I once added to function JSON, where you could declare embedded documents by listing a value of '[', and closing the document with a key of ']'.  The problem is that '[' could be a perfectly valid value! If it were to come via a variable, that would change the behaviour of the JSON call, and that's a big no-no. Also square brackets usually mean arrays in JSON-land,  so it's perhaps confusing that I chose square brackets because you already need them in the JSON([...]) call. So I changed it into a key suffix of '{', and a closing key of '}', counting on braces being really abnormal in key names. If you really need a key name that ends with '{', don't use the JSON function, but just d['x{']:='y';

A minor downside may be that JSON calls with braces in the string constants, break the comment block if you want to comment out a section of code with braces, but there's always (* *) or //. It by far doesn't outweigh how elegant this new solution is, but it's a breaking change (existing code using '[' values won't work correctly any more), so instead of version 1.0.6 I think it deserves a jump in minor version number to 1.1.0.

While I was at it I fixed some minor issues with jsonV, so remember to update that one as well. Hope you enjoy the changes. (Next up may be replacing TMongoWire's bsonDoc with jsonDoc...)

TRethinkDB

2017-03-06 23:03  TRethinkDB  delphi freeware  [permalink]

Delphi RethinkDB driver

I've given it another try and, in part thanks to a Windows executable that performs well,  and thanks to the gentle people over at the rethinkdb-dev group, I've got something working. It appears the basic functions you would expect from a driver work, but more advanced queries need some more extensive testing. (I've read about a driver testing harnass, but to replicate that around my driver, sounds like a separate project in its own.)

There are some strange sections in the design of ReSQL that show it was written for a weakly-typed language, but using variants and interfaces, I hope I've struck a nice balance between versatility and still having the best suitable methods showing up on auto-completion. Something that should be there may be hidden from view this way, but will probably be because of inaccuracy or error of my part.

Which leaves my in roughly the exact same position as I was closely after completing TMongoWire: having success learning about a new(ish) NoSQL DB the hard way by trying to connect to it directly with Delphi, but nothing in the way of an effective application that would use it for something vaguely useful. I've learned some people have effectively created things with TMongoWire (yey!) but I myself don't really have a good idea (or the motivation) to build something on its own that would use the features of these new database services to their best intent.

If you've got a good idea, please let me know. If you want to know more about TRethinkDB, let me know.

Am I an "old hand"?

2017-01-18 09:23  oldhand  coding dagboek delphi  [permalink]

Am I becoming an 'old hand'? Every time I see someone that doesn't have done as many years of work in Pascal, ask something about generics, I'm troubled again by this generics thing. I don't use it (much). I don't need it. I've been perfectly fine all of this work without them. Yes there is some hype nowadays about functional programming, but there's a grand glacial movement at the base of it: stong type systems. It all started even on the big metalwhere some bright minds had the sense of taking writing logic into a domain of itself with a gentle nod to mathematics, statistics and other adjacent domains. And a lot of great work was done in this domain. A lot. Some great names that now stand on their own came forth.

A recurring theme throughout of it is type systems, or the lack thereof. Or having it bolted on at a later stage. So by the time K&R came along, they took what they needed, what they could make work for them, what they needed to summon the Unix spirit from the depths. Pascal existed at the time, but C, like JavaScript much later, was born out of necessity. Created by the people there and then that needed a new wrench to tighten the new bolts of a new machine. With personal selections here and there, and careful considerations that copy a bit of the zeitgeist of the time.

What's left for us to do is try not to repeat ourselves too much. One way is keep a look-out for patterns and in new projects, know when to apply which one. Or even better, when to switch to another one. This, I think, is where it's starting to get really tough for the younger generation. There's a lot of base-knowledge to cover, some of it is superceded or obsolete, but we can't replace it with studying patterns. I've seen newbies take it on as lawas if it's forbidden to deviate from the design even just a little. Attacking establishments on the base of their deviations from a pattern. There's something to say about attacking the establishment, but blind fury will get you nowhere.

So this generics thing...  It also is in danger of being percieved as a rule, getting over-applied. As if the one true way is using it everywhere or as often as possible. Which is strange from where I stand, since I still see it as a convenience, where you can save on writing bits of code that would otherwise be painfully similar anyway. In operation the work would get handled by this extension to the type system, checking things for you in the background, making sure everything fits. But in the end the things that need to happen, occur much in the same way as if you would have written all those specialisations out in full.

That's why, as it turns out, I don't use them all that much. Perhaps it's also because of my fading belief in the object-oriented way. That too was presented some years ago as the way to go. Everything was supposed to be objects living on the internet. But it too kind of degraded in my view to a convenience, only one of the ways to manage the things inside of your program. Luckily before any of the ORM craze could tinge me. It makes you look around the horizon, in the past where you did more with arrays and pointers, and sideways where others pass messages around all the time. For now I feel like I'm stuck here, waiting for the ground to crack open and offer a new middle ground between interfaces, pointers, traits, channels, monads, and all of the above.

But that's just my humble opinion. I can't tell for sure what the next 60 years of programming machines will bring us.

TIL: 'method modifiers' are not reserved words.

2017-01-05 21:17  TILdirrw  coding delphi  [permalink]

Thing I learned today: 'method modifiers' are not reserved words. Apparently the Object Pascal parser 'consumes' these as labels that just happen to be there or not. Which means in other contexts they're free to use as identifiers, and so they're not reserved words... Strange but OK I guess as this keeps the tokenizer that does its job right before the parser, a little smaller... (How I stumbled upon this is a different story altogether, but one I may be sharing sometime soon...) I searched the documentation a bit and there they're called directives: Fundamental Syntactic Elements: Directives which I thought were things like {$XXX+}, but those are compiler directives...

So these variable names are accepted by the parser/compiler (it doesn't mind the smell at all):

procedure test;
var
stdcall,safecall,cdecl,overload,dynamic,register,abstract,reintroduce:integer;
begin
stdcall:=1;
safecall:=11;
cdecl:=111;
//...
end;

DirFind v2.0.4.494

2016-12-16 22:30  DirFind494  delphi freeware  [permalink]

DirFind

version 2.0.4.494


E-mail: Intensive Delphi 2016 and DMongoBR

2016-12-13 10:48  20161211d  delphi  [permalink]

I recently received an e-mail from Brazil where an event is about to take place: IntenseDelphi and one of the speakers will present a suite of components for MongoDB based in part on TMongoWire. As a starter I decided to respond with a write-up of a bit of history and background story of why TMongoWire came to be. It touches on a lot of other things as well, so I've made a slightly revised version to put on my website:
 
I can give you some background on TMongoWire, and if you want to know more, please ask.
 
First a bit about me, a little CV:
I started at a young age with a little BASIC on a CP/M (on a Kaypro II!),
later GW-BASIC but soon enough Turbo Pascal versions 3, 5 and 7. Almost by chance, as my father bought Turbo Pascal and Turbo C++ on a whim without really knowing what it was. My brother who is 4 years older also did some Pascal, but picked-up on C++ and later did heavy work in assembler as well.
Later, when I studied for what is now equivalent to Bachelor Computer Science, I was introduced to Delphi 3. Again almost by chance as someone I showed old Pascal DOS programs to, told me there was something new for Windows. After that I moved to Delphi versions 5 and then 7.
By then I attained my degree and started with a web-development firm. I first did PHP and later Cold Fusion, but did a lot of work of my own on Delphi-projects as a hobby.
But as hobbies go I did not put any money in new Delphi versions. Also I did not know any other Delphi developers professionally. I later heard about the schism in Delphi-world as the first versions after Delphi 7 had put off a lot of people because the IDE and editor was less good.
So I'm afraid I'm also one of those people that sticks with Delphi 7, but in part by consequence and only in part by choice. Still, thanks to the careful design of the Object Pascal language, code written for/with Delphi 7 still works really well in modern Delphi versions (just with taking care about a few little things like using AnsiString/WideString where possible.)
 
A funny fact is that I only learned about SQL and databases later on, around when I started to work on websites. I have done a lot of work in Delphi and not used the data-aware components! Also by doing PHP, ASP and Cold Fusion and developing websites without an IDE (just a source editor), I really disliked the different options you had to develop websites in Delphi. I'm conviced the IDE/RAD component eco-sphere is toxic to the performance-centric build-up of a website and will always make important choices for you that will cause problems later on when the project gets bigger.
 
So I started xxm. The Delphi compiler is so fast, it is possible to have a DLL that runs a website, when source-files change and you hit Refresh, to unload the DLL, compile a new one, load it and have it serve the page with updated logic. This way, developing a website without IDE or components would be almost the same as PHP and Cold Fusion, except in Object Pascal, and the website would perform as fast as a native service.
 
Programming a website like this reduces the work to creating a stream of HTML, and any visual components you would use offer extra overhead which takes a bit of your performance away. So that's where my search for 'really thin wrappers' comes from. When using Microsoft Access MDB's at first, and later MS SQL Server, Microsoft already developed the ADO components to do the heavy lifting, and thanks to the integration of ActiveX interfaces (IUnknown) and the type library importer (to generate ADODB_TLB.pas and others like MSXML2_TLB.pas) it's easy to use the ADO components directly (mainly Connection, Command and RecordSet, see xxmData.pas).
 
Later I also checked out other databases, but was displeased with the performance of the ODBC connector to SQLite databases. Other existing Delphi SQLite connectors were for modern Delphi versions and had long lists of 'overload' properties, which I really dislike. Using ADO much of the data-transport is based on OleVariant and I like that. Then I discovered the C API to SQLite is relatively straight-forward, and created TSQLite. Also PostgreSQL and MySQL/MariaDB have a C API DLL ready to use to connect almost directly. Since their SQL dialect is very similar, it should be easy to switch between the two (e.g. start a project off on SQLite, and later switch to PostgreSQL to serve a growing user base) so I created DataLank to take a lot of the work of switching away.
 
After that I also wanted to check out some of the new 'NoSQL' databases. I had no experience, and thought if I can write a connector I would understand more about how these databases worked.
With Redis I learned it had a human readable protocol, but was very basic and very much removed from the SQL world.
With CouchDB I learned some 'NoSQL' databases concentrate on storing (JSON) documents and can perhaps do more server-side but can do it fully separate from the front-end.
I tried RethinkDB several times: first I learned about Protocol Buffers and made Delphi support myself because I couldn't find any. By the time I got that working (remember, I'm a hobbyist programmer that only puts a few hours a week in my own projects), RethinkDB was moving away from 'ql2.proto' but was not ready with their new binary-level protocol. And the next time I checked, they were in some kind of trouble.
With MongoDB, I think a nice balance exists between storage of documents, and other features you need to do specific manipulations or queries. I was lucky to find my way through the layers of documentation to what I needed to know about the binary protcol. First BSON and then the wire protocol, both show careful design and are not needlessly complicated.
 
There are a few more on my to-do-list, but I kind of lost interest. I'm using TSQLite the most for the moment, and will probably start on a new connector if I'm sure there's the drive for it from a new project that's bound to have to run on this or that newfangled database. One big change I consider making somewhat soon, is reworking everything BSON but based on my own JSON parser I created because I needed it for a project that otherwise didn't need the bits specific for BSON...

Another Multi-User Dungeon

2016-12-08 23:15  amud  coding delphi freeware  [permalink]

I wanted to make an extra demonstration of WebSockets from/on/with an xxm project. What I also have dreamed about is making a multi-user game where players would navigate a virtual realm and manipulate the objects in it. Problem is that I don't have too much experience with playing existing multi-player online games, and that animated graphics design is not one of my strong points. But still, I wondered if combining the two would lead to something that somehow works.

So I started a project and set out to get something working, without losing too much effort on anything non-essential. Not even on a name for it, so I called it "Another Multi-User Dungeon". I've put the source up on github and host a running version on this computer at home I use to run my xxm projects (so don't shoot me if it's not available, it's not an always-on full-fledged web-server, but it also shows xxm projects don't ask much of a machine to run stabily). Feel free to have a try:
yoy.be/home/amud

The view shows, from top to bottom:

Anything you type into the entry box, when you press enter, is passed on to your in-game persona to speak into the room it is in. Click on items or people to get a list of actions you can do with it or them. Some actions use the last statement that was spoken by your in-game persona, for example 'make a note' from the NoteBloc object (ask for a NoteBloc from the hotel's receptionist). Click on an item again to select it, and in some cases extra actions are available on other items, for example the 'give' action to pass something from your inventory to someone else in the room. Most 'door' objects have a 'go' action that will move your in-game persona to a different room, unless you don't have the key to its lock or the room is fully occupied.

For now I haven't put too much up into the virtual realm. By default you enter the world in the first free room of the Sunburst Hotel, and apart of a welcome leaflet there's not much there. I thought a decent virtual realm would have computer-operated agents you could interact with, so I remembered ELIZA. A lot has changed in the world of chat-bots since then, but I thought it would be nice to have just that in there. It turns out the syntax of the bot-script at its base serves nicely for other roles as well, so I fashioned a receptionist for the hotel that answers back to some questions (and for now can provide you with a NoteBloc if you ask politely). At the city hall, there's a registry office where you can change your display name, and perhaps leave your e-mail address into the internal database, I might make it required for some operations later or if someone wants system support. There's also money you can pick-up and drop, but nothing else you can do with it (yet!).

Under the hood, it's heavily based on a single WebSocket, your 'feed' through which you get information about the virtual realm, and through which you can send commands back. The set of commands is limited and any extra requests lauched by the client-side script use a personalized single-use key based on your personal authentication key stored in LocalStorage. (I could have gone with a cookie, but if the WebSocket were hosted over TLS, LocalStorage offers slightly better security.)

I'm not sure where to take it from here. I've been thinking about creating a shop you could use money to buy things, but then you'd need something you can earn money with... And transporters that could move you to locations further away. But I guess designing more of a city will take some effort already... Perhaps I've got what I wanted: to create a platform that has the basics to build a game on. Only the idea for a specific game isn't there yet. And thinking up a thrilling plot of an exploratory adventure isn't one of my strong points either. If you have ideas please let me know in an issue on GitHub. Perhaps I could hand out a few RoomMaker objects to people and see what they do with it...

DirDiff v2.0.0.460

2016-11-13 21:04  dirdiff2  coding delphi freeware  [permalink]

DirDiff

It feels like this was a very long time in the making, but all the little bits of time here and there probably still amount to a recent number of man-hours... It took a couple of attempts to get "An O(ND) Difference Algorithm and Its Variations" by Eugene W. Myers in an implementation of my own that performed to my liking. I've chosen to use xxHash to speed things up. Once I got that, I continued the grand re-work of DirDiff so it would accept, not 2 files, but n files (or folders); handle the work in background threads, and have both the folder-overview (and XML three) and content in the same window. In case anyone would like to have a peek inside, I've decided to open-source it as well, under the MIT license on github

I've open-sourced my productivity tools

2016-11-12 16:49  tools  coding delphi freeware  [permalink]

I've finally decided to open-source a set of my home-made tools, some of which I use almost every (working) day. Some may be very taylored to my personal taste, others may be easier to use by the broader public. Most are missing documentation, but the typical operation should be self-explanatory. (Except for handy hidden features, should make a list of those somehwere sometime soon...)

These tools have been available here for download in binary form since long, but by putting the source code in a public repository I hope I can inspire anyone that would like to know more what's going on behind the scenes to have a look, and who knows perhaps get someone to make improvements or additions.

→ github.com/stijnsanders/tools

RethinkDB: here to stay or on the way out

2016-10-07 23:23  RethinkDB-in-or-out  coding delphi freeware  [permalink]

I'm worried and confused.  The news these days is that RethinkDB is shutting down, at least the company.  The database itself may live on.  I had been attempting to write a really-thin Delphi wrapper for RethinkDB a number of times before, but got diverted into writing DelphiProtocolBuffer first, because it was used for the base of the server protocol and I couldn't find the Delphi support I was looking for (I'm sorry, but I'm a bit picky). Later it looked like they were planning on dropping this in favor of something else, but I wasn't able to investigate further at that time. In the mean time, I did a CouchDB, PostgreSQL and a MariaDB/MySQL wrapper, and a neat way to tie them together, and kept TMongoWire working, which turns out to be the one in the series of DB connectors that has the most success to date. (And which envolved learning all about PBKDF2...)

So now I'm confused about wether to keep RethinkDB on my to-do list? Or drop it? Or move it up to have a look sooner... Just like MongoDB, it may be that I won't really have a full-grown project anytime soon that is based on RethinkDB, but demand for a thin Delphi wrapper is out there and makes it worth it, I don't know. (Please let me know.)

And I'm worried about this new tale of an open-source-but-also-corporate endeavour going sour. It's only natural that when the projected sales don't appear to manifest, a project is best to get ended cleanly (as I recently witnessed over at the day-job). So I'm rooting for the open-source project to find a good new home. That the eco-system of volunteers may survive the orphanization. That the real-world users may carry enough weight to force the project into a viable after-life.

It's important to have a good climate to have entrepreneurs get enticed to take the step and start a company, but with this series of high-profile undertakings going bust, and aqui-hires where users get left in the dark, it starting to look like it's running out of control. Some people may called it a bubble, but I find the image of a pendulum swinging more appropriate. Is it swinging back already? I'm in no position to tell accurately. But I am worried.

For the record, I have had no economic gain from my open-source undertakings what-so-ever (up till now?) though OpenHub estimates about half a million went into xxm up till now. Not bad for a hobby...
But about RethinkDB, I guess I'll have to wait a little until the dust settles down...

Update 2017-02-06: RethinkDB's assets have been bought from RethinkDB Inc. By something something Linux Foundation, and a re-license to the Apache Public License. Sky is clearing up.

Update 2017-03-06: I've given it another try, and partly because there's a downloadable Windows executable that works, I've got something working. TRethinkDB appears to be able to perform all the basic functions you could require from a driver, but may need some more testing.

xxHash: an extremely fast non-cryptographic hash algorithm

2016-09-23 23:50  xxHash  delphi freeware  [permalink]

md5

I'm slowly but surely (finally!) working towards a (long overdue!) rewrite of DirDiff, but this time using threads and perhaps a different algorithm. Apparently there's still progress to make there, and I read good things about the diff git is using internally. Looking into that, some performance can be gained by doing the actual comparing on hashes of the data, instead of all of the data itself. Which lead me to xxHash,  which should hit the sweet spot between fast enough and safe enough against collisions. (Unless I misunderstood.) I'm not sure if anyone thought of combining those two. But I may be looking into that for DirDiff 2... But since I didn't find a Delphi implementation rightaway, here it is, and it fits nicely with this collection of other hashes I did before. (Though xxHash is specifically a non-cryptographic hash...)

Delphi and MySQL or MariaDB

2016-09-15 21:12  libmysql  coding delphi freeware  [permalink]

Tadaa! I added a wrapper around libmysql.dll (to connect with MySQL or MariaDB) to my collection of really (really!) thin wrappers around things like ADO, or LibPQ (to connect with PostgreSQL) or now libmysql.dll. I also had the idea to align them as much as possible into almost the same interface. It's specifically not my point to have them be exactly the same, but very much almost so, just in case when I use this for a project and need to switch database back-end later on, the work that goes into that is minimalised and can concentrate around the different in SQL dialects. (for example the postgres branch here)

https://github.com/stijnsanders/DataLank/blob/master/MyData.pas 

(previously)

AES, DES (TripleDES)

2016-09-09 23:08  aesdes  coding delphi freeware  [permalink]

→ md5

I've added AES and DES. DES may be deprecated, and Triple-DES may be soon, in any way it's cryptographically superceded by other ciphers, but still in use by some systems. I should do some work extra and change array[0..7] of byte into record l,r:cardinal; end;  or even int64 but I needed it only for something small and this works, so I'll leave it at that for now.

jsonV 1.0.3

2016-07-08 22:52  jsonv103  delphi freeware  [permalink]

jsonV

Update: v1.0.3
jsonV now shows more sensible type labels, and only the fout-digit-hex-internal-value only on the most exotic variant types.
Ctrl+Shift+C now copies the value only, not just the node caption.
I also discovered a blocking issue when jsonDoc was trying to construct an array of int64 values. (more about that here)

jsonV 1.0.2

2016-06-16 17:40  jsonv102  delphi freeware  [permalink]

jsonV

Update: v1.0.2
jsonV now accepts json-files that contain an array at the root level.
I also changed jsonDoc so all OleVariant arguments are passed const. This may break older versions, but should offer a little improvement in performance.

Just another debugging story from the trenches

2016-05-27 23:23  a-debug-story  delphi  [permalink]

It's all logic, really. Once you get to the bottom of it. But getting there... Phew!

So I was working on something that had a line like this:

function TPostgresConnection.Insert(const TableName: UTF8String;
const Values: array of OleVariant; const PKFieldName: UTF8String=''): int64;

I forgot what went wrong in the beginning (it was somewhere yesterday between midnight and bedtime...) but to check something out, to debug without a debugger so to speak, I wanted to dump a string. Deep within the data-layer you can't really access the things that handle the output to the user, so I just do this:

  raise Exception.Create('debug:'+sql1);

And then run the application again and reproduce the issue you're tending to. But it doesn't throw this exception,  it throws a quite severe access violation instead, strange. Not the error it started with, but none the less something strange going on in the exact same bit of code you're working on. But it was running late and I felt ready to go to sleep, so that is where I left it. It's always nicer to leave a project in working order at the end if the working day, but problems like this typically look different when viewed with a freshly rested set of eyes.

So the next day, let's move the debug exception somewhere higher up in the function body. Actually, all that happens before is handing the values in the Values array... So instead, I put this in near the top:

  sql1:='';
l:=Length(Values);
for i:=0 to l-1 do sql1:=sql1+','+IntToHex(VarType(Values[i]),4);
raise Exception.Create(sql1);

Which gives me:

,0000,20C8,0015,FD18,46A4,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000

Hmm, so it looks like the list of VarType's of the Values array, but these values are totally bogus! The VarType constants are (at least in older Delphi versions) right there at the top of System.pas (the unit that's always included, even if you don't list in a uses-clause), $0000 is varEmpty, but $FD18 is impossible since only varArray = $2000 and varByRef = $4000 are used of the highest 4 bits... And I'm sure I didn't pass any empties in. What is this?! Time to break out the real debugger then, I guess.

An xxm project pre-compiles files that have HTML and server-side pascal in the same source-files, into a full-fledged set of .pas files and a .dpr file, have that compiled, and then hot-swaps this new DLL in place to handle the next web-requests. So the usual development on an xxm project doesn't involve the Delphi IDE (I actually use atom.io since a while now.) but since the source is right there, you can open it in the plain old Delphi IDE and have the debugger start the DLL using something like xxmHttp.exe (the xxm handler that doesn't auto-recompile like xxmHttpDev.exe, since that would interfere with debugging.)

So, let's see what the debugger makes of this Values array the moment of this debug execption:

Values: (Unassigned, Variant array of Unknown, Unknown type: 21, Variant array of Unknown, Variant array of Unknown, Unassigned, Delphi exception EAccessViolation at $...

Yikes. But good news since with debugger it is doing the exact same thing wrong. That's something you can't take for granted, but that's another story.

So setting a breakpoint at this specific call to Insert, to see if the values get passed in:

Hey! Those are the correct values. But just one press of the F7 key later:

Bam, they're gone. Hmm, switching on the compiler "Use Debug DCUs" setting and retrying this doesn't bring me into any underlying code (sometimes some behind-the-scenes work is done on variants and strings for you), so nothing there...

And then it dawned on me! I know of one thing that can 'eat' your memory like that and it's compiler optimization.

By putting in that 'debug raise', using only an unrelated string, the optimization might conclude it can get away with releasing the memory of that Values array sooner, apparently disregarding the work that's been done on it. (Newer versions of Delphi may handle this differently, but that's another story...) So having a quick try with this:

  sql1:='';
l:=Length(Values);
for i:=0 to l-1 do sql1:=sql1+','+IntToHex(VarType(Values[i]),4);
if l<1000 then
raise Exception.Create(sql1);

And it works! So that's the weird misbehaving variant array out of the way. Now I feel extra silly not remembering what was originally the thing that I thought went wrong, was I just getting tired? This function is rather new, so I was just trying to debug for the first time, I guess. Still, makes for a nice story about debugging deeper and deeper until hitting the system, then finding out what's causing your trouble, calling on more of your knowledge of that system than usual.

(In case you want to know more about what I was working on in the first place, look here.)

Delphi and PostgreSQL

2016-05-13 00:19  LibPQData  coding delphi freeware  [permalink]

Searching for an alternative way to use PostgreSQL from Delphi? In the spirit of the other as-light-as-it-gets database connector wrappers, I've converted the header(s) to the LibPQ dll. See the LibPQ and LibPQData units in this repository:

https://github.com/stijnsanders/DataLank#datalank 

To start an attempt to move to something more like a generic database layer, I've started this DataLank idea, but it needs more work. Actually I don't intend to develop it all the way into a full-fledged data-layer. Specialities of each specific database solution are so diverse and important, that I really don't want to hide them between an opaque abstraction layer. I want a database connector wrapper to be as light as possible, but also to allow access to the underlying technology. The main point for using DataLank, is to have less work changing DB's by using TDataConnection and TQueryResult types, but not no work. Adapting the used SQL to a different dialect will probably involve updating many (or all) calls the database anyway. But I'll post about that later when I put some more time in. (And perhaps write a libmysql.dll wrapper first...)

 

Archive... Search...