Ask HN: What is the SQLite of nosql databases?
2021-06-13 10:33 hnsqlitejson [permalink]
→ Ask HN: What is the SQLite of nosql databases?
Though I'm very much in agreement with:
The SQLite of NoSQL is still SQLite: https://www.sqlite.org/json1.html
there are situations where you could just use a single JSON object to store everything?
If your JSON implementation of choice does it correctly, and I attempted to get this right in my implementation, looking up a key from the list of key-value-pair should use a method that wastes no time and gets you to the value you want as swiftly as possible, no matter how many keys you need on the first level.
Then again, if you decide to use SQLite, and really need almost nothing else, I've created this one...
Quick, write it down: just another CPU design idea
2021-05-05 23:57 cpuidea [permalink]
Ever since learning about the Mill architecture, and watching an excellent series of MIT 6.004 lectures by Chris Terman (I can't find them on Youtube any more! But there are newer versions of 6.004 online which probably are great as well.) and reading up on what's new about x64 and RISC-V and ARM, from time to time my mind wanders if you could go crazy and design yet another CPU design that does novel things with the umptillions of available transistors and not have the downsides of the currently popular CPU's.
So here's an idea I just need to write out of my system, and let you have a look so you can see if there's anything there at all. Modern CPU's use virtual registers to have hyperthreading and speculative execution, and push whatever ahead in the hope it was the right branch to make pipelining work better for the workload at hand. The Mill architecture would handle that a little different and fill the pipeline just with more concurrent threads and interleave those. (At least that's how I understood what would be going on.)
What if you design a CPU so that it also takes in a larger set of streams of instructions, also using a bank of virtual registers that get alotted to these streams, but also have hot memory just like the closest cache is, but use that for the stack. Ideally you could expect programs to keep the current stack-frame fit within one or a few kilobytes, and by having a larger number of streams you could avoid having to switch this stack along with all the rest on context changes (by the OS). Switching the stack and overflowing the stack to and from memory is something that will have to happen, so you'll have decent support for that, but if you're after speed you'll try to avoid it as much as possible.
Let's have some numbers just to get a clearer mental picture. Let's say a core has support for 256 streams of instructions. If the work for them is pipe-lines across the rest of the core, addressing them would take 8 bits, so you would see a band of 8 lines pipe-lined to everything everywhere. It's not even required to have 256 instruction decoders, and sub-groups of streams could share commen instruction decoders, but depending on the instruction set design itself, this could turn out to be the bottleneck, but let's find that out with further design and research.
Let's say each stream gets 4KiB of hot stack memory, in total is 1MiB which is not unrealistic to have in a modern code nowadays, if I understood that correctly. This special memory could have extra lines so it automatically flows over into system's memory if the local stack index rolls over, which perhaps could also help with loading and flushing stack data on context changes.
With instruction fetching, stack handling and these streams doing their thing with alotted virtual registers, there's a lot covered, but the ofcourse the sweet magic happens in ALU's and related things, so down the pipe-line, according to what instruction decoding prescribes, the streams queue up to get one of the available ALU's to handle an operation.
This would be a point at which you start on a real design in a simulator, but that's where my knowledge is stronger than my experience. I just decide to start writing things down (right here, right now), let it simmer a bit and maybe pickup starting a first attempt sometime later. If you see what a former attempt like plasm ("play assembler") looks, don't get your hopes up too much. For now it's good that I wrote the core idea down here. If I ever get down to it again and get something working, you'll read it here. Sometime later.
Do n.GetFirstChild=nil instead of n.Count=0
2021-03-21 17:25 nodecount0 [permalink]
I wish I knew this one sooner! If you've worked with TTreeView before, and know about TTreeNode's HasChildren value, you need to check with OnExpanding, if you need to load the children or not. So I would typically have something like this:
if Node.HasChildren and (Node.Count=0) then //load children nodes
Only now after all these years I happen to come past this in the Vcl.ComCtrls.pas unit:
function TTreeNode.GetCount: Integer;
Result := 0;
Node := GetFirstChild;
while Node <> nil do
Node := Node.GetNextChild(Node);
So I've been throwing away performance all this time! Because my 'are the children loaded yet' check was taking more time if they were and there were a lot of them, expanding nodes was slower. And I never noticed!
So it's a good thing I know this, and now I can just write:
if Node.HasChildren and (Node.GetFirstChild=nil) then //load children
205B random strings and no 'Delphi'
2021-02-28 18:48 randomdelphi [permalink]
→ Delphi-PRAXiS: Can Delphi randomize string 'Delphi'?
Ah, that takes me back. A while ago at work I got someone baffled by this statement: It is said that a thousand monkeys banging away on keyboards could at some moment in infinite time produce the complete work of Shakespeare (and that the combined internet forums are a living counter-proof). Now if you search online you can find an XML download of the combined works of Shakespeare, so it's not hard to find the relative occurance of each letter of the alphabet. One can guess this will have rougly the same values per letter as the total of the English language. So, then, if you take 'random' by its definition, and the monkey's produce text at 1/26 probability for each letter in the alphabet, therefore they'll never reach a point in time where they (re)produce the works Shakespeare.
Now, I'm not a real philosopher, or a statistician, so my thesis could be complete fiction and based on nothing, but sometimes you really got to take a lesson from practice. In theory, in infinite time, it's ofcourse possible that something really really inherently possible could emerge out of a random system, but there are characteristigs to anything random, and there are exponential things at play that soon but experimantal set-ups like with the link above, that they probably would produce the expected outcome in a time-span that exceeds the number of years we've got left before the sun sheds its outer layers and devours the earth, complete with a set of silicon-based machines churning away at putting a series of random characters in sequence and comparing them of some neat stories of a long gone English playwright.
DIY Google Authenticator
2021-02-25 20:12 diytotp [permalink]
Recently, I've got a few things asking to enable two-factor-authentication, and I started using the Google Authenticator app.
I kind of like it. It's a simple enough app, there's a shared secret involved, but it gets pretty close to being airgapped and perfectly forward secure and all of those things. So I got thinking... What would it take to start using it for myself, in those little software things I create now and then...
Is there black magic or stick whittling involved? Nah, a little searching around, and it all appears to be cleanly described in RFC's 6287 and 4226... There has to be a warning here about not rolling your own crypto, but the world of hashing and encrypting really is interesting! I did SHA1 and HMAC before, and Unix' time apparently is UTC... So all you need* is the correct format of URL to put into a QR-code to load up a new key in the app. Then you can use this code to generate the 'current' pass-code for the secret:
*: and apparently base32-encoding, HashUtils was missing that...
Lobsters on Delphi: the good and the bad
2021-02-18 00:20 lobsters [permalink]
→ Lobsters: 26 Years... of Delphi
Aaw, look at that, the good and the bad on a single page.
QuickReports: R.I.P. Lut
2020-12-18 16:26 invoicetemplates [permalink]
Oh my, I didn't know. I guess we'll see this more and more with one-person open-source projects, that people stop with the project for all kinds of reasons.
But about reports, I've made this really elegant thing at work to script the design of a report with just a few basic commands (line, text, image, matrix, block, repeat...) that allows you to write what the report should look like, and make lots of small alterations later. If you really need your reports just right, I've always found a graphical designer to introduce minor artefacts that you sometimes need to work around...
I know there are tons of similar things out there, like TeX and PostScript, and even HTML or MetaFiles, but it was grown out of neccessity and suffered "dogfooding" from the very start, which shows in its design and execution. And thanks to SynPDF, (or "Print to PDF" for that matter) you can just exports PDF's with it as well. So I really should take some time and re-do it on my own time and open source it... If I can make it even better and cleaner, I might introduce the open-source version as a replacement at work! (... Oh-oh!) (... or was it this one?)
Could you do a mill architecture with RISC-V?
2020-09-29 22:26 millrisc [permalink]
I wondered today if it would be feasable to build a processing unit according to the mill architecture with the RISC-V instruction set. But as always, I also wonder if I'm out of my league and know far too little of these things to come even close to forming an accurate answer, or even if I'm anywhere close to a good idea. If I understood correctly, the mill architecture does what modern CPU's already do: use a bank of virtual registers with multiple heavily pipelined streams of translated instructions. CPU's that do hyperthreading have two streams of instructions, but that's about it. I'm really rooting for the first article I read about a real system that has more than two, but with just a little imagination, and if you know the 'price per transistor' keeps going down, you can image a huge bank of virtual registers, with a bank of ALU's close, that processes a 'large' number of instruction streams. Ideally even a totally variant number of instruction streams. This may be a caching nightmare, but instead of wasting a lot of in-system logic to do branch prediction and other bookkeepingof what an instruction stream is doing, what if you could just load another instruction stream, just up until you get all of the piplined stages fully occupied...
Alas, I'm so much out of that business that I guess I need to conclude I'm a wild dreamer. I may stumble on a good idea now and then, but know too little to be sure. So I'll note it down here. I hope I haven't bored you, and with just a little luck, you are able to judge about this idea. If so let me know...
MEAN Stack Considered Harmful
2020-09-23 10:31 meanbad [permalink]
→ MEAN Stack Considered Harmful
An important read, also reminds me of what I've written here.
Carefull with Gogole Sheet CSV export
2020-06-26 14:18 ggggrgviz [permalink]
Ready for another story from the trenches? So image a Google Sheet made by someone else, with all kinds of dat in about 30 columns, of about a few thousand rows. Yes, it's a stretch to keep using Sheets for that, but this data will serve for the analysis for a decent application to manage this with... That probably won't be my team handling that project, but I had to do a quick cross reference of this data with the data in the database of one our current projects. The best way to do cross-checks is get the sheet into a table in the database to run queries. I guess you should be able to import a CSV pretty easily, right? I searched around and found this:
Which I thought would provide the data in just the way ready for me to import. Wrong. The second column just happened to have codes for all of the items that are numeric for the first few hundreds of items, and then alphanumeric codes. At first I thought the CSV importer was fouling up, but I hadn't looked at the CSV data itself. Turns out this CSV exporter checks the first few lines (or perhaps even only the first one!), guesses the column is numeric, and then just exports an empty value for all non-numeric values in that column!
The code in that column was only in a number of cases needed to uniquely identify the items, so I first was looking for a reason why my cross-match was throwing duplicates in all of the wrong places. Ofcourse. Weep one tear for the time lost, then move on. Take solace in the wisdom gained.
I solved it by using the CSV from the Export menu. I only needed it once so I didn't get a URL for that.
Why no C++ webdev...
2020-06-05 17:23 cppwebdev [permalink]
→ Quora: Why is C++ not used in web development?
For what it's worth, in creating xxm it feels like I'm trying to create this exact thing except for with the Object Pascal language. Yes, you can do HTTP all by yourself, you can do an ISAPI extension DLL or a Apache httpd module, but you'll still get a strange hybrid between a server service and a web application that has nothing like a platform you can depend on to do the heavy lifting. And, if I'm permitted to speak frankly, in C++ this would be ugly! And probably would need a lot of code to make even the basic things happen. Too bad (Object) Pascal has been called verbose, if you know what you're doing you can write the logic you need in concise readable syntax.
Still what I'm finding in trying to get people to take a look at xxm, they either are unable to disregard the visual RAD form-designer style programming like I do, and don't get that xxm in it's current for is much more like early-days PHP but with the Delphi compiler instead of the script interpreter server-side; or they are fixed in thinking 'the web + Delphi' is all about a data-layer, doomed to only serve plain CRUD requests to and from a front-end layer, and never talk to the user's browser directly. Please! A big strong no on both accounts. Let me explain.
I've always seen — pretty much since FrontPage and DreamWeaver — that if you have a visual designer to manage what to go on a HTML page, you get really ugly code. It'll look the way you want, but a lot of decisions have been made for you. Some with negative consequences for you down the road. And the underlying code is strange and ugly, unneccessarily complex for your website-visitors' browser to work with. I guess modern front-end web-devs must have known this also as I've seen a regression towards working on big chunks of raw coding the last decade. Yes, font-ends are hacking away in CSS and HTML, and not with their bare hands, all kinds of CSS pro-processors and template engines do the heavy lifting behind. So if you know what you're doing, you can have this as well, in a Delphi project. I don't need a form designer, I make the page-builder first with dummy data, and run it in the browser. Don't forget, hitting F5 in the browser to an xxm website running over a development-handler, fires up the Delphi compiler there and then. Edit→Save→Refresh→Repeat
Then there's the other thing. If you start a conversation about webservers and Delphi, bam there's DataSnap. Strange. Is it because I'm strongly dys-convinced about ORM's? (Reminder to self: still have to write that grand essay about what's bad about ORM's.) Yeah sure, if you have things that use RTTI to serialize your data-objects in some way, you can easily use one of the available options to serve it over some web-server and bam you can call it REST and get away with it. But this is a completely different thing than having a full blown web-application serve from something you created! Complete with images and stylesheets. And yes you can have both🤯 from the same web-server-service🤯. I've had people walk away, unable to believe me. It still feels like it's a case of opening your mind to be able to see it.
Whaddayaknow, it checks out!
2020-05-27 10:30 primescheck [permalink]
→ J: Doc/Articles/Play104: The 10,000,000,000th Prime Number
http://yoy.be/home/primes/?a=2&x=10000000000&t=1&n=1 p(455052511)=10000000019 is actually the first prime past 10^10, but that does mean my primes thingy found the 455052511 primes lower than that.
Microsoft Teams was down?
2020-03-16 20:29 nottx [permalink]
→ Microsoft Teams goes down just as Europe logs on to work remotely — Verge
Tx wasn't down, especially the instance you host yourself.
About x in ['0'..'9'] and CharInSet
2019-11-05 22:42 anticharinset [permalink]
I'm still in the (slow) process of letting go of Delphi 7 and embracing the modern version(s) of Delphi, which to be honest was only possible since there's a community edition (that includes the stand-alone compiler(s)). I was in doubt if I would move over to the
WideChar based default
string, but since Windows internally switched long ago — and you get emoji-support for free — I thought I should switch also and haven't even tried to override the new default and switch
string to be
AnsiString just to keep things the old way. It works remarkably well. Any code that assumes the length of the string is also the number of bytes to move around needs to change, but that's a good opportunity to change them over to operations that use encoding support. With
char that also changes mapping (to
WideChar respectively) any coding use those just keeps working. Except in code like
x in ['0'..'9'] which I apparently use a lot. It works except when compiling it throws this warning:
Warning: W1050 WideChar reduced to byte char in set expressions. Consider using 'CharInSet' function in 'SysUtils' unit.
At first I was baffled. Why is it reduced to a byte char? I know set expressions trigger all kinds of compiler optimizations (that's why I like to use them so much), but has someone been too lazy to make them happen with 16-bit values as well? I guess we're talking assembler jump-lists here, so it probably matters if you can limit them to 256 positions, and 65536 positions would add too much of dead chunks of data into your compiled code... But still.
'0'-style literals should default to
WideChar, shouldn't they? So I regret there was no preference to support code re-use here, and CharInSet which appears to be just a wrapper around the
in syntax, strange.
I don't like warnings and hints. I typically avoid them at all cost, treating them as compile errors. It helps to keep your code clean, and avoids trouble down the line. (But if you're really into avoiding trouble down the line, you should do decent static code analysis...) I'm kind of strict on myself that way. So I need to know what's the best way to avoid this warning. In someone else's code I saw
x in [WideChar('A')..WideChar('Z')] but that doesn't appear to suppress this warning. So I settled on
AnsiChar(x) in ['0'..'9','A'..'F'] which probably calls a Unicode conversion routine, but I don't care enough. Old Delphi understands it, new Delphi no longer throws an error, and as far as I know the compiler optimizations still kick in and I haven't seen any performance loss over code like this. So I can sleep again.
Unless someone throws me a much better alternative and I won't sleep for a week because it feels like I should have known that all along...
Databases inside Delphi Ecosystem
2019-10-08 22:40 lgdbdelphi [permalink]
→ landgraf.dev: Databases inside Delphi Ecosystem: Webinar
Ugh, ORM? No thank you. I still need to write the full and extensive treatise about why exactly I prefer not to use ORM's, but every time I see something like "Databases in Delphi", and see the content rapidly veers toward heavy pre-moulded data-layers that bind you to things like ORM and REST API's, I regret so many people fall into this trap. Beginners think they're keeping things easy and scalable, but all hit the same wall later up when performance and complexity is letting them down hard, in part because of these basic choices made up front. I think it's very important to educate newcomers about the alternatives. There are downsides to ORM and heavy data-layers that you should know about.
Which I'm planning to address in a specific page "Why I dislike ORM's". One day. When I'm ready. For now I'll just say I'm glad I've stayed clear of them. I've looked beyond what's typically put forward as the only alternative, and went behind any platform, to talk to SQLite, PostgreSQL and MongoDB directly, unhindered. You get something working relatively quickly. It allows me to tap their full potential and have them scale right along with the front-end as it needs to grow to handle a larger volume of requests. I'm not saying it's perfect. This kind of architecture also has its downsides, but it has one big dependency less into the mix.
Feeder: support WordPress API "wp/v2/posts"
2019-09-14 23:40 wpv2posts [permalink]
Is it normal that a WordPress website serves it full "
https://api.w.org/" REST API from the home page (e.g. https://foreignpolicy.com/)?
I just happed to notice
<link rel="https://api.w.org/" href="..."> while I was looking for an RSS url. So I had to have a closer look. It looks like it's open for anyone to POST new posts to
Anyway, I've created support for it in feeder/eater...
PDF-web: tried forms but nought
2019-08-09 18:40 pdfweb [permalink]
A while ago I thought, since modern browsers show you PDF documents conveniently in the same window, could you make links work and have a dynamic webside that serves PDF pages?
But yes ofcourse you can. Then I also recalled it's possible for a PDF document to have form fields you can fill in. I tried, but the PDF viewers in modern browsers don't appear to support those. (Yet? Give me a ring if they ever do and I'll see what I can do.)
Update: Oh, what's this? Looks like they're working on it...
E-mail over HTTP: update
2019-07-19 21:11 jmap [permalink]
→ E-mail over HTTP (2012)
Ofcourse the magnificent people that are already behind the internet (that beefed-up telegraph with funky terminals) have been working silently on exactly this in general, but completely different in the details: RFC8620: JMAP
A PDF website
2019-07-14 00:54 pdfweb [permalink]
Also, in a PDF you can mark text or a rectangle as a hyper-link. So it should be possible to create a dynamic website that uses PDF instead of HTML, right? One way of looking at it is that PostScript in PDF is a way to layout things on your page just like HTML is.
Anyway, I had to see how much of work it would take to make a proof-of-concept. So here it is, it's not much on the dynamic side, but it's a site that opens to a PDF, and links to another page of the same site.
Local variables don't cost nothing!
2019-03-08 22:45 nocostvar [permalink]
O my god. I can't believe I have to keep explaining this. To a years-long colleague with some more years on the counter than me, the other day. He repeated a function call with the same arguments a few times in a few adjacent lines of code, with a guarantee in place that this will result the same value every time.
I asked him why. It couldn't be to have less typing: the extra
var a:integer; would replace far much more code it took for the repeated function calls. It couldn't be performance: even if the function would just be getting a value from a memory location, the call and return, building a stack frame and tearing it down it again, would end up in a multiple of the instructions the processor needs to work through.
I don't know who launched the idea the compiler should be smart enough to pick things like this out — what I read about Rust and other modern languages, I know research is actively looking at things like that — but as we currently have it, our good old trusted compiler doesn't. What is does do, and typically really well if we work a little with it, is mapping local variables onto the available registers of the CPU. In the best case no memory whatsoever is allocated for local variables.
Even if in theory some local variables get a temporary address on the stack, modern CPU's have multiple levels of cache so that the values don't even reach your DRAM silicon before you're function call completes. We're supposed to keep all functions and methods as short as possible, but even if we don't, the compiler knows if a local variable is only used for a short section of the code. So in some cases it's even better to have an extra local variable instead of re-using a local variable somewhere up front and then later near the end of a larger block, and still not pay with any performance of memory consumption...
So when in doubt, take an extra local var. It's what they're there for. It helps the compiler do better analysis. It helps you, now and later. All I need to do now is name them a little better. I know I have the nasty habit of things like
var a,b,c,d,e,f,g:integer; but I think I'm improving... Slowly...
PostgreSQL: update set from (join)*
2019-03-05 15:56 updatefrom [permalink]
In SQLServer schreef ik wel eens een query van de vorm:
update MyTable set Field1='Value1',Field2=3
from MyTable T1
inner join SomeOtherTable T2 on T2.Field3=T1.Field3
--eventueel nog joins
Maar dan probeer je hetzelfde te doen in SQLite: noppes
Of in PostgreSQL, daar blijkt het wel te bestaan maar hij plakt de referentie naar de tabel van de update niet aan het gebruik van de tabel in je from of join. Omdat je zo dan eigenlijk geen velden van de tabel in de where hebt staan, doet hij de update op alle records, ongeacht je selectie. Maar dat is gemakkelijk op te lossen natuurlijk:
update MyTable set Field1='Value1',Field2=3
from MyTable T1
inner join SomeOtherTable T2 on T2.Field3=T1.Field3
where MyTable.MyPrimaryKeyField=T1.MyPrimaryKeyField --in de veronderstelling dat dit je primary key veld is natuurlijk
Maar daar moet je dus wel zelf aan denken, lastig als je het anders gewoon was...
2019-01-18 08:53 jsondoc12 [permalink]
I don't really know where my open-source/freeware projects get used by other people (and I don't really care all that much, really), but ofcouse where I use my private libraries in the projects of my day job, I've got a first-row seat to see how they perform. No, even better, I'm the stagehand. I know it's important to keep the two separated though. At work I can only use and call libraries, never work on them. I can make changes, but I would have to do so like anybody else: clone a local repository, store the changes there, and when they're ready have the good manners of sending out a pull request. If 'at-home’-‘hobbyist-programmer’-me decides to merge it, it makes my employer a co-creator of the project, creating a new legal situation that my employer would need to know about.
What I can also do, ofcourse, is let private-me know that ‘at-work’-me would really like this or that extra feature or interface, and then when I can put some evening time into programming, I can see if I get round to it. Github issues are nice for that. Or Post-It's on my car key...
But recently I've got a real decent suggestion from a collegue. It didn't come in over a pull request, but it made sense, so why not put it in at the base:
IJSONDocument interface has
Parse to load data from a string holding JSON data, and a function
ToString to convert the contents back to a string. For jsonDoc (and bsonDoc before that) I want concise syntax, so I though I'd make
Parse a function that returns
Self so you can chain calls. But my collegue wondered why
IJSONDocument doesn't have a
property AsString:WideString;. In theory it would have
read ToString write Parse, except Parse needs to be a procedure, not a function. So I had a close look and changed it around. Another option would be to add an extra
procedure SetString, but that would mean the virtual method pointer table for the IJSONDocument interface would have an extra item, to a method with exactly the same behaviour, which makes little sense.
So I changed it around. If you really like/need chaining, there's still the
function JSON overload that takes a Variant. If you pass it a string, it'll call plain
function JSON to get a new instance, and call
Parse for you on the string.
Open source is nice, but is the protocol also open (enough)?
2019-01-15 08:57 openproto [permalink]
→ Hacker Noon: Bitcoin’s Biggest Hack In History: 184.4 Billion Bitcoin from Thin Air; Satoshi Hard Forks, Saves Bitcoin
See, this is something I'm very very worried about: things like Bitcoin — big public successful open-source projects — have the appearance of being complete open and public, but the protocol isn't really.
When I was first looking into Bitcoin and learning what it is about, really, I'm quite sure this can only have originated out of a tightly connected bunch of people that were very serious about 'disconnecting' from anything vaguely institutional. Any structure set up by people to govern any kind of transactions between them, has the tendency to limit liberties of people, for the people taking part in the system and sometimes also for those that don't. So it's only natural that Bitcoin at its code is a peer-to-peer protocol.
But. How do people that value anonymity and independence from any system, even get to find each-other and communicate to build things together? Well, the internet of course. But perhaps more importantly — and also since long before the internet — cryptography. Encoding messages so that only the one with the (correct) key can decode and read the message, helps to reduce the cloak-and-dagger stuff to exchanging these keys, and enables to send messages in the open. To the uninitiated onlooker it looks like a meaningless block of code, and in a sense it's exactly that. Unless you what to do with it, and have the key — or would like to have it.
Another use of encoded messages is proving it's really you that originally encoded a message. It's what's behind the Merkle tree that the blockchain runs on. That way the entire trail of transactions is out there in the open, all signed with safely stored private keys. The reader can verify with the public keys, and in fact these verifications buzz around the network and are used to supervise the current state of the blockchain, building a consensus. Sometimes two groups disagree and the chain forks, but that's another story.
The protocol, or the agreement of how to put this into bits and bytes in network packets, can get quite complex. It needs to be really tight and dependable from the get-go, see the article I linked to above. You could write it all down and still have nothing that works, so what typically happens is you create a program that does it and test it to see how it behaves. In this case it's a peer-to-peer networking program so you distribute it among your peers.
But when things get serious, you really need the protocol written out at some point. If you try that and can't figure out any more what really happens, you're in trouble. The protocol could help other people to create programs that do the same, if they would want to. This was something the early internet was all about: people got together to talk about "How are we going to do things?" and then several people went out and did it. And could interoperate just fine. (Or worked out their differences. In the best case.) It typically resulted in clean and clear protocols with the essence up front and a clear path to some additional things.
The existence of the open-source software culture it another story altogether, but I'm very worried it is starting to erode the requirement for clean protocols more and more. If people think "if we can't find out how the protocol exactly works, we can just copy the source of the original client/server" nobody will take the time to guard how the protocol behaves in corner cases and inadvertently backdoors will get left open, ready for use by people with bad intent.
NoSQL agent with SQL back-end(s)
2018-12-29 15:41 nosqlidea [permalink]
It happened again. I get this great idea that slowly develops, and gives the feeling I'm on to something, but nowhere near any time in the forseeable future to put in to it and get a proof-of-concept of a first project that makes it work. So, for what it's worth, I write it down here fast in the hope sharing it with you may give more chance to this idea getting useful.
I've read that some NoSQL solutions are actually about eventual consistency, meaning in the best case of a query for data that was just inserted or updated could already return this new data if the server(farm) ad already fully processed it. Worst case is that it just for a few milliseconds totally disappears, but that's another story altogether.
I haven't done anything serious with NoSQL yet, and really a lot on good old SQL, and recently with SQLite which I've really grown to love in a short period of time. But still there's something there that's really suited for the new style of programming that is going on with all the new web projects and this 'Internet of Things' everything is on about... To find out, I've been trying interfacing with a number of them from Delphi in the most direct way I could possibly find and make work with reasonable effort. I like how TMongoWire worked out, to talk with PostgreSQL all you need is in the libPQ.dll, but a number of others just stick with a plain HTTP API where you PUT and GET things on their own URL. There's a beauty to that, really. The structure of your documents is nicely contained in JSON, and HTTP is such a stable platform you're sure to be able to access it from almost any platform.
So that's where the idea came from: what if I made my own service where you can just put or get JSON documents? But on the back-end jsonDoc would do the heavy lifting and the storage itself could be in a decent SQL service. And/or it could be in something intermediate like memcached. And/or the saving to storage could be asynchronous somewhere close after the actual PUT call (hence the eventual consistency).
For example, you fill a collection of items with things with a number of fields, for example one is "Price", but later you need the items above or below a certain price, you would do SQL "select * from Items where Price<@p". So in this service I'm imagining, there would be meta-description on the collection that you've provided a SQL database somewhere, but the service is responsible for having done the "create table Items (ID some primary key, DATA json, Price decimal(8,2))" and filling it with the data.
And this would be the beauty of it: if you need an extra column later, you just say so, and the connector would be responsible for the "alter table Items add ..." and filling that column with the data from the stored items. Perhaps even slowly, asynchronously together with the other work. Or even another connector alltogether, let's say PostgreSQL and MySQL side by side, perhaps even as a fail-over for eachother.
But I'm dreaming. It would be a load of work just to get something to work, and even more work to get enough connectors to work good enough to even demonstrate how it would work. And then there's the performance trails... And the evangelising to see wether it solves other people's problems anyway... It would be a really great opportinity to finally cut my teeth on this IOCP thing.
Update to SQLite 3.26 as soon as possible
2018-12-24 14:52 sqlite326 [permalink]
It's very rare something turns up in the SQLite package, but when it does is best to give it some attention. So, very esteemed users of TSQLite — and anyone else, really — I strongly suggest you upgrade to version 3.26 of sqlite3.dll since recently some dangerous vulnerabilities have been uncovered. I checked, and nothing extensive changed to the API so chances are nothing should break if you only update sqlite3.dll and not SQLite.pas and SQLiteData.pas. Also, happy Christmas!