yoy.be "Why-o-Why"

Open source is nice, but is the protocol also open (enough)?

2019-01-15 08:57 openproto [permalink]

→ Hacker Noon: Bitcoin’s Biggest Hack In History: 184.4 Billion Bitcoin from Thin Air; Satoshi Hard Forks, Saves Bitcoin

See, this is something I'm very very worried about: things like Bitcoin — big public successful open-source projects — have the appearance of being complete open and public, but the protocol isn't really.

When I was first looking into Bitcoin and learning what it is about, really, I'm quite sure this can only have originated out of a tightly connected bunch of people that were very serious about 'disconnecting' from anything vaguely institutional. Any structure set up by people to govern any kind of transactions between them, has the tendency to limit liberties of people, for the people taking part in the system and sometimes also for those that don't. So it's only natural that Bitcoin at its code is a peer-to-peer protocol.

But. How do people that value anonymity and independence from any system, even get to find each-other and communicate to build things together? Well, the internet of course. But perhaps more importantly — and also since long before the internet — cryptography. Encoding messages so that only the one with the (correct) key can decode and read the message, helps to reduce the cloak-and-dagger stuff to exchanging these keys, and enables to send messages in the open. To the uninitiated onlooker it looks like a meaningless block of code, and in a sense it's exactly that. Unless you what to do with it, and have the key — or would like to have it.

Another use of encoded messages is proving it's really you that originally encoded a message. It's what's behind the Merkle tree that the blockchain runs on. That way the entire trail of transactions is out there in the open, all signed with safely stored private keys. The reader can verify with the public keys, and in fact these verifications buzz around the network and are used to supervise the current state of the blockchain, building a consensus. Sometimes two groups disagree and the chain forks, but that's another story.

The protocol, or the agreement of how to put this into bits and bytes in network packets, can get quite complex. It needs to be really tight and dependable from the get-go, see the article I linked to above. You could write it all down and still have nothing that works, so what typically happens is you create a program that does it and test it to see how it behaves. In this case it's a peer-to-peer networking program so you distribute it among your peers.

But when things get serious, you really need the protocol written out at some point. If you try that and can't figure out any more what really happens, you're in trouble. The protocol could help other people to create programs that do the same, if they would want to. This was something the early internet was all about: people got together to talk about "How are we going to do things?" and then several people went out and did it. And could interoperate just fine. (Or worked out their differences. In the best case.) It typically resulted in clean and clear protocols with the essence up front and a clear path to some additional things.

The existence of the open-source software culture it another story altogether, but I'm very worried it is starting to erode the requirement for clean protocols more and more. If people think "if we can't find out how the protocol exactly works, we can just copy the source of the original client/server" nobody will take the time to guard how the protocol behaves in corner cases and inadvertently backdoors will get left open, ready for use by people with bad intent.

→

NoSQL agent with SQL back-end(s)

2018-12-29 15:41 nosqlidea [permalink]

It happened again. I get this great idea that slowly develops, and gives the feeling I'm on to something, but nowhere near any time in the forseeable future to put in to it and get a proof-of-concept of a first project that makes it work. So, for what it's worth, I write it down here fast in the hope sharing it with you may give more chance to this idea getting useful.

I've read that some NoSQL solutions are actually about eventual consistency, meaning in the best case of a query for data that was just inserted or updated could already return this new data if the server(farm) ad already fully processed it. Worst case is that it just for a few milliseconds totally disappears, but that's another story altogether.

I haven't done anything serious with NoSQL yet, and really a lot on good old SQL, and recently with SQLite which I've really grown to love in a short period of time. But still there's something there that's really suited for the new style of programming that is going on with all the new web projects and this 'Internet of Things' everything is on about... To find out, I've been trying interfacing with a number of them from Delphi in the most direct way I could possibly find and make work with reasonable effort. I like how TMongoWire worked out, to talk with PostgreSQL all you need is in the libPQ.dll, but a number of others just stick with a plain HTTP API where you PUT and GET things on their own URL. There's a beauty to that, really. The structure of your documents is nicely contained in JSON, and HTTP is such a stable platform you're sure to be able to access it from almost any platform.

So that's where the idea came from: what if I made my own service where you can just put or get JSON documents? But on the back-end jsonDoc would do the heavy lifting and the storage itself could be in a decent SQL service. And/or it could be in something intermediate like memcached. And/or the saving to storage could be asynchronous somewhere close after the actual PUT call (hence the eventual consistency).

For example, you fill a collection of items with things with a number of fields, for example one is "Price", but later you need the items above or below a certain price, you would do SQL "select * from Items where Price<@p". So in this service I'm imagining, there would be meta-description on the collection that you've provided a SQL database somewhere, but the service is responsible for having done the "create table Items (ID some primary key, DATA json, Price decimal(8,2))" and filling it with the data.

And this would be the beauty of it: if you need an extra column later, you just say so, and the connector would be responsible for the "alter table Items add ..." and filling that column with the data from the stored items. Perhaps even slowly, asynchronously together with the other work. Or even another connector alltogether, let's say PostgreSQL and MySQL side by side, perhaps even as a fail-over for eachother.

But I'm dreaming. It would be a load of work just to get something to work, and even more work to get enough connectors to work good enough to even demonstrate how it would work. And then there's the performance trails... And the evangelising to see wether it solves other people's problems anyway... It would be a really great opportinity to finally cut my teeth on this IOCP thing.

→

Update to SQLite 3.26 as soon as possible

2018-12-24 14:52 sqlite326 [permalink]

It's very rare something turns up in the SQLite package, but when it does is best to give it some attention. So, very esteemed users of TSQLite — and anyone else, really — I strongly suggest you upgrade to version 3.26 of sqlite3.dll since recently some dangerous vulnerabilities have been uncovered. I checked, and nothing extensive changed to the API so chances are nothing should break if you only update sqlite3.dll and not SQLite.pas and SQLiteData.pas. Also, happy Christmas!

→

TOML? YAML? ini-files? JSON? Loosen up!

2018-12-10 11:26 jsonloose [permalink]

TOML? YAML? plain old INI-files? or shiny and new — but strict — JSON? Nàh. I've made myself something better. In jsonDoc.pas I've recently added an optional compiler define JSONDOC_JSON_LOOSE that makes the JSON parser a whole lot genter to work with. Some things are no longer required: the root document doesn't need to start and end with braces ("{}"), the quotes around the key names are no longer required (unless they contain special chars), the comma's between key-value pairs are no longer required, it accepts equal ("=") between keys and values instead of colon (":") and can even do entirely without.

And while I was at it there's also JSONDOC_JSON_PASCAL_STRINGS that allows you to write string literals the Delphi way: between single quotes, and double single quotes where you want a single quote in the string value. This way — really handy in Windows environments — it doesn't require you to escape backslashes.

The only downside maybe is that you need to add the defines to the compiler parameters, but for plain old configuration something like this:

x=5
y=10
output='C:\test\output\'

looks a lot nicer and like what we're used to, and gets parsed just the same as if it was written like this:

{
  "x":5,
  "y":10,
  "output":"C:\\test\\output\\"
}

→

xxm 2.0? About the big plan

2018-11-28 22:50 xxm2plan [permalink]

Should I begin on xxm v2.0? Delphi having a Community Edition is really really great news, not only for all of the Delphi community, but also for me as a hobbyist programmer with a number of freeware opensource projects out there. Except I was one of those that stuck with Delphi 7 up till now, so next to myself being fully convinced to finally take the step, I also need to have a look project-per-project how to lift them into these modern times and upgrade them to full Delphi 10 projects.

One of my darlings is xxm. You can compile xxm with modern Delphi just fine. I switched string to AnsiString or WideString explicitly nearly everywhere, so you shouldn't get as much as a hint. (Except mayby somwhere where I still have case x in ['A'..'Z'] of or something like that where x is a char and thus is a WideChar, and the compiler doesn't like that). But I'm worried about how I chose to set up the project. I thought I was best to just inherit from IUnknown and define a decent interface and build everything around that. But several xxm projects later, some that are even running some decent production load at work, I learned some things, most importantly these two:

It looks like xxm doesn't need to work with object instances. Requests are coming in, something needs to build a response, but having to have your objects set up first before you can get crackin' now appears as something I should have tried to avoid. Specifics and identities are handled by the underlying communication framework anyway, so you can get by with just a reference, and request more with that reference only just when you need it.

The other is that with the dynamics of IUnknown pointers, you're still dragging this COM thing with you. Yes it may be wafer-thin really and proven tech, but it's still something removing you from what really counts: pointers to where the code is to run. If you're really (really!) pushing to get the last drop of jouce from the orange performance-wise, it may be just that last thing that's in the way. Even when you avoid any locking in reference counting.

So this is the big plan for the moment: if I get to put some decent time into the project, over the next year(s), I'm considering starting a blank project xxm 2.0, start with a plain C style interface with just procedure and function pointers — a really really ancient looking interface — that has everything that IXxmContext and related interfaces have now. Then re-start with xxlProto and xxmHttp to make that work, then a new xxmCGI, xxmSCGI, xxmISAPI, xxmAhttpd (and perhaps now really a xxmFastCGI, but no xxm IInternetProtocol any more, sorry) and then have a first new push to see how it performs, with IOCP and http pipelining, and who knows HTTP/2 or HTTP/3 that'll have materialized by then.

After that I was thinking I need a new implementation of IXxmContext that just patches through to the new xxm 2.0 calls, so you can very easily and without a worry add it to an existing xxm v1 project and run it with the xxm v2 handlers, enjoy the performance benifits and slowly change things over to xxm v2 where you need to — or not at all, really.

I hope it's a good plan, but for now it's all a dream. I haven't been able to put much time into my hobby projects lately, but it fluctuates and if it picks up some I'll see how far I'll get. Along the way I still hope to find more people that are searching for a good way to combine good old web-scripting with the power and speed of the Delphi compiler, and can count on xxm to offer good debugging, easy live-updating and a stable and secure platform for their web projects.

→

Don't panic: Bitcoin's usual pre-end-of-year dip is here.

2018-11-23 00:29 btceoy [permalink]

Don't worry. The price is going down a bit, yes. But I think there's no reason for massive public panic and the cyber-world's equivalent of a run on the bank. The holiday season is here. Black Friday. We need presents for the family, and that costs money. We may have chosen to put the value of our earnings into this new thing designed especially for that, and now need to convert some back to good old local currency, so it probably pushes the exchange rate a bit down. It may even be a good sign of institutionalisation that automated agents kick in and join in on the selling, pushing rates even further down, dramatically so. But it's best for all of us if that's only a marginal effect. If I were speculating, I would guess it would start to look like a buying opportunity, if I could statistically detect when the bottom would come into view, but I am not. Actually I don't care. But since we're guessing, I guess things will bounce back in January. It did last January. Perhaps it won't get to the same levels as we had up till before this dip, but that's OK. If it's all rather stable for a few months, that would be good, but if it follows real events in the real world, that's also good. It's just normal. I'm not worrying. Perhaps next year we'll be paying Christmas gifts in bitcoin. (Though I wouldn't put money on that...)

→

TMongoWire on jsonDoc v1.1.8

2018-11-01 17:24 mwjd118 [permalink]

→ TMongoWire commit 78c7c62: jsonDoc v1.1.8

I may have decoupled the bit that works with structured data (using Variants), and converts to/from JSON, into a separate project jsonDoc, and have converted TMongoWire to run on jsonDoc, I also should take the time to update TMongoWide when jsonDoc gets improvements. The fact is I'm doing much more with JSON than on MongoDB recently, so I risk letting this slip out of sight.

What I also should do is set the TMongoWire repository up so it uses the jsonDoc repository as a git module, but for now it's just about one single file so I'll see when I get to it (...).

→

Browsers with less and less UI...

2018-10-05 17:19 browserslessui [permalink]

Here's a wicked idea: With browsers trying to have less and less UI, the line of death getting more and more important to help guard your safety, and some even contemplating seeing the address bar as a nuisance — who types a full URL there nowadays anyways? — what if there was a browser that always opens fully full screen. No need for F11. You still need a back and a refresh button, and something that gives access to all the rest like settings, stored page addresses, and if you really really need it, the address of the current page. But it is hidden from view most of the time, except when you make a certain gesture, like a small counter-clock-rotation. It should look different enough so it contrasts with the page, and should be different every time, so it isn't corruptable by any webpage. And even then should be obviously not part of the page.

And people need to find it intuitive and self-explanatory.

Oh never mind.

→

Barney Cools: Tunes

2018-10-02 23:31 bct [permalink]

O jee, ik heb in een eeuwigheid niets meer geblogd over muziek! Terwijl er toch wel een hoop te vertellen is. Hoewel, zoveel nu ook weer niet. Na meer te willen weten over Oliver Koletzki, en SoundCloud ontdekt te hebben, kwam ik daar langs deze:

→ Barney Cools: Tunes

Eerst wist ik niet goed wat er van te maken, huis-mixes van een open-lucht-zwembad-beach-bar in Australië? Blijkt het een kleding-merk te zijn die ook mixes de wereld in zet. Tot nu toe vind ik ze behoorlijk de moeite. Op hun beurt zijn ze dan ook weer een manier om nieuwe artisten te leren kennen. Zoals Aeonix of Haert. Of dit pareltje... Of deze!

Verder vind ik het vreemd dat je zo weinig hoort van dat Chvrches nieuw werk heeft...

→

ECB should plan to issue a digital currency!

2018-09-18 00:07 nodraghi [permalink]

→ Reuters: ECB has no plan to issue digital currency: Draghi

Here's an idea. Just an idea, floating it here to see what you think, no concrete plans yet. The internet should float a new digital currency. "Wait, what? We have Bitcoin/Litecoin/Ether/... already, are those not internet's digical currencies?" I hear you think. Well, no. They're intended to perhaps become currency, but that kind-of totally failed. The world wasn't ready for Bitcoin when it hit us, and all the nice plans kind of prescribed to one day use bitcoin as currency, but as the hype and dust are now somewhat settling, it's clearly unfit for that purpose. It's still great at what it does though, and it could perhaps serve really well as something like gold: something that holds value you can buy and sell and will most probably get bought and sold in the forseeable future, according to current market behaviours. And there's the blockchain which it all runs on, it's a great proof-of-concept of a public ledger that some industrial settings could greatly benefit from, who knows perhaps in a slow movement from the fortified castle paradigm to the zero-trust concept.

But as a currency? No. Currency is allergic to strong ups and downs in the inherent value. "Didn't we have all this already, the US dollar doesn't have inherent value as well, since we've let go of the Gold Standard?" Well, no. The price of gold may now be free-floating, but since the entire US economy and a sizable part of the world's economy is running on US dollars, you could consider the entire economy as what's carrying the real value of all those dollars. I'm oversimplifying here, but some big large-scale economic metrics appear to work reversed for the US-dollar because of this. A currency as we know them now also had a central body that goversn both the internal use of it, and the powers that exert on it from outside, other currencies and macro-economic movements.

So here's my idea: because Europe is looking to do something about copyright on the web, and newspapers — and perhaps journalism in general — are struggling, something like the European Union should float a digital currency, specifically to make micro-transactions on the web. And I really mean micro. Listening to a song? Bam, something tiny moves from your online wallet to the musician(s). Viewing a video? Bam, something tiny moves from your wallet to actors, directors, lighters, screenwriters and background-painters. Read an article? Bam, you get the idea. How much? How many articles are in an avarage newpaper? How much does a regular newspaper cost? Calculate back from that to get a good first unit of value.

As an alternative way of payment, it could complement the Euro, and only later move up the ladder if there's a base of users with accustomedness. But to get there some important details need to be set up just right. It will need a governing something, but I wouldn't hand it over to Frankfurt. The time is right to involve the people. Bitcoin is doing just right without central oversight, but the required checks and balances need to be baked in. Anything new like this should also be design 'of and for the people'.It will need its proper legislation to get to serve as anything official, an get it accepted as a bearer of value, but by limiting who can exchange how much to and from real currency, for example a weekly global limit on conversion, could dampen the risk of large-scale mutations induced by panic. Or by limiting the maximum amount you could hold per user or per device or per account, could limit the importance of this new stream of cash in regard with the entire economy.

Also as an internet-centric application, every user wanting to participate needs to run the software, but it should be entirely open so each of us can govern that our security and privacy is catered for. Only then it's ready for designing the conduit with which you let the websites you visit know what credit you provide when consuming songs and articles. There needs to be something like a public ledger, since that would make it a new skool digital currency, but requiring every mobile device to keep a full copy of the ever growing full ledger is absurd. And it is also limiting the maximal number of transactions that can get processed in limited time, so that needs to get decentralised as well. I'm not sure how, but I'm sure there are people smarter than me that have been deep enough into the theoreticals that could draft what it takes.

But I'm just dreaming aloud here. Innovation hurts and is hard work. And there are always those that don't want anybody to challenge the status quo.

Update: look, look, this is also about something like that!

→

"Leren programmeren is even belangrijk als leren lezen schrijven"

2018-09-11 14:38 nwsprog [permalink]

→ VRTNWS: Leren programmeren is even belangrijk als leren lezen schrijven

O jee. Hoe gaat het lesmateriaal daarvoor er uit zien? Wie zou dat kunnen samenstellen? En wat gaan ze geven als programmeren, want het is intussen best een behoorlijk breed kennisdomain met veel onderverdelingen. Bepaalde dingen van het computerwereldje liggen bepaalde mensen zelfs beter dan andere dingen. Denk maar aan het verschil tussen taal-vakken en wetenschap-vakken.

Maar misschien nog belangrijker: zit er ook een stuk etiek in? Je kan alles programmeren wat je maar wil, en dus ook programma's die doelbewust misbruik maken van het vertrouwen van de achteloze gebruiker. Of erger nog, schade berokkenen. Om nog maar te zwijgen van de recente revelatie dat alle complexe software die werkt op maatschappelijke data blijkbaar ook latent racisme en discriminatie braafjes toepast zonder verpinken.

Toen ik het middelbaar doorworstelde hadden ze net een vak 'technologische opvoeding' ingevoerd. We hebben een bureaulamp gemaakt. En iets gedaan met blauwe bakjes met in het groot "OR" en "AND" op. Het zou uiteindelijk een verkeerslichtje moeten doen maar dat werkte niet (meer?). En er was ook iets met kleine baksteentjes en lijm in plaats van mortel, maar dat had het gelukkig niet gehaald van de onzin-filter die de professionals vooraan in de klas gelukkig aan de dag leggen.

Ik herinner me dat we enkele jaren later toch al wat puntjes konden verdienen met een mooie pivot-chart of een etiketten mail-merge. Dus ergens denk ik als ze nu eerst zien dat scholen iets gemakkelijker wat computers in de klassen krijgen, dan zal de rest wel volgen.

→

Waarom niet gewoon zomaar wat belastingen bij?

2018-09-08 16:50 belasting [permalink]

Iedereen met een laag loon gewoon een bedrag bij geven is een slecht idee. Alleen al vanwege het drempeleffect. Voor iedereen net boven de limiet voelt heel sterk dat ze net uit de boot vallen, iedereen net onder de limiet zal vreemd genoeg zelfs een inspanning moeten doen om iets minder te verdienen om het voordeel te behouden. En in een situatie waar je net de grootste moeite hebt om aan een inkomen te raken, is dat ongewenst en het zelfs ontwrichten.

Ik dacht een tijdje terug aan een ander voorstel. We betalen in theorie allemaal inkomstenbelasting. Zelfs al is dat de ronde som van 0€, je doet jaarlijks een aangifte van dingen die belast worden of je een vermindering opleveren. Stel dat de berekening no zo wordt aangepast dat het finale cijfer voor iedereen nog eens door een formule moet van de tweede graad. Een dergelijke parabolische exponentiële Curver kent een verloop met opstaande armen zowel links als rechts en een breed diep dal. Mits te kiezen voor specifieke factoren in de formule kan je dat zo mikken dat een origineel bedrag van 0€ uitkomt op 500€ (of meer), en dan gradueel naar beneden gaat, misschien zelfs onder nul voor de modale burger met een gemiddeld loon, en daarna terug omhoog voor de veelverdiener, en bijgevolg extreem hoog voor de absurd onbeschaamde geldscheppende.

Dit plan lijdt niet onder nadelige gevolgen van een opgelegde drempelwaarde, maar valt mogelijk wel door de mand als je rekening houdt met de gewoonte van mensen die met meer geld omgaan dan de modale burger om creatieve boekhouders aan te spreken zodat er een aanzienlijk deel toch niet onder de inkomstenbelasting valt.

Maar ik ben dan ook geen doorwinterde legistator die daar wel iets op kan bedenken.

→

Done! Feeder now on PostgreSQL

2018-08-31 20:25 feederpq [permalink]

Tadaa! Thanks to DataLank, I was able to switch feeder from SQLite to PostgreSQL in roughly one evening. You just need to change ifnull to coalesce, (and SQLiteData.pas for LibPQData.pas ofcourse) and storing Delphi's TDateTime values now map to genuine timestamp values, so it's a little different to do arithmetic with...

But I was getting some 'database locked' errors, so I guess it was the best thing to do, since SQLite works really well but isn't quite designed for use by a website, let alone multiple processes. Sorting by timestamps appears to take a little more processing power than I was expecting, so I may try to change back to storing them as double (since that's also the value TDateTime uses internally), or I may just be missing an index that could speed up the query. It takes some investigating to find out, but that's for another day...

Update 2018-10-12: With some regret, I need to inform you that I have switched back to SQLite, perhaps for the time being. Though operation was acceptable, I noticed quite some performance was lost on queries that depend on the values in the fields of type timestamp without timezone. I've had a look if I could fix this by adding indexes, but wasn't successful. I was also thinking about changing the date-time fields to float, as that's what's actually used in the Delphi code that handles the values, and would probably get better performance in comparing and sorting in PostgreSQL, but the time it would take to switch that around would exceed the time it takes to switch back to SQLite. And switching back to SQLite offers a situation that has shown to perform well in all cases. With this switch I can find some time (somewhere?) to read up on PostgreSQL and timestamp fields, and what I'm missing that was causing some queries to take abnormally long. The query that averages times between posts, prior to checking feeds for new posts would regularly take over 100 seconds, a query that SQLite would only take seconds over.

→

"How can i do C++ for free?"

2018-08-30 10:54 freecpp [permalink]

Someone at work asked me "Is there a way to do some C++ for free?", and told me to mail some links. I did and thought it's a kind of nice overview, so I post it here as well.

If you're looking for a free C++ development environment, the first thing I can think of is:

http://www.codeblocks.org/

but one of the main trendsetters has ofcourse always been Microsoft Visual C++, you can obtain here:
https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community&rel=15

or ex-Borland's version with language extensions to make its UI designer work:

https://www.embarcadero.com/products/cbuilder/starter

if you're willing to get to work with just a compiler, then this one is getting good remarks recently, not only because it's giving usefull error messages:
https://clang.llvm.org/

but there are plenty more:
https://gcc.gnu.org/
https://digitalmars.com/download/freecompiler.html
http://openwatcom.org/download.php
https://software.intel.com/en-us/c-compilers
https://www.embarcadero.com/free-tools/ccompiler (or perhaps this one https://edn.embarcadero.com/article/20633 )

(or the list from the creator of C++ itself: http://www.stroustrup.com/compilers.html )

but C++ is getting some negative remarks lately:
https://duckduckgo.com/?q=why+C%2B%2B+sucks

so what is recommendable to learn? currently good candidates are:

https://dlang.org/
https://golang.org/
https://www.rust-lang.org/en-US/
https://www.lua.org/
https://docs.microsoft.com/en-us/dotnet/csharp/

and ofcourse:
https://www.embarcadero.com/products/delphi/starter/free-download

→

RSS is far from dead!

2018-08-08 00:55 feeder [permalink]

I've been using RSS/Atom feeds on and off since I've learned about them. A long time ago, Google had a nice feed reader, but decided to discontinue it. Users were left to search something new, and I settled on The Old Reader, combined with gReader since I had a smartphone, and all was well. For a while. After some time you notice you still get disturbed by some tiny issues you can't seem to get to go away, either with tweaking the configuration or with Stylus. So what does a developer do? Start to think about developing their own solution. Then plan to develop their own solution. Then develop their own solution. So I'm somewhat proud to present this little thing I've been tinkering on in off-hours the last month:

→ github.com/stijnsanders/feeder

I have a live version to try out here: http://yoy.be/home/feeder/ but it uses the neighbouring instance of tx for authentication. I should enable Google/Facebook/Github OAuth things instead, but finding out how that works is a few items lower on my wish-list (of things I wished I had the time to put into).

I wanted a feed reader without the extra's. I wanted to mark items as read that move out of view by scrolling down, and plays somewhat nice with the surrounding HTML and the browser. For now I like how it works. There's an issue with emoji's that apparently get eaten by UTF8Decode, but that could be a bug that got solved since good old Delphi 7. But now that Delphi has a community edition, I think I should bring most if not all of my other projects to this version instead of sticking to Delphi 7... But that's another story. (One you might notice some time in the future on my Delphi RSS feed...)

→

How to do timezones?

2018-07-17 23:53 timezones [permalink]

I feel stupid. I guess I've been put out of my comfort zone. I was thinking about doing something, and then I did, and I thought I'll just primarily make it work for me, but then I noticed "new" items were still two hours behind. Ofcourse that's due to the bias against UTC we have here in my current local timezone. So there you have it. All my programming career, I would just parse dates, and stop before this "+0200" or "-0900" thing that's there at the end. So the right thing to do is go all the way, and with a little effort make it work correctly for not just me, but for any user of this little thing.

But how? I remember these drop-downs with timezones where I typically look-up Brussels or Paris. But it's summer daylight-saving-time for the moment. So it switches to and from only one hour between UTC twice a year (until policy changes), how do I keep track of that? Is there something that I can have the system do this automatically for me? It feels like these are things I should have known all along, but dont. I guess I'll have to start reading somewhere...

→

StackOverFlow/Delphi: new blood?

2018-07-12 21:30 sodnewbies [permalink]

→ Stack Overflow — Newest 'delphi' questions

Am I seeing this correctly? I've been following this page somewhat less closely lately, but the majority of new questions is by people with not too high of a reputation score. And that's actually a good thing. Let me explain:

A while ago it looked like we were 'past StackOverflow peak'. StackOverflow started as an alternative to outperform all other question-and-answer sites for techies, by having a really well developed reputation system that allows a community to self-regulate. And it did. Both the reputation-system created a really fine repository of good questions and good answers, and all other question-and-answer websites were oblitherated (at least from the google top results on typical search-queries).

A few years later, StackOverflow appeared to struggle with having lost it's reason d'être: people with actual questions would easily mistake StackOverflow as a forum and saw most questions rapidly closed and reprimanded for not attaining an expected level of quality the community would hold to. This is a bad deal for newcomers and in general a source of bad feelings. They know about this over at StackOverflow. And have committed to take action. I haven't kept up to speed about what they're exactly plannig to do, but it could already be working.

Specifically for the questions tagged 'delphi', it's not only good that this way more people that just started collecting a reputation saldo, are posting valid questions and are getting helpful responses; I also think you can derive from it that more people are getting into Delphi. It's not up to me to tell whether that's in part because the most recent Delphi versions also successfully target mobile platforms, but if it's true I'm glad to know more people are about to experience the solidness of the Delphi eco-system, both in tooling and available components, and in solidity and reliability of your final result you're offering your users.

→

Got style?

2018-07-06 21:29 whatsyourstyle [permalink]

→ Firefox and Chrome Pull Popular Browser Extension Stylish From Their Stores After Report Claimed It Logs and Shares Browsing History, Credentials — Slashdot
→ “Stylish” extension with 2M downloads banned for tracking every site visit — Ars Technica

Oh, what's this? Note to self: switch to Stylus, (also here and here)

→

Another store from the trenches: GDI vs PDF

2018-07-06 18:31 gdipdf [permalink]

One of the great things about Delphi is this object-oriented way of manipulating things called the Visual Component Library. There's much more to it than that, but in a sense you could describe it as a wrapper around Windows API's. One of these is GDI, it's old but it's still there and used for graphics and text-layout. It was born, way back when, from the idea of What You See Is What You Get (WYSIWYG) taken even further and that the same program logic could work for display on screen and writing to a printer. Delphi's VCL declares the TCanvas class you can use to design the output, and wether you pass a Printer.Canvas, Screen.Canvas, Form.Canvas or Bitmap.Canvas shouldn't really matter. This works great for forms and reports.

But in the printing business, there's much more than forms and reports to put on paper. Over there the PDF reigns supreme. It performs a similar task, but originated out of a very different set of parameters. A big module we created of the production software at work generates them according to specifications. How this works is very different as the components that make up a PDF document are constructed one by one and this doesn't look like the work you normally would do with a TCanvas instance.

But now and again these two worlds need to bridge over to eachother. One day I had to keep automatic PDF archive copies of some reports, and discovered the great SynPDF project from the same people that created the mORMot framework. It offers a TCanvas instance (TPdfDocumentGDI.VCLCanvas) that you can pass to code that knows what to with a TCanvas reference. It works great.

I haven't needed to think about crossing the two the other way round, but recently I noticed a collegue did, and he also found a simple straight-forward solution for the problem: You can just ask the PDF reader on the system to render a PDF to a printer. It's as simple as starting a process with:

"C:\Program Files (x86)\Adobe\Acrobat Reader DC\Reader\AcroRd32.exe" /n /t "<document to print>" "<printer name>"

"C:\Program Files (x86)\SumatraPDF\SumatraPDF.exe" -print-to "<printer name>" "<document to print>"

This works just great and even doesn't add too much code to your program. You may need to include the PDF viewer installer with your installer, but that's a minor issue and perhaps not even required as most people have a PDF reader installed by default.

→

DirDiff v2.0.4.535

2018-06-23 00:31 dirdiff535 [permalink]

→ DirDiff v2.0.4.535

There was a bug when diff-ing more than 2 files or folders, where lines matching between other files than the first/left one would show as matched.

→

Delphi project and subversion: set the build number to revision number

2018-06-09 09:37 dsvnbuild [permalink]

At work, we have a number of interrelated projects in a single subversion repository. We've agreed to change the build number in the project's version properties to the current revision number of your work folder. And specifically not the revision number you might predict your changes will land in when you commit them.

We're currently a rather small team so it may be tempting to assume you'll just get the next revision number, but we stress newcomers to ignore that urge. As long as we're a small team, this works manually, but if we grow we should move over to scripted version resource entries and commit hooks that update the build number automatically.

So with a major and minor version of 2.0 for example, and a release number of 8, the full version could look like "2.0.8.51873".

This way of working has a number of advantages.

The binaries that get used in production, but exert a bug in their behaviour, show with their full version number (and the previous version number from before the bug occurred) the vicinity of revision numbers that introduced the code change that may lay at the cause of the bug. To get the exact revision number, you need to look up the SVN log (or blame) of the .dof or .dproj file, but it's quite sure it's a number closely above the number in the version. We've stressed second line support personnel to list this number when reporting the bug, which helps when researching reproducibility.

But even long before that, when it happens two (or) more of us inadvertently make changes to the same project, we either started from the same revision number in the work folder and see when comitting that something has to get merged and it does so automatically or we get a merge conflict; or we started from different revision numbers and get a conflict right away or a message that we're over due for an update to or work folder.

As I said we're a small team for the moment and it rarely happens so it saves us an update task with risk of conflict before we commit, and in general we can split the work that needs to be done between us so we shouldn't make changes to the same projects.

Then again, if the team would ever grow to something like really big, we would probably have to switch to something else than subversion, or even repositories per project, who knows...

→

GMail: make the labels list compacter

2018-06-08 10:32 GMailLabelsCompacter [permalink]

If you remember from before, I have so much labels in GMail that I didn't like that the box to change the labels on a message with, is so small. Stylus to the rescue.

Now there's this new GMail design, and even in compact display, the list of labels on the left doesn' fit my screen. Also I don't like the font the subject line is rendered in. So a bit of inspection later, I add these lines to my overrides:

.z0 {
  margin: 0px;
  height: 32px;
  padding: 0px 0px 0px 64px;
}
.z0>.L3{
  height:24px;
}
.ha>.hP {
  font-family: "PT Sans", sans-serif;
}
.aim {
  height: 18px;
}
.J-N {
  padding: 0px 12px 0px 32px;
}
.J-LC
{
  padding: 0px 12px 0px 48px;
}

→

AES v1.0.1

2018-05-21 20:18 aes101 [permalink]

→ md5

I noticed something was wrong with the key generation schedule in my AES implementation. I had a close look with the FIPS 197 example vectors at hand and fixed it. I still should make some time to run it under pressure and see how it relates to reference implementations and popular implementations.

→

xxm v1.2.5.450

2018-05-03 22:43 xxm450 [permalink]

For a while I thought there's not enough there to have a full release, but with the switch from XML to JSON, and NTLM and WebSocket support in xxmHttp and xxmHSys2, plus all the minor changes and tweaks that improve performance and security, it turns out to be quite a formidable release. Regretfully, by finally deciding to have all string arguments const anyway, any project's xxmp.pas will have to change. A long time I was hesitant about this because I feared binary incompatibilities, but I've tested all permutations between caller/called with/without const, and it all works. Except when building old code with the new xxm.pas will serve you with this minor one-time inconvenience of having to add const yourself to the arguments... I've thought about trying to create something automatic — like the XML to JSON converters — but since you're quite free to modify xxmp.pas to your liking, it may be more complex than it seems. (If you would like to see this anyway, let me know and I'll put some time into that.)

It's advised to do an extensive test-run with this new version before you update the live environment. If anything unexpected turns up, let me know!

→ xxm v1.2.5.450 (→gh) (→sf)

project registry "xxm.json" (auto-conversion of "xxm.xml")
project file "Web.xxmp" JSON (auto-conversion from XML)

### ATTENTION ### ATTENTION ### ATTENTION ###

xxmProject and xxm handlers are fitted with a transitional
automatic conversion procedure that will convert XML project files (Web.xxmp)
and the project registry (xxm.xml) into JSON files. (Web.xxmp in place,
storing a Web_xxmp.bak copy; xxm.json)
This may cause incompatibilities if different xxm versions are required to
coexist on the same installation(s).
This procedure (and all code using XML) is scheduled to be removed at a
later, currently undetermined, version.
fixed issue with custom parserValues spilling into next file processed
IXxm* interfaces: all string arguments now with 'const'

### ATTENTION ### ATTENTION ### ATTENTION ###

This may cause errors on generated xxmp.pas:
xxmp.pas() Error: Declaration of 'LoadPage' differs from previous declaration
xxmp.pas() Error: Declaration of 'LoadFragment' differs from previous declaration
Please update both methods by adding 'const' to the arguments with a string type.
xxmIsapi*: Redirect: set 'Cache-control' to 'nocache' unless it was set already
xxmHttpAU: added manifest to avoid default UAC elevation request
xxmHttp,xxmHSys: Transfer-Encoding chunked (set Context.BufferSize to something >0 !!!)
xxmHttp,xxmHSys: NTLM authentication (using SSPI, add "ntlm":true to the project in xxm.json)
xxmHSys2: WebSocket support (see demo's: 13 WebSockets)
DispositionAttach: replace unacceptable characters (\/:*?"<>|) with '_'
SendFile, SendStream before header sent, will by default have ContentType
application/octet-stream instead of text/html, unless already set before
(with Context.ContentType!)

→

What do you think I should do?

2018-04-20 23:26 sendfile404 [permalink]

Dilemma, dilemma! What should I do? Sometimes, you need to have a certain URL in a web-project that just serves op the contents of a local file. You could try to have the file in a static section of the website, or even a sub-domain for static content (if your budget allows it), but then it's out there for anyone. Best is to have a request be processed dynamically, so you can do some server-side logic first, for example to do authentication control, or generate the file first (perhaps with a graphics library?). That's where the SendFile method of the IXxmContext comes in. The code that implements it is pretty straight-forward:

procedure TXxmGeneralContext.SendFile(const FilePath: WideString);
begin
  inherited;
  if State=ctHeaderNotSent then FSingleFileSent:=FilePath;
  SendStream(TStreamAdapter.Create(TFileStream.Create(FilePath,fmOpenRead or fmShareDenyNone),soOwned));
end;

If the request's header has not been sent already, the private value FSingleFileSent is set to the file path, assuming the request is meant to have this file's data as response data. This is used for error handling and logging.

Then SendStream is called, which takes an IStream pointer, so a TStreamAdapter is used to wrap around a TFileStream, with ownership so the last IStream._Release will call the file stream's destructor.

One thing that's missing here perhaps is trying to figure out if the HTTP response header Content-Type is set to some suitable MIME-type for the file, but this is so untrivial that it's best left to the developer. So give it a second of thought when you're coding a call to SendFile.

So about this dilemma I'm having. Just imagine for a second you have this code in a project. Attention: this is very bad practice! It should be glaringly obvious to everybody that doing this opens the back door wide open and people with malicious intent can access any file they want on the machine, include system files, so please don't ever really do this:

[[Context.SendFile(Context['f'].Value);]]

Yikes. Very very bad! I feel dirty just for typing that, but just as an example, this code has a high probability of trying to open a file that doesn't exist, or otherwise have the TFileStream.Create throw an EFOpenError exception.

In that case, would it be better if xxm answers with a proper HTTP 404 (page not found) response? Now default exception handling kicks in, and left unhandled (hint) xxm will fashion a HTTP 500 response for you with the exception data. There's also a bit that will see if your local fragment registry can load a fragment for '404.xxm' that lets you design a nicer 'page not found' page than the default, but I'm having a hard time to guess if that would be something unexpected for someone somewhat new to xxm, calling SendFile on an inexistant file for the first time...

I'd love to hear from anyone on this, but for now I'll just let it be like this and let the normal exception play out, if any.

But wait, there's more. Deep burried within the dark corners of the Windows API, there's a thing called TransferFile. It basically lets you tell the system to take a file handle and a network handle and stream all the data from the one to the other, as much as possible right from the kernel. The way the friendly people over at Microsoft worked it out, and tied it to the running system so it would only work on Windows Server versions, makes it kind of unsuitable for where I want to take the xxm project.

But wait, there's more. You may have noticed the web-sphere is gripped with a frenzy for all things asynchronous. There are a few good things there, but it is mainly the best way to serve a magnitude more of concurrent requests by the same server. The short story is you try to avoid waiting on the system while it waits on network or disk. The long story envolves completion ports or libuv, but is in essence unfit to combine with what xxm is doing: having a separate DLL with code you just call to have a response generated for a request, since to do it properly every request to the operating system needs to be re-routed over your job/thread/task/fiber/yarn-management.

This shouldn't hold me back to get as close as possible to the middle ground between the two, where you have the option to build a response, but can hand over a (file)stream and have the HTTP server spool that as it sees fit, once the situation really is just that straightforward. But this may be what xxm 2.0 could be about, if there ever will be something like that.

→

Archive... Search...