CQL by Example

Part one of the main document on queries in the conventional domain is in fulfillment of a commitment to the upstream project. When the CQL engine was first integrated with the upstream executable and merged into that project, the need for a large collection of examples addressing over-the-board games became apparent.

In addressing that commitment, we hoped to demonstrate the broad range of the application of the language in a conventional chess context. We also hoped to lay a decent foundation for the harder things to come. Even experienced CQL users might find a few of the examples to be entertaining.

While the integration of the CQL and Popeye engines with the Scid application is well-warranted in its own right, the entire point of the Scid++ project lands right here, squarely in the problem domain. If CQL has practical, demonstrable application in that domain, the proof of it (or at least the supporting evidence for it) will be found in part two of the main document.

Progress on the problem domain will come in fits and starts. We'll periodically go off on a tangent to work on a companion document or to add a feature to the Scid tool chest. What's there looks promising, but we have a long way to go.

This first companion was inspired by a suggestion from an early reviewer of the main document. We were initially skeptical, but Understanding Sets in a CQL Universe has received a good bit of positive feedback from newcomers as it addresses a fundamental pillar of the language.

Square sets are everywhere in CQL, and visualization of those sets on the board is apparently a key to comprehending the language. After one's initial review of the online reference docs, this should probably be the next stop.

When we first moved from the mainline to the solution tree with its scores of variations, we found ourselves in a bit of a state of shock. After years of dabbling, we thought we had a fair grasp of the language. The truth is that we had barely scratched the surface.

To this day, we still have no intuitive feel for much of the traversal mechanics employed by the language. The behavior of the move filter with linearization disabled, for example, is exactly what one would expect after thinking about it for several hours over a pitcher of Heineken.

We authored Game Tree Traversal in a CQL Universe not for the benefit of others but for our own edification. Had we not, we would be hopelessly lost as we wander our way through the solutions forest looking for themes and patterns. The document represents just a beginning of an understanding of the traversal of the game tree, and most probably with a few lies inaccuracies thrown in.

One should take this companion as a hint of what lies ahead in the problem domain and as a guide to pursuing one's own forensics. When trying to understand the inner workings of, say, the line filter with linearization enabled and repetition qualifiers in play, the message filter is one's indispensable friend.

Handling Twins in a CQL Universe began as a bit of a lark. A challenge. A dare (she knows who she is). It quickly became one of the more interesting and fun things that we have ever done with the language. It helps if one's mind is a bit twisted, and in the wake of a session of pattern matching across twins one might feel the need for a long visit with one's therapist. But the experience has been most rewarding.

Twinning represents a significant slice of the problem domain, and so we look on this companion as more than just a sidebar to our larger "proof". Some very interesting patterns may manifest across twins and the techniques developed in their handling will likely have application elsewhere.

Update: Visits to the therapist may be a thing of the past. (Or not.) We have mixed emotions about the release of CQL v6.1, as the mind requires less twisting now that we have a completely new set of lexical toys which just happen to be uberly handy in a twinning context. It's as if the CQL gods could no longer bear to witness the tortuous querying and deigned to take pity on the poor plebe. We suppose we should be grateful. We're still pondering.

Not exactly boards. Representations of boards. Think set variables (bitboards) or perhaps even a FEN string. Rumor has it that transformations may be realized with nothing more than direction filters. We've experimented with the assertion and are mostly convinced that it holds.

The need for such a capability arises in, e.g., cross-twin queries where positions are persistently carried across a series and might require shifting or rotating, for example, for comparison purposes. We might also find a similar approach useful in detecting certain forms of symmetry.

The Transforming Boards in a CQL Universe document will be developed alongside the Handling Twins companion, and may ultimately lead to the assembly of a crude "library" of functionality.

Huh? From what and to what? Well, bitboards, of course.

In our work on the Transforming Boards companion we found ourselves in need of a bi-directional translation facility between FEN strings and a representative suite of bitboards. We're having difficulty visualizing transformed bitboards, especially with the flips and rotations. If we can translate between representations, we can write our transformed boards to an EPD file for easy viewing. Problem solved.

This little diversion — Translating FEN in a CQL Universe — probably belongs in the String Theory companion, but we have no idea what that document looks like yet and we're in a hurry. So there you have it.

Well, no. Not that string theory.

With the introduction of first class strings and regular expressions in version 6.1 of the language, the world of possibilities appears to be unbounded. What can one not represent with a string? Carrying information across solutions and queries just became much easier.

We suspect that string handling techniques developed for twinning and prophylaxis will have application in other areas. Putting it all in a companion document makes sense.

We've learned a few things about constructing queries looking for some rather nebulous chess concepts and ideas. The direct approach is often the least effective approach. For example, rather than asking what are the essential elements of a pattern, we might do better to ask what is the pattern not.

This companion will also likely include a section on multi-pass queries. When and why to employ them, and how to carry persistent information across them.

Distantly related to the task of searching for patterns is the challenge of recognizing patterns as one is solving problems. We set out with this document to defeat the whole purpose of problem solving. But we have the novice (namely ourselves) in mind, who might be wondering how does one even get started? And how much of the nuance in the composition escapes one when the theme runs deep?

We'll use the Babson task as a case study in demonstrating the use of a handful of Scid tools in analysing compositions. CQL and Stockfish play a role, but the solving engine is at the center of the analysis and some lesser tools can make the process slightly less daunting. Taking a deep dive into the solution tree and considering some branches as having stipulations of their own can lead to some revealing insights.

As a bit of a diversion from our activities in the problem domain, we're entertaining a rather unlikely application of the language. Namely, the search for deep prophylactic ideas in a database of grandmaster works of art. That we're even considering such a thing speaks volumes of the language.

The impetus for this idea originated in an e-mail exchange with an Indian GM who had come across our example of a search for shallow (superficial) prophylaxis in the mainline document. That example employs a language technique for anticipating a move that does not exist in the game tree. The benefit of such an approach is somewhat limited.

But if we first tag positions of interest with an initial CQL pass, then we can expand the game tree at those positions — with multiPV threat analysis from an engine — giving us evaluations and actual lines of play for examination by a follow-on CQL query. Threat analysis takes place on the other side of a null move at the tagged position, and the threat lines can be compared against the main line. It's an interesting idea.

We expect that prophylaxis is just one of many applications of the multi-pass query, especially now that version 6.1 of the language allows for persistence across queries. This particular application requires a minor extension to the Scid analysis handling subsystem, but that's likely an exceptional intervention. Multi-pass queries may well become commonplace in the not-too-distant future.

The mass of example queries found in this document suite originated with the release of CQLv6.0, and as such are lagging considerably behind the language's latest features, syntax, and capabilites. While succesive CQL releases are generally back-compatible, there are the inevitable exceptions to the rule. A significant number of the examples in their present state will fail to execute with the latest CQL releases:

• because persistent declarations sans assignment are no longer allowed (v6.1), or
• because in many cases the dictionary key/value types now need to be "declared" early on in the query (v6.2).

We are presently working to address these version back-incompatibilities and, beyond that, the glaring complications manifest in earlier language techniques which we have employed but which have been relieved by the follow-on v6.1 and v6.2 revisions of the language.

A second issue that we'll eventually have to deal with is the deprecation of filters and syntactical constructs which are in wide use throughout the document suite. These deprecations are numerous and include some rather significant filters (most notably the move and line filters). For some of the examples this may require non-trivial resolution, and so we may just decide to leave them unrevised with a note indicating their version compatibilities.

Our priorities in addressing these issues are first (and overwhelmingly) with the Twins document -- which has garnered the greatest amount of attention by a huge margin -- and then with other companion documents having a direct impact on that document. Revisions are already underway (October 2023) but will take some considerable amount of time to complete.

We've introduced a number of pseudo-parameters to the CQL header which are subject to pre-processing by the Scid++ CQL search facility prior to launching the CQL sub-process. These parameters are removed from the CQL header prior to writing the temporary CQL file.

A couple of useful CQL command line options are not otherwise available as header parameters. Scid++ users may give the alwayscomment and nolinearize command line options as header parameters and those parameters will be converted to their respective command line options.

We've also introduced an include pseudo-directive which may be used to load libraries of CQL functionality which may be required by a query. The libraries will be inserted into the CQL file immediately following the header. Numerous examples of this parameter's use will be found throughout the document suite. The project's support files include a collection of libraries in use primarily by examples in the Twins companion document.