WeBWorK Main Forum

PGML and $questionNumber: possible bug?

PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
Number of replies: 19
Below is a minimal example that shows that within a PGML_ enclosure the variables $setNumber and $questionNumber cannot be accessed. Or maybe it shows something else. But it makes PGML slightly less useful.
DOCUMENT();

loadMacros("PGstandard.pl",
"PGML.pl",
);

TEXT(&beginproblem);

$altTitle = 'alt="$setNumber:$questionNumber"';

BEGIN_TEXT
\{ $altTitle \}
$BR$BR
END_TEXT

BEGIN_PGML

[@ $altTitle @]
[@ $altTitle @]*
[@ $altTitle @]**
[@ $altTitle @]***

END_PGML


ENDDOCUMENT();

In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Davide Cervone -
Just to get it out of the way, this is not a bug; PGML is working as it is supposed to. So the issue is one of expectations. Let me analyze the situation.

FIrst, note that BEGIN_TEXT/END_TEXT (which I will call TEXT for brevity) works very differently from BEGIN_PGML/END_PGML (which I will refer to as PGML), as described below.

In TEXT mode, the text string is processed by a series of regular expression substitutions. First command substitution is performed (so \{...\} is replaced by the result of running to code between the delimiters), then variable substitution is performed (where $var is replaced by its contents), and finally LaTeX math is processed (so \(...\) and \[...\] are repaved by the typeset mathematics in whatever form is currently selected by the user, e.g., Images, MathJax, etc.).

What this means is that the substitutions later in the list can act on the results of the earlier substitutions. So in your example, your original text is

\{ $altTitle \}
$BR$BR
and after command substitution it is
alt="$setNumber:$questionNumber"
$BR$BR
(because that $altTitle contains the litter value alt="$setNumber:$questionNumber").

At this point, variable substitution is performed, and so all four variables are substituted into the string. The result will be something like

alt="3:7"
<br/><br>

Note that this relies on your having used command substitution to perform the initial including of $altTitle. Had you used normal variable substitution, as in

$altTitle
$BR$BR
then nothing would change during command substitution, and during variable substitution, you would get
alt="$setNumber:$questionNumber"
<br/><br>
Since no further variable substitution is performed, you would not get the set and question numbers inserted into the result. So you are taking advantage of the substitution sequence to get variable substitution performed in the results of command substitutions.

As an aside, note that the reverse is not possible: you can't get commands performed in the results of variable substitutions (since command substitution is performed before variable substitution). So

$cmd = '\{ BeginTable().AlignRow(["A","B"]).EndTable() \}';

BEGIN_TEXT
$cmd
END_TEXT
would not cause the table to be included in the output (instead, the text \{ BeginTable().AlignRow(['A','B']).EndTable() \} would be shown. The same for
$cmd = '\{ BeginTable().AlignRow(["A","B"]).EndTable() \}';

BEGIN_TEXT
\{$cmd\}
END_TEXT

PGML, on the other hand, does not do iterated regular expression substitution, but instead actually parses the string and performs actions on the parts in a programmatic way. That means that the results of a substitution are not affected by later substitutions (there are no later substitutions). Your use of [@ $altTitle @] rather than the more natural (and completely equivalent) [$altTitle] suggests that you are trying to apply the TEXT substitution paradigm incorrectly to PGML, which doesn't work that way.

The four forms of substitution (with different numbers of stars) work as follow:

  • With no stars, the value of $altTitle are inserted into the resulting output with any special characters escaped so that they show up properly in whatever output mode you are using. That is, you should see exactly on screen or in hardcopy the characters that are in the content of the variable. In this case, that means the dollar signs and the names of the variables, as those are the content of the variable.

With one star, the contents of the variable are inserted into the output without change (no escaping is performed). This is intended for use with values that have already been formatted for output (e.g., they already contain HTML ready for the screen). No further processing of any kind (including variable substitution) is performed, so your HTML page would include the literal string alt="$setNumber:$questionNumber" in your case.

With two stars, the contents of the variable are further processed as PGML code, and the result of that is inserted into the output. Since there are no PGML commands in your string, it ends up being treated as plain text in PGML, where any special characters are escaped for the current output mode (i.e., it is effectively the same as no stars in this case). Note that PGML doesn't treat $setNumber as variable substitution; that would have to be [$setNumber]. So if your originally string were
$altTitle = 'alt="[$setNumber]:[$questionNumber]"'
then the double-star form would have inserted the values of these variables into the result, as you desired. Note, however, that it would also have quoted any HTML special characters, and since your $altTitle seems to be an HTML fragment, I suspect it is a simplified version of something where you are trying to create HTML by hand, which means you would need a single-star substitution, not a double-star one.

With three stars, the results are processed for LaTeX math delimiters (but no variable substitutions) and the results used verbatim (like for a single star). So none of these are expected to do the substitution that you are after (PGML never treats $var as variable substitution, whether that comes from another variable or not).

The confusing part to me is why you are putting off the variable substitution until the TEXT or PGML step rather than doing it when $altTitle is created originally (the most natural location for the substitution). Because you are using the single-quote form of a string literal, that prevents the variable substitution from occurring. So perhaps

$altTitle = "alt=\"$setNumber:$problemNumber\"";
would work better for you, or
$altTitle = qq/alt="$setNumber:$problemNumber"/;
if you prefer not to have to quote the quotation marks.

Note that PGML's double-star approach allows you to do variable substitution after command substitution, or command substitution after variable substitution (or even variable substitution after variable substitution, or command substitution after command substitution), so it is more flexible than TEXT's substitutions in that sense, but you have to use nested PGML commands, not TEXT commands. E.g.,

$x = 3;
$varsub = 'variable [$x]';
$cmdsub = 'command [@ $x+1 @]';

BEGIN_PGML
[$varsub]**, [$cmdsub]**

[@ $varsub @]**, [@ $cmdsub @]**
END_PGML
should produce the output
variable 3, command 4

variable 3, command 4
Indeed, you could nest as deeply as you like.

So the upshot is that PGML is doing exactly what it is supposed to, and that you probably want to do the variable substitution when you first create $altTitle, not during the PGML processing.

In reply to Davide Cervone

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
Bingo:

"Because you are using the single-quote form of a string literal, that prevents the variable substitution from occurring."

I've been learning Perl on a need-to-know basis, this is something I've missed or forgotten.

Thanks.

Btw, can we all agree, with the benefit of hindsight, that Perl was a tragic choice for WW's foundation?
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Alex Jordan -
> Btw, can we all agree, with the benefit of hindsight, that Perl was a tragic choice for WW's foundation?

I can't agree with that. It's extremely powerful and fast, once you are familiar with it.
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Alex Jordan -
But you aren't the first person to think so. I could be mistaken, but I believe this is what led David Lippman to build MyOpenMath, which I believe uses php instead of perl. (So many "I believe" disclaimers because I'm not very familiar with MyOpenMath).
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Davide Cervone -
I also don't agree. One must remember that WeBWorK is about 20 years old, and at the time it was first designed, there were not very many options, and it was certainly the best choice back then.

Any language takes getting used to, and perl certainly has its quirks, but it is fast, well documented, universally available, well maintained, and very powerful. It has many libraries that extend its capabilities, and was one of the early interpreted languages to have an object-oriented programming model (which was essential to MathObjects). Finally, its tight coupling to Apache through mod_perl was an important performance enhancement for WeBWorK2 that could not have been done in any other language at the time.

While there are other reasonable choices today, perl was certainly the right choice for much of WeBWorK's lifetime, and is still not a bad choice even today. Were I writing it now, I might have chosen javascript, because you can run the same code in both the browser and on the server, and that would provide some interesting possibilities.
In reply to Davide Cervone

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
I understand why perl was adopted 20 years ago: it was the hottest scripting language around, it was designed around text munging, and it had an unfair advantage in mod_perl. But none of the features David and Alex mentioned are still unique to Perl.

In fact Perl seems now to be in danger of losing mod_perl. I recently had to downgrade my Perl installation from version 5.22 to 5.18 because mod_perl doesn't support the newer version, and there's no upgrade in sight.

The way for webwork to grow is for teachers to be able to easily new problems. Perl is notorious for being hard to read and maintain. May I rant?

1. Does anyone really think that

sub foo{
my ( $x, $y ) = @_;
...

is in any way preferable to

function foo( x, y ){
... ?
2. The sigils are evil. It is way too easy for tyro like me to write $var instead of @var or vice-versa, which will cause the webwork page to fail even to load and produce a useless error message (i.e. no line number of the offending statement).

3. A language that had to wait till version 5 to get multidimensional arrays is not cut out for the 21st century.

4. The "cure" for #3, references, was almost worse than the disease. It's been said that Perl now has most of the downsides of c but none of the its advantages.

I fear that despite all the great contributions you have made to Webwork over the years that it is doomed to die a slow death. Younger people now entering mathematics teaching are going to be put off by Perl and instead opt for something like the project a bunch of grad students at my school are working on: examscram.com. It has a much simpler authoring language, which means that the currently tiny problem library has the potential to grow quickly.

In the 15 or so years since webwork started being used here at IU I believe that only one instructor has taken it upon himself to add problems to the bank, and he was the dept's webwork guru to start with. This despite frequent complaints that the problem bank was inadequate and the introduction of new courses with material not covered in the existing problem libraries. No one else here wants to deal with writing webwork code.

Except me. :) But I don't think my job security should really trump the good of the mathematics community.
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Davide Cervone -
While I agree in general with your comments 1 through 4, your original premiss was that Perl was a tragic choice for WeBWorK. The fact of the matter is that it was essentially the only choice at the time. Saying that it would not be what you would choose today doesn't make it tragic that it was chosen then. Unless you have another alternative that should have been chosen 20 years ago, anything you say about other languages now is immaterial.

If you are suggesting that PG should be rewritten to use a different language, there is certainly an argument that can be made there. But it would be a big investment to do that, and there is a lot of existing code and problems that would have to be converted to the new format. The advantages would have to be worth that effort, and you would need to have people interested in doing that work. Perhaps there are; if so, the WeBWorK community has always welcomed innovation.

My own feeling is that the real issue is that problem authors are writing PG code directly. I think there is a big need for a more structured editing system that would handle more of the details of the problem for the author. Most of the things authors do in that would not be direct programming. There would be places for some code snippets, but the language used for those doesn't have to be PG directly. It could be something transpired to PG that is easier for authors to work with.

Alternatively, there is nothing stopping one from embedding other languages in PG. I can see having PG be connected to a node.js service that would allow PG programs to process problems written in javascript, for example (something like this is already done with Sage now).

So if you have some other language you want to see be used for programming your problems, perhaps there is something there for you to contribute.
In reply to Davide Cervone

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
"The fact of the matter is that it was essentially the only choice at the time."

Yes, you're right about that. I was thinking that Python and Lua were options, but if Apache didn't support them they were impossible.

But what about the lack of mod_perl support for Perl 5.22? Is that something we should be worried about now?

My own feeling is that the real issue is that problem authors are writing PG code directly. I think there is a big need for a more structured editing system that would handle more of the details of the problem for the author.

Yes, that would be great. I assume that's the idea behind Math Objects and PGML? Unfortunately since most (all?) of the work is done by volunteers progress has been very slow. This summer I'm writing problems in game theory and graph theory, and there are no relevant Math Objects extant. I doubt that I'd be up to the task, but if one wanted to contribute new Math Objects how would one go about learning to do so?

I know absolutely nothing about server side programming (beyond the extent to which writing PG code is such), but it might be nice to get Lua or Python operative in WW. Lua in particular might enable some useful synergies since it is now the principal scripting language for Latex. I already have lots of Lua code for writing exams, it would be nice to be able to use it for WW problems.


In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Michael Gage -
Embedding Lua and using it as an input language sounds like an interesting idea, particularly given it's connection with TeX.  

It looks like there might  already be some work in this direction.

Have you looked at

http://search.cpan.org/~vparseval/Inline-Lua-0.03/lib/Inline/Lua.pm

or more thoroughly
http://search.cpan.org/~djerius/Lua-API-0.04/lib/Lua/API.pm

and
http://search.cpan.org/~djerius/Lua-API-0.04/lib/Lua/API.pm

There are a few other projects as well. 

-- Mike


In reply to Michael Gage

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
Nice, I'd like to try inline-lua on my desktop WW installation. I'll have to add a line to defaults.config. Anything else I should be aware of?
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Michael Gage -
Its cleaner to add it to localOverrides.conf

push ${pg}{modules}, [qw(Inline::Lua)];

(you want to add an entry, not overwrite the entire $(pg){modules} anonymous array.  )

(older versions of perl require manual dereferencing of the array reference:

push @{ ${pg}{modules} }, [qw(Inline::Lua)];

)

After that you will have to deal with the fact that Lua is now operating within a Safe compartment which restricts the kind of operations that are allowed. That may or may not cause difficulties. With creativity these difficulties can usually be overcome but there is no easy way to give advice in advance. 

It could be that the best way to use Inline::Lua is as an external editor that spits out PG code that can then be imported by hand as a PG problem. This avoids the Safe compartment problem -- at least initially -- and would allow you to explore whether useful problems could be written this way.  If that proves fruitful then one could struggle with embedding the Lua compiler directly into the Safe compartment or perhaps elsewhere in the WeBWorK software stack such as inside the editor.  

I think this is an interesting project and has a lot of potential but I have several things on my  plate that need to be cleared before I would be willing to dive deeply into this.  Please keep us informed as to what you find out.

-- Mike 


In reply to Michael Gage

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
Should I assume from your silence that a) the future of support for mod_perl is indeed something we should worried about and b) you don't really want anyone contributing Math Objects?
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Davide Cervone -
No, you should assume that I am very busy, and as a volunteer, I have to budget how much time I spend answering questions, which could take up as much time as there is. I frequently don't get to answer questions immediately, and there are people waiting for responses from me for quite a while. Four days is not really enough to draw any conclusions.

The mod_perl 2.0.9 release notes indicate that "Note that Perl 5.22.x is currently not supported. This is logged as CPAN RT#101962 and will hopefully be addressed in 2.0.10" So it looks like there are plans for support in the future. I don't know their release schedule, but since 2.0.9 was just released in June, it may be a while before 2.0.10.

As for MathObjects, I'm happy to have people contribute. The bad thing is that, while there is good documentation for problem authors, there is virtually none for the internals of MathObjects. This is certainly a glaring deficiency, but producing such documentation would be about a month's worth of work, and I simply can't afford to volunteer that effort. So you would have to rely on looking at the source code, and use the various context and parser files in the pg/macros as examples. The pg/macros/contextPartition.pl might be a good one to look at as it is not all that complicated, and pg/macros/contextPermutation.pl might be a second good example.

Without much perl coding experience, however, I suspect it may be difficult to figure out what is going on.
In reply to Davide Cervone

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -
Btw, I want to thank all you contributors for helping me out this summer, which by my modest standards was a very productive one. I wrote almost 40 new problems in graph and game theories, with maybe 1000 lines of accompanying macros. Without your prompt and patient help I would have been hospitalized before breaking three.

Thank you!
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Davide Cervone -
There are some examples in the docs/MathObjects folder of the pg directory. They are pretty old, but do include some comments about what is going on.
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Andrew Dabrowski -

I really thought you had solved the problem, but I tried your first suggestion (code below) and that produces an error also.

Edit: What does work is qq(alt="$setNumber:$questionNumber"). I'm still puzzled why the escapes don't do the trick.

PG question failed to render

Unable to obtain error messages from within the PG question.
WeBWorK Warnings

WeBWorK has encountered warnings while processing your request. If this occured when viewing a problem, it was likely caused by an error or ambiguity in that problem. Otherwise, it may indicate a problem with the WeBWorK system itself. If you are a student, report these warnings to your professor to have them corrected. If you are a professor, please consult the warning output below for more information.
Warning messages

Scalar found where operator expected at line 22 of (eval 4160), near ""alt=\\"$setNumber"
(Missing operator before $setNumber?)
Backslash found where operator expected at line 22 of (eval 4160), near "$questionNumber\"
(Missing operator before \?)
Processing of this PG problem was not completed. Probably because of a syntax error.
The translator died prematurely and no PG warning messages were transmitted. at /home/dabrowsa/webwork/webwork2/lib/WeBWorK/ContentGenerator/Problem.pm line 700.

Request information

Time Fri Aug 14 14:18:26 2015
Method GET
URI /webwork2/NewGenEd/Undefined_Set/1/


Now what am I doing wrong?

DOCUMENT();

loadMacros("PGstandard.pl",
"PGML.pl",
);

TEXT(&beginproblem);

$altTitle = "alt=\"$setNumber:$questionNumber\"";

BEGIN_TEXT
\{ $altTitle \}
$BR$BR
END_TEXT

BEGIN_PGML
[@ $altTitle @]
[@ $altTitle @]*
[@ $altTitle @]**
[@ $altTitle @]***

END_PGML
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Michael Gage -
Ahh.  That's a PG compromise that often catches me as well.  Because TeX is used so frequently in PG problems and because backslash is not an escape in the TeX language but part of the code, some preprocessing is done to make all backslashes appear as literal backslashes (useful for TeX).  The perl backslash (which is used less often in PG code) has been replaced by ~~  .  See if something like  ~~" works.  

This holds in the .pg files.  For the macro (.pl ) files the reverse is true -- there is much more perl and little TeX.  In those files backslash works as you expect for perl and any tex commands have to be written using double backslashes -- e.g.  \\par. 

That said I prefer the qq!................! double quoting scheme available in perl (and in Ruby) since I'm always forgetting to escape at least one interior quote.)
 
In the original version of PG this ability to change the quote character at will was one of the handiest features of writing problems in perl.  It was a great improvement over the original CAPA which was the homework system (used at the time mostly for physics and the sciences) that inspired the creation of WeBWorK.  CAPA has since evolved into lon-capa, acquiring in the process a lot of the features in earlier versions of WeBWorK.  

-- Mike
In reply to Andrew Dabrowski

Re: PGML and $questionNumber: possible bug?

by Davide Cervone -
Mike has it right, the command should have been
$altTitle = "alt=~~"$setNumber:$questionNumber~~"";
My apologies. As Mike said, we get caught by this often as well.