rishi/scryer-prolog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #913 from panasenco/json

Browse Source 
Another thorough rewrite of library(json) to better preserve the complete set of answers as much as possible
This commit is contained in:
Mark Thom
2021-04-26 13:34:04 -06:00
committed by GitHub
parent 7bec37bfd8 02b89362b2
commit 48225c3c0a
5 changed files with 149 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
src/lib/dcgs.pl
View File

@@ -1,7 +1,9 @@
:- module(dcgs,
[op(1105, xfy, '|'),
phrase/2,
phrase/3]).
phrase/3,
seq//1,
seqq//1]).
:- use_module(library(error)).
:- use_module(library(lists), [append/3]).
@@ -178,3 +180,11 @@ user:term_expansion(Term0, Term) :-
nonvar(Term0),
dcg_rule(Term0, (Head :- Body)),
Term = (Head :- Body).
% Describes a sequence
seq([]) --> [].
seq([E|Es]) --> [E], seq(Es).
% Describes a sequence of sequences
seqq([]) --> [].
seqq([Es|Ess]) --> seq(Es), seqq(Ess).

208
src/lib/json.pl
View File

@@ -37,14 +37,9 @@
json_chars//1
]).
:- use_module(library(assoc)).
:- use_module(library(between)).
:- use_module(library(charsio)).
:- use_module(library(dcgs)).
:- use_module(library(dif)).
:- use_module(library(error)).
:- use_module(library(lists)).
:- use_module(library(reif)).
/* The DCGs are written to match the McKeeman form presented on the right side of https://www.json.org/json-en.html
as closely as possible. Note that the names in the McKeeman form conflict with the pictures on the site. */
@@ -76,7 +71,6 @@ json_object([Pair|Pairs]) -->
This is a pure performance-driven decision that doesn't affect the logic. The predicate could equivalently be
implementes as `json_members//1` below:
```
json_members([Key-Value]) --> json_member(Key, Value).
json_members([Key-Value, Pair2 | Pairs]) --> json_member(Key, Value), ",", json_members([Pair2 | Pairs]).
```
That's a logically equivalent and equally clean representation to the lagged argument. However, it leaves
@@ -107,106 +101,98 @@ json_string(Chars) --> "\"", json_characters(Chars), "\"".
json_characters("") --> "".
json_characters([Char|Chars]) --> json_character(Char), json_characters(Chars).
letter_escape('"', '"').
letter_escape('\\', '\\').
letter_escape('/', '/').
letter_escape('\b', 'b').
letter_escape('\f', 'f').
letter_escape('\n', 'n').
letter_escape('\r', 'r').
letter_escape('\t', 't').
/* Note on variable instantiation checks (`var/1` and `nonvar/1`) used below and in Prolog in general.
Instantiation checks should ideally never be used to change the logic of your program. Instead, they are one of
Instantiation checks should never be used to change the logic of your program. Instead, they are one of
many tools to adjust the 'control' or 'search strategy' used by Prolog to execute the logic of your program.
For a general overview of the idea, read Bob Kowalski's "Algorithm = Logic + Control":
https://www.doc.ic.ac.uk/~rak/papers/algorithm%20=%20logic%20+%20control.pdf
For an introduction to search strategies in Prolog, read: https://www.metalevel.at/prolog/sorting#searching
However, when dealing with a real-world data format standard, real differences arise in how a string should be
parsed vs generated. Usually, parsing should allow multiple ways of doing things, while generating should only
happen in one best way.
JSON characters are parsed/generated in one of three ways:
1. Directly. All characters in the range 20.10FFFF, except '"' and '\\' must be generated and parsed directly,
escape for the forward slash '/', which must not be generated directly, but can be parsed directly.
2. Backslash followed by a single special character defined in the escape map - both parsing and generating.
3. Backslash followed by 'u' and 4 hex values defining the character code of the internal character.
When generating, only allow range 0.20 excepting characters in the escape map.
When parsing, allow any value.
In order to take advantage of first argument indexing, we must reify this distinction in a single predicate. */
json_character(InternalChar) -->
{ ( nonvar(InternalChar) ->
( letter_escape(InternalChar, _) ->
Type = letter_escape
; char_code(InternalChar, InternalCharCode),
( InternalCharCode >= 32 ->
Type = direct
; Type = hex_escape
)
)
; true
) },
json_character(Type, InternalChar).
It's tempting to use instantiation checks to be more strict while generating and more relaxed while parsing.
In fact, the early version of this library aimed to return exactly one result when generating. However, doing that
is **wrong** and leads to difficult-to-catch bugs. Instead, adjust the search strategy to return the most ideal
and strictest answer FIRST and then return less ideal answers on backtracking.
As an example, consider a string containing just the forward slash. The JSON standard recommends the forward slash
be escaped with a backslash, but allows it to not be escaped. Attempting to force stricter behavior with
instantiation checks can lead to this confusing mess:
```
phrase(json:json_characters("/"), External).
External = "\\/".
?- phrase(json:json_characters(Internal), "/").
Internal = "/"
; false.
?- phrase(json:json_characters("/"), "/").
false.
```
To avoid such bugs, never use instantiation checks to reduce the number of right answers, but rather to adjust
the *path* used to traverse those answers. */
json_character(direct, PrintChar) --> [PrintChar].
json_character(letter_escape, EscapeChar) -->
{ letter_escape(EscapeChar, PrintChar) },
escape_char('"', '"').
escape_char('\\', '\\').
escape_char('/', '/').
escape_char('\b', 'b').
escape_char('\f', 'f').
escape_char('\n', 'n').
escape_char('\r', 'r').
escape_char('\t', 't').
json_character(EscapeChar) -->
{ escape_char(EscapeChar, PrintChar) },
"\\",
[PrintChar].
json_character(hex_escape, EscapeChar) -->
json_character(PrintChar) -->
[PrintChar],
{ dif(PrintChar, '\\'),
dif(PrintChar, '"'),
char_code(PrintChar, PrintCharCode),
PrintCharCode >= 32 }.
json_character(EscapeChar) -->
"\\u",
{ ( nonvar(EscapeChar) ->
char_code(EscapeChar, EscapeCharCode),
H1 = 0,
H2 = 0,
H3 is EscapeCharCode // 16,
H4 is EscapeCharCode mod 16
; true
) },
json_hex(H1),
json_hex(H2),
json_hex(H3),
json_hex(H4),
{ ( var(EscapeChar) ->
{ ( nonvar(H1) ->
EscapeCharCode is H1 * 16^3 + H2 * 16^2 + H3 * 16 + H4,
char_code(EscapeChar, EscapeCharCode)
; true
; char_code(EscapeChar, EscapeCharCode),
H1 is (EscapeCharCode // 16^3) mod 16,
H2 is (EscapeCharCode // 16^2) mod 16,
H3 is (EscapeCharCode // 16^1) mod 16,
H4 is (EscapeCharCode // 16^0) mod 16
) }.
json_hex(Value) -->
{ ( nonvar(Value) ->
( between(0, 9, Value) ->
Code is Value + 48
; ( between(10, 15, Value) ->
Code is Value + 87
; false
)
),
char_code(Char, Code)
; true
)
},
[Char],
{ ( var(Value) ->
char_code(Char, Code),
( between(48, 57, Code) ->
Value is Code - 48
; ( between(65, 70, Code) ->
Value is Code - 55
; ( between(97, 102, Code) ->
Value is Code - 87
; false
)
)
)
; true
) }.
json_hex(Digit) --> json_digit(Digit).
json_hex(10) --> "a".
json_hex(11) --> "b".
json_hex(12) --> "c".
json_hex(13) --> "d".
json_hex(14) --> "e".
json_hex(15) --> "f".
json_hex(10) --> "A".
json_hex(11) --> "B".
json_hex(12) --> "C".
json_hex(13) --> "D".
json_hex(14) --> "E".
json_hex(15) --> "F".
/* I can't think of any alternatives to using `number_chars/2` when generating, though this leads
to under-reporting of correct solutions. At least matching solutions unify when both are instantiated...
```
?- phrase(json:json_number(N), "123E2").
N = 12300
; false.
?- phrase(json:json_number(12300), Cs).
Cs = "12300".
?- phrase(json:json_number(12300), "123E2").
true
; false.
```
*/
parsing, [C] --> [C], { nonvar(C) }.
/* Here we are going to simply rely on `number_chars/2` when generating. */
json_number(Number) -->
( { nonvar(Number) } ->
{ number_chars(Number, NumberChars) },
NumberChars
; json_sign_noplus(Sign),
( parsing ->
json_sign_noplus(Sign),
json_integer(Integer),
json_fraction(Fraction),
json_exponent(Exponent),
@@ -215,6 +201,8 @@ json_number(Number) -->
; Base = 10.0
),
Number is Sign * (Integer + Fraction) * Base ^ Exponent }
; { number_chars(Number, NumberChars) },
NumberChars
).
json_integer(Digit) --> json_digit(Digit).
@@ -230,6 +218,9 @@ json_digits(Value, Power) -->
{ Power is NextPower + 1,
Value is FirstDigit * 10^Power + RemainingValue }.
json_digit(0) --> "0".
json_digit(Digit) --> json_onenine(Digit).
json_onenine(1) --> "1".
json_onenine(2) --> "2".
json_onenine(3) --> "3".
@@ -240,29 +231,18 @@ json_onenine(7) --> "7".
json_onenine(8) --> "8".
json_onenine(9) --> "9".
json_digit(0) --> "0".
json_digit(1) --> "1".
json_digit(2) --> "2".
json_digit(3) --> "3".
json_digit(4) --> "4".
json_digit(5) --> "5".
json_digit(6) --> "6".
json_digit(7) --> "7".
json_digit(8) --> "8".
json_digit(9) --> "9".
json_fraction(0) --> "".
json_fraction(Fraction) -->
".",
json_digits(Value, Power),
{ Fraction is Value / 10 ^ (Power + 1) }.
".",
json_digits(Value, Power),
{ Fraction is Value / 10.0 ^ (Power + 1) }.
json_exponent(0) --> "".
json_exponent(Exponent) -->
json_exponent_signifier,
json_sign(Sign),
json_digits(Value, _),
{ Exponent is Sign * Value }.
json_exponent_signifier,
json_sign(Sign),
json_digits(Value, _),
{ Exponent is Sign * Value }.
json_exponent_signifier --> "E".
json_exponent_signifier --> "e".
@@ -273,6 +253,18 @@ json_sign_noplus(-1) --> "-".
json_sign(Sign) --> json_sign_noplus(Sign).
json_sign(1) --> "+".
/* Make sure json_ws doesn't attempt to generate whitespace and succeeds without choicepoints when generating */
json_ws --> [C], {nonvar(C), member(C, " \n\r\t")}, json_ws.
json_ws --> "".
/* Make `json_ws/0` greedy when parsing, lazy when generating */
json_ws_empty --> "".
json_ws_nonempty --> " ".
json_ws_nonempty --> "\n".
json_ws_nonempty --> "\r".
json_ws_nonempty --> "\t".
json_ws_greedy --> json_ws_nonempty, json_ws_greedy.
json_ws_greedy --> json_ws_empty.
json_ws_lazy --> json_ws_empty.
json_ws_lazy --> json_ws_nonempty, json_ws_lazy.
json_ws -->
( parsing ->
json_ws_greedy
; json_ws_lazy
).

16
src/tests/json/README.md
View File

@@ -1,5 +1,8 @@
## Benchmarks
These are honestly super flawed (single read/write of a single file) but that can at least capture movement on the
order of 0.1 s, and that's good enough for me.
### Read
With CLP(Z):
@@ -22,6 +25,12 @@ With first argument indexing optimizations:
% CPU time: 0.310 seconds
```
After making the code more general:
```
?- test_json:test_json_read.
% CPU time: 0.217 seconds
```
### Write
Without first argument indexing optimizations:
@@ -31,8 +40,13 @@ Without first argument indexing optimizations:
```
With first argument indexing optimizations:
```
?- test_json:test_json_minify.
% CPU time: 0.015 seconds
```
After making the code more general:
```
?- test_json:test_json_minify.
% CPU time: 0.013 seconds
```

2
src/tests/json/pass_everything.min.json
View File

@@ -1 +1 @@
["JSON Test Pattern pass1",{"object with 1 member":["array with 1 element"]},{},[],-42,true,false,null,{"":23456789012000000000000000000000000000000000000000000000000000000000000000000," s p a c e d ":[1,2,3,4,5,6,7],"# -- --> *\/":" ","\/\\\"쫾몾ꮘﳞ볚\b\f\n\r\t`1~!@#$%^&*()_+-=[]{}|;:',.\/<>?":"A key can be any string","ALPHA":"ABCDEFGHIJKLMNOPQRSTUVWYZ","E":12345678900000000000000000000000000.0,"address":"50 St. James Street","alpha":"abcdefghijklmnopqrstuvwyz","array":[],"backslash":"\\","comment":"\/\/ \/* <!-- --","compact":[1,2,3,4,5,6,7],"controls":"\b\f\n\r\t","digit":"0123456789","e":0.000000000000123456789,"false":false,"hex":"ģ䕧覫\u0001췯ꯍ\u001a","integer":1234567890,"jsontext":"{\"object with 1 member\":[\"array with 1 element\"]}","null":null,"object":{},"one":1,"quote":"\"","quotes":"&#34; \" %22 0x22 034 &#x22;","real":-9876.54321,"slash":"\/ & \/","space":" ","special":"`1~!@#$%^&*()_+-={':[,]}|;.<\/>?","true":true,"url":"http:\/\/www.JSON.org\/","zero":0},0.5,98.6,99.44,1066,"rosebud"]
["JSON Test Pattern pass1",{"object with 1 member":["array with 1 element"]},{},[],-42,true,false,null,{"integer":1234567890,"real":-9876.54321,"e":0.000000000000123456789,"E":12345678900000000000000000000000000.0,"":23456789012000000000000000000000000000000000000000000000000000000000000000000,"zero":0,"one":1,"space":" ","quote":"\"","backslash":"\\","controls":"\b\f\n\r\t","slash":"\/ & \/","alpha":"abcdefghijklmnopqrstuvwyz","ALPHA":"ABCDEFGHIJKLMNOPQRSTUVWYZ","digit":"0123456789","special":"`1~!@#$%^&*()_+-={':[,]}|;.<\/>?","hex":"ģ䕧覫\u0001췯ꯍ\u001a","true":true,"false":false,"null":null,"array":[],"object":{},"address":"50 St. James Street","url":"http:\/\/www.JSON.org\/","comment":"\/\/ \/* <!-- --","# -- --> *\/":" "," s p a c e d ":[1,2,3,4,5,6,7],"compact":[1,2,3,4,5,6,7],"jsontext":"{\"object with 1 member\":[\"array with 1 element\"]}","quotes":"&#34; \" %22 0x22 034 &#x22;","\/\\\"쫾몾ꮘﳞ볚\b\f\n\r\t`1~!@#$%^&*()_+-=[]{}|;:',.\/<>?":"A key can be any string"},0.5,98.6,99.44,1066,"rosebud"]

34
src/tests/json/test_json.pl
View File

@@ -2,10 +2,12 @@
:- use_module(library(charsio)).
:- use_module(library(dcgs)).
:- use_module(library(format)).
:- use_module(library(json)).
:- use_module(library(lists)).
:- use_module(library(os)).
:- use_module(library(pio)).
:- use_module(library(iso_ext)).
:- use_module(library(time)).
test_path(TestName, TestPath) :-
@@ -18,23 +20,31 @@ name_parse(Name, Json) :-
once(phrase_from_file(json_chars(Json), Path)).
test_json_read :-
name_parse("pass_null.json", _),
name_parse("pass_alnum.json", _),
name_parse("pass_special.json", _),
name_parse("pass_mandatory_escapes.json", _),
name_parse("pass_forward_slash.json", _),
name_parse("pass_hex.json", _),
name_parse("pass_smallfloat.json", _),
name_parse("pass_bigfloat.json", _),
name_parse("pass_null.json", null),
name_parse("pass_alnum.json", string("ABCDEFGHIJKLMNOPQRSTUVWYZabcdefghijklmnopqrstuvwyz0123456789")),
name_parse("pass_special.json", string("`1~!@#$%^&*()_+-={':[,]}|;.</>?")),
name_parse("pass_mandatory_escapes.json", string(" \" \\ \b\f\n\r\t ")),
name_parse("pass_forward_slash.json", string("/ & /")),
name_parse("pass_hex.json", string("ģ\x4567\\x89ab\\xcdef\\xabcd\\xef4a\")),
name_parse("pass_smallfloat.json", number(0.000000000000123456789)),
name_parse("pass_bigfloat.json", number(12345678900000000000000000000000000.0)),
time(name_parse("pass_everything.json", _)).
minify_sample_json :-
name_parse("pass_everything.json", Json),
time(once(phrase(json_chars(Json), MinChars))),
test_path("pass_everything.min.json", MinPath),
setup_call_cleanup(
open(MinPath, write, Stream),
format(Stream, "~s", [MinChars]),
close(Stream)
).
test_json_minify :-
test_path("pass_everything.min.json", MinPath),
open(MinPath, read, RefMin),
read_line_to_chars(RefMin, RefChars, []),
close(RefMin),
once(phrase_from_file(seq(RefChars), MinPath)),
name_parse("pass_everything.json", Json),
time(phrase(json_chars(Json), MinChars)),
time(once(phrase(json_chars(Json), MinChars))),
RefChars = MinChars.
test_json_int_float :-