Updated 2018-03-31 02:27:16 by SEH

JSON is a lightweight, text-based, language-independent data interchange format.

See also  edit

JSON-RPC
JSON is also used as the data format for a lightweight RPC protocol

Alternatives  edit

Tclon
YAML
TDL
vrtcl

Implementations  edit

A Tcl extension written in C for parsing JSON, by Mikhail Teterin
announcement, comp.lang.tcl, 2014-01-14. This extension made its way into Tcllib, with slight modifications.
jimhttp
Contains a pure-Tcl implementation of a JSON generator and parser for Jim Tcl and Tcl 8.5+. It can treat JSON arrays as either lists or dicts with integers 0, 1, 2... for keys. The JSON generator part supports [schema]s with which you can specify things like whether the Tcl string "true" should be converted to a JSON string or a JSON boolean.
jimson
a jimhttp-inspired json stringifier hacked up in response to an excellent discussion on handling types
jmsn
Jim bindings for jsmn.
jq
A JSON-parsing command line tool that can be called from Tcl.
JSON, Rosetta Code
includes code for translating Tcl data structures to json using 8.6's ::tcl::unsupported::representation feature:
rl_json, courtesy of Ruby Lane
Provides a json command to access and modify json values with an interface similar to dict and a template mechanism for generating json values. Performance similar to or better than yajl-tcl.
SQLite extension JSON1
An SQLite extension for manipulating JSON.
tcl-duktape
a JavaScript interpreter library that includes a TclOO interface for manipulating JSON as JavaScript objects.
tcl-json
provides a typed version of a scanner, parser and encoder for the JSON format. Scanner and parser make use of the yeti scanner and parser generator tools.
tcljsonnet
Tcl wrapper for Jsonnet library
Tcllib JSON
ton
Tcl Object Notation, a fast JSON parser, amongst other things
yajl-tcl, by Karl Lehenbauer
a Tcl binding for yajl (yet another json library). Purportedly much faster than Tcllib json or huddle.
Alternative JSON
Yet another pure-Tcl JSON implementation with a strong emphasis on preserving type information.

Comparison

JSON value extraction benchmark
Compares the performance of jq, jimhttp JSON, Tcllib JSON, rl_json, SQLite's JSON1, tcl-duktape and yajl-tcl when extracting a value from a large JSON blob.

Documentation  edit

RFC 7158, The Javascript Object Notation (JSON) Data Interchange Format
JSON Example

Reference  edit

What is JSON, JSON-RPC and JSON-RPC-Java?, Slashdot, 2005-01-24

Description  edit

The JSON site probably explains it best:
"JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999. This feature can also be found in Python. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, TCL, and many others. These properties make JSON an ideal data-interchange language."

string is double gotcha  edit

[gchung] 2012-11-06: Tcl's string is double does not conform to JSON specification. For testing valid numbers, you should use the following regexp:
proc is_valid_json_number {value} {
    regexp -- {-?(?:[1-9][[:digit:]]*|0)(?:\.[[:digit:]]+)?(?:[eE][+-]?[[:digit:]]+)?} $value
}

Using string is double allows non-valid numbers to slip through. E.g. "00", "0x0", "0.".

AMG: That regular expression may be fine for limiting the acceptable number forms, but also continue to use string is double to check for over/underflow.

JSON in OpenACS  edit

OpenACS has its own take on handling JSON data, see [1]. From the intro:
   Utility ad_procs for Tcl <-> JSON conversion.

   This is based on the tcllib json package written by Andreas Kupries, and
   later rewritten to parse via regular expressions by Thomas Maeder.

   The tcllib version suffers from generating Tcl structures from JSON strings
   with no type (JSON array or object) information.  The resulting structures can't
   be converted back to JSON strings, you have to munge them with type information
   first.  And the code making use the Tcl structure also needs to know whether each
   field is an object or array.

   It also depends on the DICT package or Tcl 8.5.

   This rewrite doesn't depend on DICT, declares procs using ad_proc (so the API
   will be picked up by our API browser), and is symmetrical (you can convert from
   JSON to the Tcl representation and back again).

Misc  edit


JMN 2009-01: The way I see it, a JSON generator is a little more useful than a JSON parser. The primary usecase I've had for JSON is pumping a chunk of structured data to the client where there is a JavaScript engine available to consume it.

Whilst in some scenarios JSON-formatted structured data might also be sent to the server where another language such as Tcl or PHP might need to parse it - it seems more common that the server-side language gets its data as simple values already split into the fields of a query string or form POST data.

TCV 2009-03-08: I've been solving this problem lately myself, the primary difficulty being how to encode the multiple types which are available in JSON in an ambiguously typed Tcl value. The solution for me is to make all values lists where the first element is the type name and the rest are the data for that type. Objects are tagged "obj" and then followed by alternating keys and values (the keys are strings so there's no reason to tag them, but values must be). Lists are tagged "list" and every subsequent element is an item in the list. Strings are "string", numbers are "num", and booleans are "bool". The special "null" when appearing alone is null. I have some preliminary code, not quite ready for release, on my website [2].

slebetman 2009-04-01: The huddle package in tcllib does the tagged data thing quite well. And huddle has a built-in jsondump subcommand. I've been working on a different approach though. Having tagged data means I either have to jump through hoops to access the data in Tcl or rely on third party APIs (can't use regular list and dict commands for example). What I've done instead is to implement a JSON compiler. Given a specification of the data structure, and the tcl data, generate a JSON string:
# data is plain old tcl values
# spec is defined as follows:
# {string} - data is simply a string, "quote" it if it's not a number
# {list} - data is a tcl list of strings, convert to JSON arrays
# {list list} - data is a tcl list of lists
# {list dict} - data is a tcl list of dicts
# {dict} - data is a tcl dict of strings
# {dict xx list} - data is a tcl dict where the value of key xx is a tcl list
# {dict * list} - data is a tcl dict of lists
# etc..
proc compile_json {spec data} {
    while [llength $spec] {
        set type [lindex $spec 0]
        set spec [lrange $spec 1 end]
        
        switch -- $type {
            dict {
                lappend spec * string
                
                set json {}
                foreach {key val} $data {
                    foreach {keymatch valtype} $spec {
                        if {[string match $keymatch $key]} {
                            lappend json [subst {"$key":[
                                compile_json $valtype $val]}]
                            break
                        }
                    }
                }
                return "{[join $json ,]}"
            }
            list {
                if {![llength $spec]} {
                    set spec string
                } else {
                    set spec [lindex $spec 0]
                }
                set json {}
                foreach {val} $data {
                    lappend json [compile_json $spec $val]
                }
                return "\[[join $json ,]\]"
            }
            string {
                if {[string is double -strict $data]} {
                    return $data
                } else {
                    return "\"$data\""
                }
            }
            default {error "Invalid type"}
        }
    }
}

# Usage:
% compile_json {dict * list} {a {1 2 3} b {4 5}}
{"a":[1,2,3],"b":[4,5]}

# Data may be nested:
% compile_json {dict * {list {dict d list}}} {a {{c 1} {d {2 2 2} e 3}} b {{f 4 g 5}}}
{"a":[{"c":1},{"d":[2,2,2],"e":3}],"b":[{"f":4,"g":5}]}

For the last example, the specification reads as:
{
    dict <---------------- a dict
        * {list     <------- where all (*) values are lists
            {dict     <----- of dicts
                d list  <----- where the value of "d" is a list
            }                and all other values are strings (default)
          }
}

Steve Bennett 2010-10-13: I independently came up with something almost identical. My version correctly escapes string values using:
return \"[string map [list \\ \\\\ \" \\" \n \\n / \\/ \b \\b \r \\r \t \\t] $value]\"

Also, rather than automatically deciding whether to encode a string or a number, I use 'num' as a type. This also allows the special values false, true and null to be encoded as bare values rather than being quoted.

kanryu 2009-04-15: It's good way! I implement one for huddle as "huddle compile" :)
% huddle jsondump [huddle compile {dict * {list {dict d list}}} {a {{c 1} {d {2 2 2} e 3}} b {{f 4 g 5}}}] {} {} 
{"a":[{"c":1},{"d":[2,2,2],"e":3}],"b":[{"f":4,"g":5}]}

slebetman 2009-04-12: That's great kanryu. However I'd prefer this to be implemented as part of the json package rather than huddle. It would avoid having to do the conversion twice (dict->huddle->json). Could the maintainer of the json package in tcllib please steal my code and make a nice json::compile command? Pretty please..? ;-)

Some tests of json package using examples given on the JSON page.
if {[info script] eq $argv0} {
    set examples {
        {
            "glossary": {
                "title": "example glossary",
                "GlossDiv": {
                    "title": "S",
                    "GlossList": [{
                        "ID": "SGML",
                        "SortAs": "SGML",
                        "GlossTerm": "Standard Generalized Markup Language",
                        "Acronym": "SGML",
                        "Abbrev": "ISO 8879:1986",
                        "GlossDef": 
                        "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML", "markup"]}]}}
        }
        
        {"menu": {
            "id": "file",
            "value": "File:",
            "popup": {
                "menuitem": [
                    {"value": "New", "onclick": "CreateNewDoc()"},
                    {"value": "Open", "onclick": "OpenDoc()"},
                    {"value": "Close", "onclick": "CloseDoc()"}
                ]
            }
        }
        }
        
        {"widget": {
            "debug": "on",
            "window": {
                "title": "Sample Konfabulator Widget",
                "name": "main_window",
                "width": 500,
                "height": 500},
            "image": { 
                "src": "Images/Sun.png",
                "name": "sun1",
                "hOffset": 250,
                "vOffset": 250,
                "alignment": "center"},
            "text": {
                "data": "Click Here",
                "size": 36,
                "style": "bold",
                "name": "text1",
                "hOffset": 250,
                "vOffset": 100,
                "alignment": "center",
                "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
            }
        }
        }            
    
        {"menu": {
            "header": "SVG Viewer",
            "items": [
                {"id": "Open"},
                {"id": "OpenNew", "label": "Open New"},
                null,
                {"id": "ZoomIn", "label": "Zoom In"},
                {"id": "ZoomOut", "label": "Zoom Out"},
                {"id": "OriginalView", "label": "Original View"},
                null,
                {"id": "Quality"},
                {"id": "Pause"},
                {"id": "Mute"},
                null,
                {"id": "Find", "label": "Find..."},
                {"id": "FindAgain", "label": "Find Again"},
                {"id": "Copy"},
                {"id": "CopyAgain", "label": "Copy Again"},
                {"id": "CopySVG", "label": "Copy SVG"},
                {"id": "ViewSVG", "label": "View SVG"},
                {"id": "ViewSource", "label": "View Source"},
                {"id": "SaveAs", "label": "Save As"},
                null,
                {"id": "Help"},
                {"id": "About", "label": "About Adobe CVG Viewer..."}]}
        }
    }
    
    foreach example $examples {
    puts "convert: $example"
    puts [json::json2dict "\{$example\}"]
    }
}

RLH: I do a package require JSON on OSX and the library isn't there or my setup is broken?

JH: The package name is "json".

RLH: Is there any kind of "rule" about package names? I should have tried the lowercase json but didn't think of it. Just wondering.

RLH: I found it after doing a find. I am stupid sometimes.

HolgerJ 2008-08-14: when I do a package require json, I get a message "can't find package dict". I checked the Tcl version (Kubuntu Hardy 8.04) and found out it was 8.4.16; after installing 8.5.0 and calling the interpreter with tclsh8.5, it seems to work.

glennj: you can get the dict package for tcl 8.4: see the dict page

"tom.rmadilo" writes on comp.lang.tcl:
I put up a JSON text to Tcl list converter for my own testing, if anyone wants to see what the Tcl format looks like:

wttp://www.rmadilo.com/json/

It also prints out debugging info on a char-by-char basis

There is also http://code.google.com/p/tcl-json/, a package using Yeti.

An example of http and JSON doing PHP page queries IP-geolocation

[Antender] 2010-12-25 13:51:59:

I'll post here my 1 line JSON parser which has such advantages:

  • Parses JSON directly in dict structure
  • Handles all JSON datatypes: arrays, objects, strings, other values
  • Has O(N) difficulty, because of the use of strict replaces

Reformatted version by Antender and AMG. It's indented for readability, it combines many of the regsubs into a string map, and uses a single string range instead of two executions of string replace.
proc json2dict JSONtext {
string range [
    string trim [
        string trimleft [
            string map {\t {} \n {} \r {} , { } : { } \[ \{ \] \}} $JSONtext
            ] {\uFEFF}
        ]
    ] 1 end-1
}

Antender: Added dict2json. Have no support of JSON arrays due to Tcl's dynamical typing.
proc dict2json {dictionary} {
    dict for {key value} $dictionary {
        if {[string match {\[*\]} $value]} {
            lappend Result "\"$key\":$value"
        } elseif {![catch {dict size $value}]} {
            lappend Result "\"$key\":\"[dict2json $value]\""
        } else {
            lappend Result "\"$key\":\"$value\""
        }
    }
    return "\{[join $Result ",\n"]\}"
}

dbohdan 2014-06-01: Unfortunately, because of how it uses string map the above implementation of json2dict will corrupt URLs stored in JSON values. A minimal example that demonstrates the problem:
eltclsh > set j {{"results": [{"artworkUrl512":"http://a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png"}]}}
{"results": [{"artworkUrl512":"http://a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png"}]}
eltclsh > puts [::json::json2dict $j]
results {{artworkUrl512 http://a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png}}
eltclsh > puts [json2dict $j]
"results"  {{"artworkUrl512" "http //a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png"}}

dbohdan 2014-06-01: I found the following function useful for accessing JSON data converted with ::json::json2dict.
proc jsonget {json args} {
    foreach key $args {
        if {[dict exists $json $key]} {
            set json [dict get $json $key]
        } elseif {[string is integer $key]} {
            if {$key >= 0 && $key < [llength $json]} {
                set json [lindex $json $key]
            } else {
                error "can't get item number $key from {$json}"
            }
        } else {
            error "can't get \"$key\": no such key in {$json}"
        }
    }
    return $json
}

The motivation behind it is that right now ::json::json2dict converts JSON arrays into lists. As a result you need to chain calls to lindex and dict get to access values stored inside complexly structured JSON blobs. There is an issue on the Tcllib tracker to add the option -indexlists to ::json::json2dict that would make it index arrays into dicts of the form {0 data1 1 data2...} but it is not yet merged: https://core.tcl.tk/tcllib/tktview?name=2967134fff. With that in mind the above function allows you to say
jsonget [json::json2dict $j] results 0 artworkUrl512

instead of
dict get [lindex [dict get [json2dict $j] results] 0] artworkUrl512

dbohdan 2014-06-02: Fixed unnecessary exprs.

samoc:

Here is a hack that I use to convert AWS IAM security policies from a more readable (to me) Tcl format to JSON. It relies on an upper-case / lower-case heuristic to differentiate between lists and dicts. This works most of the time in my use case. But please don't use this in your own code unless fully understand how wrong it is.
proc json_string {s} {
    return \"[string map [list \\ \\\\ \" \\\"] $s]\"
}

proc _json  {v} {
    # Recursive JSON formatter...

    if {[llength $v] == 1} {
        return [json_string [lindex $v 0]]
    } elseif {[regexp {^[A-Z]} [lindex $v 0]]} {
        if {[lindex $v 0] eq "JSONDict:"} {
            set v [lrange $v 1 end]
        }
        for {n v} in $v {lappend items "\"$n\": [_json $v]"}
        return \{\n[join $items ,\n]\n\}
    } else {
        for v in $v {lappend items [_json $v]}
        return "\[\n[join $items ,\n]\n\]"
    }
}


proc json {dict} {

    Format "dict" as a JSON string.

    Note: Value lists begining with an upper-case letter are treated as
    nested dictionaries. Other value lists are treated as plain lists.
    To embed a nested dictionary with lower-case keys prepend "JSONDict:"
    to the start of the nested dictionary.

    e.g.
    json {A 1 B 2 L {i j k}}
    {"A": "1", "B": "2", "L": [ "i", "j", "k" ]}

} do {

    # Generate JSON format...
    set lines [_json $dict]

    # Pretty indenting...
    set indent ""
    for l in [lines $lines] {
        if {[regexp {[\}\]]} $l]} {
            set indent [range $indent 0 end-4]
        }
        append result $indent$l\n
        if {[regexp {[\{\[]} $l]} {
            append indent "    "
        }
    }

    return $result
}

example:
set policy [json [subst {
    {
        Effect Allow
        Action s3:GetObject
        Resource {
            arn:aws:s3:::$instance.au.deploy/latest/lib-Linux-i686.tar.bz2
            arn:aws:s3:::$instance.au.deploy/latest/$server_name.tar.bz2
        }
    } {
        Effect Allow
        Action s3:ListBucket
        Resource arn:aws:s3:::*.com.au.deploy
    } {
        Effect Allow
        Action sts:AssumeRole
        Resource arn:aws:iam::*:role/$region-web-user-role
    } {
        Effect Allow
        Action iam:GetRole
        Resource arn:aws:iam::*:role/$region-web-user-role
    }
}]]