Processing JSON

Press (X). "JASON!" Press (X). "JASON!"

JSON basics

JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and easy for machines to parse and generate. It's a text-based format that uses a simple syntax to represent data objects in a hierarchical structure.

JSON documents consist of key-value pairs, where each key is a string and the value can be a string, number, boolean, null, array, or another JSON object.

JSON is widely used for transmitting data between web servers and clients, as well as for storing data in databases and files. It is a popular alternative to XML because it is simpler and more compact, making it easier to parse and generate.

JSON documents are often used in web development for AJAX (Asynchronous JavaScript and XML) applications, which allow web pages to update content dynamically without reloading the entire page. JSON is also commonly used in web APIs for exchanging data between different systems.

JSON documents can be an excellent way to store configuration values:

{
    "option1": "some value",
    "option2": "some other value",
    "timeout": 10
}

Parsing JSON from payloads

Payloads can process JSON using the jshn tool.

jshn is a simple way to create and process JSON files, and is built into the Packet Squirrel.

Including jshn in a payload

jshn is included using the source command in Bash. This is a feature we have not covered before: The source command (or simply . ) includes another script and interprets it immediately. If you are familiar with other programming languages, this is similar to an include, use, or import statement.

#!/bin/bash

# Title: JSON processing
#
# Description: A JSON processing demo

# Include the system jshn library
. /usr/share/libubox/jshn.sh

# Set default netmode
NETMODE NAT

By importing the jshn library from the system, we now have access to a new set of functions for processing JSON files.

Initializing the library

Before doing anyting else, jshn should be initialized. This is simply:

json_init

This ensures no partial JSON data is held.

Loading data

To use a JSON document via jshn first it has to be loaded with json_load. This function parses the JSON and prepares it for all the other functions.

json_load "the-raw-json-string"

Content can also be loaded from a file, using the json_load_file function:

json_load_file /tmp/some-json-file.json

Getting basic data

At the simplest, jshn allows us to extract content from a JSON document. The json_get_var function takes two arguments: the name of the variable to fill, and the name of the JSON field to fetch.

Putting this together with the previous examples:

#!/bin/bash

. /usr/share/libubox/jshn.sh

# Define an example JSON.  Notice how we use a 
# single quoted string here!
JSON='{"user": "timmy", "retries": 10}'

# Initialize json handling
json_init

# Load our JSON string
json_load "$JSON"

# Extract the two variables               
json_get_var uservar "user"
json_get_var retriesvar "retries"

echo $uservar $retriesvar

Notice the use of single quotes when defining static JSON here: This keeps Bash from interpreting the braces and quotes. See the Quotes and Expansions chapter for more information.

Getting complex data

jshn can parse more complex documents as well, of course.

You can navigate into nested objects with the json_select {name} function, and navigate to the previous level of the document with json_select ..

For example, given a more complex JSON document:

{
    "config": {
        "user": "test",
        "host": "some-test-host"
    },
    
    "retries": 10
}

We could then extract values using jshn via:

#!/bin/bash

. /usr/share/libubox/jshn.sh

json_init
json_load_file test.json

# Select the "config" object
json_select "config"

# Load variables from inside "config"
json_get_var uservar "user"
json_get_var hostvar "host"

# Leave the "config" object
json_select ..

# Get "retries" from the top object
json_get_var retryvar "retries"

echo "user: ${uservar}"
echo "host: ${hostvar}"
echo "retries: ${retryvar}"

Handling multiple objects

Since JSON can contain arrays, jshn provides methods for iterating over lists of items. The function json_for_each_item takes the name of a function and the JSON value, and calls that function for each value.

Given an example JSON with a list of URLs to fetch in a payload:

{
    "urls":  [
        "https://host1.fake/file1", 
        "https://host2.fake/file2", 
        "https://host3.fake/file3"
    ]
}

We could then fetch each file with:

#!/bin/bash

. /usr/share/libubox/jshn.sh

function fetch_file() {
    wget "$1"
}

json_init
json_load_file urls.json

json_for_each_item fetch_file "urls"

Loading JSON from the web

As one of the main places JSON is used is in web services, it would be nice if we could load our JSON data directly.

Fortunately, this is (of course) possible!

To accomplish this, we combine jshn with wget.

#!/bin/bash

. /usr/share/libubox/jshn.sh

json_init
json_load "$(wget -O - https://random.host/test.json 2>/dev/null)"

Here we combine wget outputting to stdout, suppress the status by sending stderr to /dev/null, and we use the $(..) construct to retrieve the output of wget.

Notice how we enclosed the wget command in quotes! This is vital; otherwise spaces and other special characters in the returned data will break the JSON parsing.

Creating JSON in payloads

The jshn tool can also be used to create JSON files.

Creating a JSON file can be useful for saving states, or creating requests to API endpoints that expect JSON.

You can, of course, create JSON manually:

#!/bin/bash

uservar="mary"
retryvar=10
url1="https://fake.host/file1"
url2="https://fake.host/file2"

cat <<EOF
{
    "user": "${uservar}",
    "retries": "${retryvar}",
    "urls": ["${url1}", "${url2}"]
}
EOF

In simpler payloads, manually creating JSON may make the most sense.

jshn creation functions

For more complex payloads and JSON documents, using the jshn creation framework can eliminate common trouble spots and simplify creating elements like arrays.

jshn provides several creation functions:

json_add_object to create a JSON object or dictionary (balanced by json_close_object)
json_add_array to create a JSON array (balanced by json_close_array)
json_add_boolean to create a true/false value
json_add_int to create a number
json_add_string to create a string
json_dump to create the JSON itself

Lets recreate the document above using jshn functions:

. /usr/share/libubox/jshn.sh
#!/bin/bash

uservar="mary"
retryvar=10
url1="https://fake.host/file1"
url2="https://fake.host/file2"

json_init

json_add_string "user" "${uservar}"
json_add_int "retries" "${retryvar}"

# Start the array
json_add_array "urls"

# Add strings to it
json_add_string "" "${url1}"
json_add_string "" "${url2}"

# Finish the array
json_close_array

# Print the JSON
json_dump

This would print out the JSON string:

{ "user": "mary", "retries": 10, 
    "urls": [ "https:\/\/fake.host\/file1", 
              "https:\/\/fake.host\/file2" ] }

Bringing it all together

For a complete example, we'll create a payload that goes into BRIDGE mode then fetches a list of ports and filters to monitor.

We'll use this as our configuration JSON file, hosted on a server:

{
    "ledcolor": "B",
    "ledpattern": "FAST",
    "streams": [
        {"port": 80, "match": "Basic Authentication"},
        {"port": 80, "match": "Potato"},
        {"port": 12345, "match": "Something else"}
    ]
}

To handle this, we'll put together several examples so far. Our payload might look like:

#!/bin/bash

# Title: JSON Streamwatch
#
# Description: Pull data from JSON for streams

. /usr/share/libubox/jshn.sh

# Set network mode
NETMODE BRIDGE

# Wait 10 seconds to get an IP
sleep 10

# Initialize jshn
json_init

# Fetch our JSON
json_load "$(wget -O - https://random.host/payload.json 2>/dev/null)"

# Get the LED settings from the JSON
json_get_var led_v "ledcolor"
json_get_var ledpat_v "ledpattern"

LED ${led_v} ${ledpat_v}

# This function will get called for each item in the 
# streams element of the JSON
function watch_match() {
    # Select the nested object by **index**
    json_select $2
    json_get_var port_v "port"
    json_get_var match_v "match"
    json_select ..
    
    # Run the command in the background
    MATCHSTREAM eth0 TCP ${port_v} "$match_v" &
}

# Use a command group to group all the streams
{
    json_for_each_item watch_match "streams"

    # Wait for any stream to trigger
    wait -n
    # Kill the other streams
    pkill -P $$
}

# We've matched a stream, send the device to jail
NETMODE JAIL

# Set the LED
LED R SOLID

There's a lot going on here! Let's break it down:

We define a payload like usual, and set the network mode
We wait 10 seconds to get an IP because we want to use the network
We initialize the jshn library
We use the wget command to download our configuration payload
We extract some variables from the JSON payload and set the LED color
We define a function that gets called for each object in the streams element.
We use the second argument of the function, which is the index value, to descend into the nested object to obtain the port and match values.
We launch MATCHSTREAM in the background
We use a command group to run the MATCHSTREAM commands and wait for any of them to complete
The device is placed in JAIL mode and the payload exits.

Last updated 1 year ago