Processing Json
description: Press (X). “JASON!” Press (X). “JASON!”
JSON basics
JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and easy for machines to parse and generate. It’s a text-based format that uses a simple syntax to represent data objects in a hierarchical structure.
JSON documents consist of key-value pairs, where each key is a string and the value can be a string, number, boolean, null, array, or another JSON object.
JSON is widely used for transmitting data between web servers and clients, as well as for storing data in databases and files. It is a popular alternative to XML because it is simpler and more compact, making it easier to parse and generate.
JSON documents are often used in web development for AJAX (Asynchronous JavaScript and XML) applications, which allow web pages to update content dynamically without reloading the entire page. JSON is also commonly used in web APIs for exchanging data between different systems.
JSON documents can be an excellent way to store configuration values:
{
"option1": "some value",
"option2": "some other value",
"timeout": 10
}
Parsing JSON from payloads
Payloads can process JSON using the jshn
tool.
jshn
is a simple way to create and process JSON files, and is built into the Packet Squirrel.
Including jshn in a payload
jshn
is included using the source
command in Bash. This is a feature we have not covered before: The source
command (or simply .
) includes another script and interprets it immediately. If you are familiar with other programming languages, this is similar to an include
, use
, or import
statement.
#!/bin/bash
#
# Description: A JSON processing demo
# Include the system jshn library
. /usr/share/libubox/jshn.sh
# Set default netmode
NETMODE NAT
By importing the jshn
library from the system, we now have access to a new set of functions for processing JSON files.
Initializing the library
Before doing anyting else, jshn
should be initialized. This is simply:
json_init
This ensures no partial JSON data is held.
Loading data
To use a JSON document via jshn
first it has to be loaded with json_load
. This function parses the JSON and prepares it for all the other functions.
json_load "the-raw-json-string"
Content can also be loaded from a file, using the json_load_file
function:
json_load_file /tmp/some-json-file.json
Getting basic data
At the simplest, jshn
allows us to extract content from a JSON document. The json_get_var
function takes two arguments: the name of the variable to fill, and the name of the JSON field to fetch.
Putting this together with the previous examples:
#!/bin/bash
. /usr/share/libubox/jshn.sh
# Define an example JSON. Notice how we use a
# single quoted string here!
JSON='{"user": "timmy", "retries": 10}'
# Initialize json handling
json_init
# Load our JSON string
json_load "$JSON"
# Extract the two variables
json_get_var uservar "user"
json_get_var retriesvar "retries"
echo $uservar $retriesvar
Notice the use of single quotes when defining static JSON here: This keeps Bash from interpreting the braces and quotes. See the Quotes and Expansions chapter for more information.
Getting complex data
jshn
can parse more complex documents as well, of course.
You can navigate into nested objects with the json_select {name}
function, and navigate to the previous level of the document with json_select ..
For example, given a more complex JSON document:
{
"config": {
"user": "test",
"host": "some-test-host"
},
"retries": 10
}
We could then extract values using jshn
via:
#!/bin/bash
. /usr/share/libubox/jshn.sh
json_init
json_load_file test.json
# Select the "config" object
json_select "config"
# Load variables from inside "config"
json_get_var uservar "user"
json_get_var hostvar "host"
# Leave the "config" object
json_select ..
# Get "retries" from the top object
json_get_var retryvar "retries"
echo "user: ${uservar}"
echo "host: ${hostvar}"
echo "retries: ${retryvar}"
Handling multiple objects
Since JSON can contain arrays, jshn
provides methods for iterating over lists of items. The function json_for_each_item
takes the name of a function and the JSON value, and calls that function for each value.
Given an example JSON with a list of URLs to fetch in a payload:
{
"urls": [
"https://host1.fake/file1",
"https://host2.fake/file2",
"https://host3.fake/file3"
]
}
We could then fetch each file with:
#!/bin/bash
. /usr/share/libubox/jshn.sh
function fetch_file() {
wget "$1"
}
json_init
json_load_file urls.json
json_for_each_item fetch_file "urls"
Loading JSON from the web
As one of the main places JSON is used is in web services, it would be nice if we could load our JSON data directly.
Fortunately, this is (of course) possible!
To accomplish this, we combine jshn
with wget
.
#!/bin/bash
. /usr/share/libubox/jshn.sh
json_init
json_load "$(wget -O - https://random.host/test.json 2>/dev/null)"
Here we combine wget
outputting to stdout
, suppress the status by sending stderr
to /dev/null
, and we use the $(..)
construct to retrieve the output of wget
.
Notice how we enclosed the wget
command in quotes! This is vital; otherwise spaces and other special characters in the returned data will break the JSON parsing.
Creating JSON in payloads
The jshn
tool can also be used to create JSON files.
Creating a JSON file can be useful for saving states, or creating requests to API endpoints that expect JSON.
You can, of course, create JSON manually:
#!/bin/bash
uservar="mary"
retryvar=10
url1="https://fake.host/file1"
url2="https://fake.host/file2"
cat <<EOF
{
"user": "${uservar}",
"retries": "${retryvar}",
"urls": ["${url1}", "${url2}"]
}
EOF
In simpler payloads, manually creating JSON may make the most sense.
jshn creation functions
For more complex payloads and JSON documents, using the jshn
creation framework can eliminate common trouble spots and simplify creating elements like arrays.
jshn
provides several creation functions:
json_add_object
to create a JSON object or dictionary (balanced byjson_close_object
)json_add_array
to create a JSON array (balanced byjson_close_array
)json_add_boolean
to create a true/false valuejson_add_int
to create a numberjson_add_string
to create a stringjson_dump
to create the JSON itself
Lets recreate the document above using jshn
functions:
. /usr/share/libubox/jshn.sh
#!/bin/bash
uservar="mary"
retryvar=10
url1="https://fake.host/file1"
url2="https://fake.host/file2"
json_init
json_add_string "user" "${uservar}"
json_add_int "retries" "${retryvar}"
# Start the array
json_add_array "urls"
# Add strings to it
json_add_string "" "${url1}"
json_add_string "" "${url2}"
# Finish the array
json_close_array
# Print the JSON
json_dump
This would print out the JSON string:
{ "user": "mary", "retries": 10,
"urls": [ "https:\/\/fake.host\/file1",
"https:\/\/fake.host\/file2" ] }
Bringing it all together
For a complete example, we’ll create a payload that goes into BRIDGE
mode then fetches a list of ports and filters to monitor.
We’ll use this as our configuration JSON file, hosted on a server:
{
"ledcolor": "B",
"ledpattern": "FAST",
"streams": [
{"port": 80, "match": "Basic Authentication"},
{"port": 80, "match": "Potato"},
{"port": 12345, "match": "Something else"}
]
}
To handle this, we’ll put together several examples so far. Our payload might look like:
#!/bin/bash
# Title: JSON Streamwatch
#
# Description: Pull data from JSON for streams
. /usr/share/libubox/jshn.sh
# Set network mode
NETMODE BRIDGE
# Wait 10 seconds to get an IP
sleep 10
# Initialize jshn
json_init
# Fetch our JSON
json_load "$(wget -O - https://random.host/payload.json 2>/dev/null)"
# Get the LED settings from the JSON
json_get_var led_v "ledcolor"
json_get_var ledpat_v "ledpattern"
LED ${led_v} ${ledpat_v}
# This function will get called for each item in the
# streams element of the JSON
function watch_match() {
# Select the nested object by **index**
json_select $2
json_get_var port_v "port"
json_get_var match_v "match"
json_select ..
# Run the command in the background
MATCHSTREAM eth0 TCP ${port_v} "$match_v" &
}
# Use a command group to group all the streams
{
json_for_each_item watch_match "streams"
# Wait for any stream to trigger
wait -n
# Kill the other streams
pkill -P $$
}
# We've matched a stream, send the device to jail
NETMODE JAIL
# Set the LED
LED R SOLID
There’s a lot going on here! Let’s break it down:
- We define a payload like usual, and set the network mode
- We wait 10 seconds to get an IP because we want to use the network
- We initialize the
jshn
library - We use the
wget
command to download our configuration payload - We extract some variables from the JSON payload and set the LED color
- We define a function that gets called for each object in the
streams
element. - We use the second argument of the function, which is the index value, to descend into the nested object to obtain the port and match values.
- We launch
MATCHSTREAM
in the background - We use a command group to run the
MATCHSTREAM
commands and wait for any of them to complete - The device is placed in
JAIL
mode and the payload exits.