New YouTube Channel

I know it’s way past due, but I finally created a YouTube channel.  Now in between binge watching Spiderman/Elsa videos and the Gummy Bear song you can watch moderately interesting programming videos.

I don’t think everything can become a video but I will try to create accompanying videos for any code or projects that make sense.

Check out the channel here:  An Errant Programmer

The first post is a walk through of some code that was requested, a custom chart UI Page in ServiceNow using ChartJS.

Recovering Redacted Text From a Document

With the impending release of the final cache of documents related to the JFK assasination I thought it would be interesting to take a look into the topic of information leakage, or the unintentional revealing of information in a secure system. In this case the leaking of information through a redaction in a document.

For a properly redacted document the possibility is almost 0 that you will retrieve any information but according to a paper by Daniel Lopresti and A. Lawrence Spitz of Lehigh University, it may be possible to take an educated guess at a redaction not done properly.

This could be a few cases:

* The redaction method is not sufficient to cover the original text. This can mean that even though the color of the ink (usually black) matches, the original text would still be visible through the redaction.

* The redaction only partially obscures the original text. If you were to just draw a redaction through the center of some text, the tops (ascenders) and bottoms (descenders) of the letters may still be visible. These may give clues as to some of the letters being redacted.

* Analysis of the redaction size combined with knowledge of the font, from the unredacted portions of the document. If you can guess at the amount of letters redacted, even if its not a single word you can start to take a guess as to what the word or phrase is.

If any clues are to be found it may be possible to combine such clues with some Natural Language Processing and a dictionary to reveal possible word combinations for a redaction.

Although this sounded like a very interesting project to me, in the end, I decided I would rather leave the redacted documents to spies. I still felt there were some fun things to learn however and so I decided to create a prank library instead to do some fake analysis of a redacted document instead.

I wanted to learn and refresh myself of the HTML Canvas object and functions and so I decided I’d like my algorithm to scan a Canvas object, automatically finding any redaction and then fill in the redaction with song lyrics while matching the font size.

I started with a very simple HTML page with a canvas object.

<!DOCTYPE html>
<html>
<head>
<title>Deredactyl</title>
<style type="text/css">
#canvas{
border: 1px solid silver;
}
</style>
</head>
<body>
<h1>Drag a redacted image into the area below.</h1>
<p>Click a redaction to try and recover the text.</p>
<canvas id="canvas" width="600" height="800"></canvas>
<script type="text/javascript" src="deredactyl.js"></script>
</body>

</html>

I grabbed a redacted document off the internet and then started to create the JavaScript.

var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');
var img = new Image();

img.addEventListener('load', function() {
canvas.width = img.width;
canvas.height = img.height;
ctx.drawImage(img, 0, 0);
}, false);

img.src = "test.png";

This was the first revision, I wanted to make sure I could at the very least get an image loaded onto a Canvas.

From there it was all about scanning the document looking for large areas of redacted text.

I started at the top and worked my way through using ctx.getImageData(x,y,1,1) to get the RGB data for a given pixel.

It was a complete failure, crashing my browser several times. It turns out this method was just too slow. I removed console output, I even started incrementing by 2,3,5,10 at a time and it was still too slow, forcing me to rethink my algorithm.

I ended up opting for a user click even to find a specific area for a redaction. The new algorithm was this:

* A user clicks a coordinate in a canvas (hopefully in a redaction)
* Determine the left and right bounds by searching in either direction starting from the point of the click until the edge of the document is found or a pixel is found that is lighter than our threshold.
* Determine the upper and lower bounds with the same technique
* Calcuate the width and height of the area
* If the width is greater than 10 pixels and the height greater than 3, fill in the redaction. (This was just actually an arbitrary guess, I could have improved this by seeing if the size met the minimum ratio for a monospace font letter – about 1/1.7)

Here is some of the relevant code.

canvas.addEventListener('mousedown', selectRedaction);

function selectRedaction(event) {
var x = event.layerX;
var y = event.layerY;
var bounds = getRedactionBoundaries(x, y);

if (bounds.width > 10 && bounds.height > 3) {
clearTextAnalysisCalled = false;
for (var q = 0; q < (bounds.width / bounds.height); q++) { analyzeText(bounds); } } } function getRedactionBoundaries(x, y) { var bounds = { left: findLeft(x, y), right: findRight(x, y), top: findTop(x, y), bottom: findBottom(x, y), } bounds.height = bounds.bottom - bounds.top; bounds.width = bounds.right - bounds.left; return bounds } function findLeft(x, y) { for (var search = x; search > 0; search--) {
var pixel = ctx.getImageData(search, y, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity;
if (!isDark)
return search
}
}

function findRight(x, y) {
for (var search = x; search < img.width; search++) {
var pixel = ctx.getImageData(search, y, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity; if (!isDark) return search } } function findTop(x, y) { for (var search = y; search > 0; search--) {
var pixel = ctx.getImageData(x, search, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity;
if (!isDark)
return search
}
}

function findBottom(x, y) {
for (var search = y; search < img.height; search++) {
var pixel = ctx.getImageData(x, search, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity;
if (!isDark)
return search
}
}

After determining the space I decided I wanted to add some visual effects to make it seem like the code was actually doing some analysis. I really wanted to create a ‘snow like’ effect so again I looped over the height and width of the redaction and again it was way too slow.

I tried just randomly putting up dots on the redaction but this was actually too fast. Of course there was no easy way to slow this down, so I ended up adding a setTimeout calling the function again and again until a limit was reached. This again was too slow.

Finally, after much annoyance with what should be a trivial feature, I decided to spawn several instances of the function each of which would continue for a pre-determined amount of time before calling the next function, a wipe effect.

Of course, I created a race condition in the process. To fix this condition I created a global flag that would be set by the first function to finish, thereby collapsing the x amount of threads into a single one.

The wipe function was comparatively easy, just going from left to right, filling the redaction with white.

Finally, with the smoke and mirrors portion done, I could fill in the text.

I needed a couple pieces of information, one was the size of the font, which I determined by the height of the redaction. The second was the length of text that could fit in the space. For this I took the raw width and divided it by the rough width of a font that would fit in the space given the height

var fontSize = bounds.height;
var textLength = Math.floor((bounds.width / fontSize) * 1.7);

Now with this length I could start to populate words. I am nothing if not a sucker for the classics so I chose the lyrics to Rick Astley’s Never Gonna Give You Up.

To determine which words to use I iterated over the lyrics pulling words whose length would fit into the remaining space. I’d repeat this for one loop and finally give up if I couldn’t fill the space exactly.

function scrapeText(length) {
var words = "";
var attempts = 0;

while (words.length < length && attempts < sourceWords.length) {
var randomWord = sourceWords[wordCounter];
wordCounter++;

if (wordCounter == sourceWords.length)
wordCounter = 0;

if ((words.length + randomWord.length) < length) { words = words + " " + randomWord; attempts = 0; } else { attempts++ } } return words; } 

Once I was certain this was all working I added one last feature – which was more of a re-learning exercise – and that was to drag and drop files on to the canvas. It turned out to be much easier than I had remembered.

 canvas.addEventListener("dragover", function(evt) { evt.preventDefault(); }, false); canvas.addEventListener("drop", function(evt) { var files = evt.dataTransfer.files; if (files.length > 0) {
var file = files[0];

if (typeof FileReader !== "undefined" && file.type.indexOf("image") != -1) {
var reader = new FileReader();

reader.onload = function(evt) {
img.src = evt.target.result;
};

reader.readAsDataURL(file);
}
}

evt.preventDefault();
}, false);

Finally, not wanting to ruin the joke I took the plain text of the lyrics and fed that portion through a JavaScript obfuscator, nothing someone couldn’t reverse if they really wanted, but just enough to fool people on the first glance.

All in all it took roughly an hour to get everything working and to my surprise I was even able to Rick Roll a couple friends although I am sure they were shaking their heads.

 

 

All source code from this article available at: https://github.com/AnErrantProgrammer/deredactyl

Sources:
* Daniel Lopresti and A. Lawrence Spitz – Information Leakage Through Document Redaction: Attacks and Countermeasures (www.cse.lehigh.edu/~lopresti/Publications/2005/spie05a.pdf)

Site Rebranding

So this has been a long time coming.  I have been spending less & less time in ServiceNow over the past year.  I still use SNOW quite a bit but professionally and personally I am using other technologies and tools and would like my blog to reflect what I am actually doing.

As I find interesting bits in ServiceNow I will write about them but I will also open the scope of the blog to include other areas as well.

I hope everyone continues read and enjoy.

Getting the Current URL

Recently ran into this wall which stumped me for a few.  Had to get the full URL including the path from a UI Page on the server.  Using my book as a reference (shameless plug) I found RP.getReferringURL(), but this only returns the page name I found out and RP is not a usable object everywhere.

The solution was to find the actual Request object through the Transaction.

//
var tr = new GlideTransaction.get();
gs.print(tr.getRequest().getRequestURL());
//

So what is happening here? GlideTransaction.get() returns the current transaction. It has a scriptable method ‘getRequest()’ which returns the underlying request for this transaction which an HTTPServletRequest. Not every method is going to be available since SN has whitelisted/blacklisted many but getRequestURL().

There you have it the full URL with the path! What other methods have you found to get the URL?

YouTube Like Loader

Just a simple script to add a YouTube like loading bar to the top of the page for AJAX Requests. Many users said they wanted a way to know when the system was “thinking”. There actually is a loading GIF in the top right (I noticed it in Fuji) but it’s a bit hard to see. This adds a very obvious loader to every page, even popups and CMS page.

This should go in a Global UI Script. Tweak as necessary!

if(self == top){
var loadingDiv = document.createElement("div");
var loadingInterval = false;
var loadingIntervalWidth = 0;
var stopLoading = false;

loadingDiv.style.width = "100%";
loadingDiv.style.position = "fixed";
loadingDiv.style.height = "2px"
loadingDiv.style.top = "0";
loadingDiv.style.left = "0";
loadingDiv.style.zIndex = "2000";
loadingDiv.style.background = "#29d";
loadingDiv.style.display = "none";

document.addEventListener("DOMContentLoaded", function(event) {
document.body.appendChild(loadingDiv)
});

function startProgress(){

if(loadingInterval == false){
stopLoading = false;
loadingDiv.style.display = "block";
loadingDiv.style.width = "0%";
loadingIntervalWidth = 0;
loadingInterval = setTimeout(incrementProgress, 50);

}
}

function incrementProgress(){

loadingIntervalWidth++;
loadingDiv.style.width = loadingIntervalWidth.toString() + "%";
if(loadingIntervalWidth < 100 && stopLoading == false){ loadingInterval =setTimeout(incrementProgress, 50); } else { loadingDiv.display = "none"; stopProgress() } } function stopProgress(){ stopLoading = true; loadingDiv.style.width = "100%"; loadingInterval = setTimeout(hideProgress, 100); } function hideProgress(){ loadingInterval = false; loadingIntervalWidth = 0; loadingDiv.display = "none"; loadingDiv.style.width = "0%"; clearTimeout(loadingInterval); } CustomEvent.observe('ajax.loading.start', function() { startProgress() }); CustomEvent.observe('ajax.loading.end', function() { stopProgress(); }); }

Using Jelly In ServiceNow Release Candidate Shipping

I am happy to announce I have started sending out the first Release Candidate for Using Jelly In ServiceNow.  I decided to keep the pricing as is at $25 USD for now until the final version comes out.

To purchase you can use the following form and I will send as soon as PayPal alerts me.  I try to do it immediately but sometimes it takes up to a day.

Update: I have started using Gumroad to distribute the book now instead of manually sending copies out via email.

Get the book now!

Table of Contents

0 Introduction
1 Jelly
1.1 Key Concepts
2 Jelly and ServiceNow
2.1 UI Pages
2.1.1 Page Template
2.2 UI Macros
2.2.1 Formatters
2.3 Other Jelly Scriptable Areas
3 Jelly Scripting
3.1 Jelly Tag Library
3.1.1 Variables
3.1.2 Output
3.1.3 JEXL Expressions
3.1.4 Conditions
3.1.5 Looping
3.1.6 Undocumented Jelly Tags
3.2 Glide Tag Library
3.2.1 Using References
3.2.2 Scripting Tags
3.2.2.1 Running JavaScript
3.2.2.2 Scripting Helpers
3.2.2.3 Using Functions and Macros
3.2.2.4 Miscellaneous Functions
3.2.3 UI Tags
3.2.3.1 HTML Elements
3.2.3.2 Form Elements
3.2.3.3 List Elements
3.2.3.4 Choice Lists
3.2.3.5 Content Management Components
3.2.3.6 Knowledge Base Components
3.2.3.7 Chart Components
3.2.3.8 Extended UI Components
3.2.3.9 Other ServiceNow Components
3.2.4 Undocumented Tags
3.3 Debugging
3.3.1 Debugging Tags
4 Advanced Usage
4.1 Glide and Jelly JavaScript Objects
4.1.1 Executing Jelly
4.1.2 Render Properties
4.1.3 Jelly Contexts
4.1.4 Evaluating Expressions
4.2 Creating UI Extensions
4.3 Decorators
4.4 Overriding Default Templates
4.5 System Properties
5 Jelly Examples
5.1 Core Concepts
5.1.1 Multiple Phases
5.1.2 Fizz Buzz
5.1.3 Fibonacci Numbers
5.1.4 Bubble Sort
5.1.5 Calling Functions
5.1.6 Towers of Hanoi
5.2 Static Page
5.2.1 Hello World
5.2.2 Simple Calculator
5.2.3 Output A List
5.2.4 Simple Form
5.2.5 Form Using Macros
5.3 Dynamic Page
5.3.1 Dynamic Calculator
5.4 Multi-Page Application
5.4.1 Simple Portal
5.4.2 Advanced Form
5.5 AJAX Application
5.5.1 AJAX Form
5.5.3 AJAX Form Widget
5.6 Custom Formatter
5.6.1 Custom Element
5.6.2 Form Widgets
5.7 Mobile Page
5.7.1 Mobile Only Page
5.7.2 Responsive Page
5.8 Frameworks and Libraries
5.8.1 Datatables List
5.8.2 Twitter Bootstrap
6 Index
6.1 Jelly Language Reference
6.2 JavaScript Objects Reference
6.3 LIcenses and Copyrights

Two Big Announcements

One, I am at Knowledge15!  Please come say “Hi” at the Cerna Solutions booth.

Second, and this is the reason I have not been blogging much lately, I am nearly done writing my first – and I am pretty sure the world’s only -and I am certain the world’s LAST – book on Jelly in ServiceNow!

Here is a quick summary of what the book “Using Jelly in ServiceNow” contains:

  • Start to end walkthrough of every function with examples
  • Full page examples with walkthroughs
    • Everything from simple pages to portals and AJAX pages
    • Examples showing how to use Frameworks
  • Documentation on all Glide API Interfaces for Jelly
  • Full language reference
  • Full JavaScript Object reference
  • Over 200 Pages of content

During the conference I am offering pre-orders at a special price of $25.  As soon as the book is ready (~Mid April or earlier) it will be delivered by email.

Any questions feel free to stop by the booth or leave a comment here!  Thanks.


Get the book now!

Update: Some folks were asking for more details so here is the Table of Contents for Using Jelly In ServiceNow.

  • 0. Introduction
  • 1. Jelly
  • 1.1. Key Concepts
  • 1.2. Jelly and ServiceNow
  • 1.2.1. UI Pages
  • 1.2.2. UI Macros
  • 1.2.2.1. Formatters
  • 1.2.3. Other Scriptable Areas
  • 2. Jelly Scripting
  • 2.1. Jelly Tag Library
  • 2.1.1. Variables
  • 2.1.2. Output
  • 2.1.3. JEXL Expressions
  • 2.1.4. Conditions
  • 2.1.5. Looping
  • 2.1.6. Unused Jelly Tags
  • 2.2. Glide Tag Library
  • 2.2.1. Setting Up A Reference
  • 2.2.2. Scripting Tags
  • 2.2.2.1. Running JavaScript
  • 2.2.2.2. Scripting Helpers
  • 2.2.2.3. Using Functions and Macros
  • 2.2.2.4. Debugging Tags
  • 2.2.2.5. Miscellaneous Functions
  • 2.2.3. UI Tags
  • 2.2.3.1. Basic HTML Elements
  • 2.2.3.1.1. HTML Elements
  • 2.2.3.1.2. Form Elements
  • 2.2.3.1.3. Choice Lists
  • 2.2.3.2. ServiceNow Elements
  • 2.2.3.3. Miscellaneous Tags
  • 2.2.3.3.1. Undocumented Tags
  • 2.3. Debugging
  • 3. Advanced Usage
  • 3.1. Glide and Jelly JavaScript Objects
  • 3.1.1. Executing Jelly
  • 3.1.2. Render Properties
  • 3.1.3. Jelly Contexts
  • 3.1.4. Evaluating Expressions
  • 3.2. Overriding Default Templates
  • 3.3. System Properties
  • 3.4. UI Element Tags
  • 3.5. Creating Custom UI Tags
  • 4. Jelly Examples
  • 4.1. Core Concepts
  • 4.1.1. Multiple Phases
  • 4.1.2. Fibonacci Numbers
  • 4.1.3. Bubble Sort
  • 4.2. Static Page
  • 4.2.1. Hello World
  • 4.2.2. Simple Calculator
  • 4.2.3. Output A List
  • 4.2.4. Simple Form
  • 4.2.5. Form Using Macros
  • 4.3. Dynamic Page
  • 4.3.1. Dynamic Calculator
  • 4.4. Multi-Page Application
  • 4.4.1. Simple Portal
  • 4.4.2. Advanced Form
  • 4.5. AJAX Application
  • 4.5.1. AJAX Form
  • 4.5.3. AJAX Form Widget
  • 4.6. Custom Formatter
  • 4.6.1. Custom Element
  • 4.6.2. Form Widgets
  • 4.7. Mobile Page
  • 4.7.1. Mobile Only Page
  • 4.7.2. Responsive Page
  • 4.8. Frameworks and Libraries
  • 4.8.1. Datatables List
  • 4.8.2. Twitter Bootstrap
  • 5. Jelly XML Language Reference
  • 6. JavaScript Objects Reference

Using cURL To Upload Files

Uploading files is fairly easy in SN but what if you need to upload a lot of files?

There are some solutions using the AttachmentCreator service which work great but I wondered if it was possible to do it without any programming at all. So I racked my brain then I remembered a tool: cURL.

cURL is a tool used for making all sorts of web related requests. The best part about is it a command line tool so if you are in need of automating web requests this is the tool to use. Theres several versions for every OS out there and its fairly well documented.

So lets take a look at how to use it…

If you haven’t already grab a copy of cURL from http://curl.haxx.se/

Logging in is a two step process, first we need to initialize our connection and set up a cookie:

The –cookie-jar cjar parameter creates a cookie called cjar so that we can save our information

-k tells cURL to just accept any certificates that the site offers, this gets us by HTTPS if that’s enabled on your instance.

-o output1.log tells curl to output to a log file instead of the screen

–location tells curl to follow any redirects

Finally we put our instance URL.

Once you run this command and you have a cookie jar created you can attempt to log in.

This command reuses some of the same parameters but also specifies all the data used to log in. Just replace USERNAME, PASSWORD and INSTANCE and you should get a positive response back.

The way to check that everything is working properly is in the output files. If they return the code that would be the homepage you are golden.

Finally once you have your cookie and are logged in you can start uploading files.

In this case I am uploading a PNG file called test that’s located in the same directory as cURL. If this is successful you will now be able to see the newly uploaded file on your instance.

Because this is an image file it will be available in the Image file browser but also at https://INSTANCE.service-now.com/scs/test.png or https://INSTANCE.service-now.com/test.pngx

Note: Theres also no need to keep logging in, once you are in you can continue to upload files.

If you’ve got tons of files no problem, just couple this technique with the for command (Windows Systems) and you can loop through all the files in a directory.

Heres a quick tutorial on that command: http://www.suite101.com/content/batch-file-programming-for-command-a37305

RemoteGlideRecord Revisited

So there was a request to bring back an old article on RemoteGlideRecord.  I am fairly certain ServiceNow will want to deprecate this Package.  I have tested the following in Eureka and it still works but I don’t know for how long.  Of course, even if you use this and it gets deprecated you always have the option of another integration type.

So what is RemoteGlideRecord?  It is a Package that allows you to use GlideRecord to query from one ServiceNow Instance to another.  If you are interested, behind the scenes this is executing a SOAP Request and uses Basic Authentication to authenticate to the second instance.

To create a basic instance of the Package use the following syntax.


var gr = new Packages.com.glide.communications.RemoteGlideRecord("https://INSTANCE.service-now.com", "TABLE");
gr.setBasicAuth("USER", "PASS");

 

Of course replace INSTANCE with your target instance, TABLE with the target TABLE and USER/PASS with the user and password.

WOAH!  Wait a second.  I don’t want to store my passwords in plain text in a script file!

Great point.  So here is an example of the Encrypter class.  You can encrypt and store passwords as properties and decrypt them when you need them.


var encrypter = new GlideEncrypter();
var encryptedPass = encrypter.encrypt("test");

gs.log("NEVER LOG PASSWORDS!");
gs.log( encrypter.decrypt(encryptedPass) );

 

Once we have the basic object set up the query can be executed.  Here is a sample of the code to run that query complete with some error checking to make sure the connection is valid.


if (gr.isValid()) {
gr.orderBy("number");
gr.query();

while (gr.next()) {
gs.print(gr.getValue("number"));
gs.print(gr.getDisplayValue("cmdb_ci"));
}

} else {
gs.print("not valid");
}

 

Notice that you cannot just dot.walk to the field name you must use getValue, getDisplayValue, getIntValue, getBooleanValue.

Likewise setting a value is done with setValue(field, value).

There are also a few configuration options which may or may not help you.

setAcceptGZIP(boolean) and setSendGZIP(boolean) obviously instruct the RemoteGlideRecord to send and receive GZIP.  This should be true.

setRedirectSupported(boolean) I can only assume tells RemoteGlideRecord to follow redirects, although I cannot test this theory.

setupProxy( host, port, user, pass ) seems like it can route the call through a proxy although I can think of no reason why that would ever happen.

setReturnDisplayValue(boolean) will turn of returning display values, if you aren’t planning on using them it can save some bytes off of what gets transferred.

setDebug(boolean) turns off and on debugging.  Off by default this will add invaluable log outputs that will definitely help if you don’t get connected straight away.

There is also a RemoteGlideAggregate which extends RemoteGlideRecord.  Seems to have the same methods as a standard GlideAggregate.  So there’s that to try as well.

Giving ServiceNow A Brain

In this, the first monthly project, I will be looking at trying to give ServiceNow the ability to automatically classify user input and suggest resolutions.

To start lets talk briefly about text classification.  I will say upfront that I am no expert in natural language processing.   I understand just enough to get myself in trouble.   For my purpose I will be using what is known as a Naive Bayes Classifier.

A super simple explanation is that based on some given data one can predict the probable output. This is done by creating a sort of map by training the system with previous data for which the output is already known.

The data that is important for text classification is word frequency and really anything that can be expressed in terms of data can be classified this way.

The intention will be to train the system based on a user input “description” and classify that text in a couple dimensions.  Possibly the type of resolution: Knowledge Base, Request, Incident and Category: Software, Hardware, Network and Topic.  The resulting classification will be fed into a ServiceNow application to make suggestions and automatically resolve the users issue if possible.

Since the libraries to do the classification are too large to bring them into ServiceNow I think the best solution would be to create a WebService Integration to a NodeJS app.  The integration will be able to add training data directly from ServiceNow and ask for a suggestion based on text.

The interface should accept users input and make suggestions which the user can accept or reject (and look for another solution.)  This could either be from the Ticket table or possibly a custom form, like a mock chat system.

Join me for the next post in this series where I will prototype the Node Service and Integration.  In the final post I will fully train the system and let end users try it.