New YouTube Channel

I know it’s way past due, but I finally created a YouTube channel.  Now in between binge watching Spiderman/Elsa videos and the Gummy Bear song you can watch moderately interesting programming videos.

I don’t think everything can become a video but I will try to create accompanying videos for any code or projects that make sense.

Check out the channel here:  An Errant Programmer

The first post is a walk through of some code that was requested, a custom chart UI Page in ServiceNow using ChartJS.

Automating Flippy Knife for Android

Every so often a friend of mine will start talking about Hearthstone and it will pique my interest. Hearthstone is a collectible card game where you acquire cards and build decks which you use to battle other players.

Thinking about it triggers thoughts of my own pubescent hobby of Magic: The Gathering and I usually instinctively cringe at the thought of myself and stop, but something about the atmosphere of the game and popularity got me interested this time and I decided to try it out.

Popped open the Play Store and started the download. While it was going another suggested game popped up though, Flippy Knife. I immediately downloaded it based on the artwork alone.

Flippy Knife is actually a collection of mini-games themed around…flipping a knife (or axe, or sword, or some other weapon). It starts off pretty simply, but it has just the right balance of frustration and rewards to keep you hooked. Another clever mechanism the game has is easily switching between the different mini-games. Instead of going back to a menu you switch games by buttons on either side of the screen. As soon as you get bored of a game, you can switch to the next one. My favorite part was of course, unlocking new knifes, I was grinding out getting coins when I got a thought. Could I cheat at this game? Would that be OK?

I’ll tell you, I wrestled with it. This wasn’t like loading up Super Mario Brothers 3 that I beat 100x in the Game Genie (Actually the first game I used with the Game Genie was TMNT2), these people make their living off of people playing the game fairly. But, in the interest of learning, I thought it was ok as long as I did not “hack” the game to unlock the knives but instead automated the playing of the game.

There are 4 mini games in the app, one where you swipe to flip. This I thought would be too hard to perfectly do and imagined there would be some randomization that prevents any level of automation.

Two others were just too complex, the “catch the drone” and “side scroller” ones had no real predictable pattern and therefore would be a nightmare to even attempt.

The one I tried was the target game. Basically you have a moving target on the right side of the screen, but it only moves on the Y axis. You swipe to throw the knife at it. As long as you compensate for the height of the target by changing the angle, you can pretty much always hit it. After about 10 hits you get coins. After about 20 hits the target starts moving, which would be bad normally but if you miss the game resets.

My algorithm was simple:

* Take a screenshot
* Find the height of the target by looking for the first red pixel along the right and side of the screen where the target would be.
* Perform some “math” using the height of the target to find the coordinates I should automate a swipe to
* Swipe from the starting point to the target

Before any of that I had to find out how to even communicate with the device. I’d never done it before but some quick Googling revealed there is in fact a way to send raw commands to a device called the Android Debug Bridge (ADB).

ADB can do pretty much anything from sending a keystroke, swipe or touch, to installing APKs and taking screenshots. Exactly what I needed.

To start, I had to install ADB I used (https://forum.xda-developers.com/showthread.php?t=2588979). Fairly old link but still seemed to work for me.

I figured this might be kind of slow so I took a slight detour at this point and looked for an npm module that had the same functionality and found adbkit. (https://github.com/openstf/adbkit)

After a few quick tests I had it running sending a single random click to the screen.

Once I knew I could send the raw events, I had to figure out how to actually process the image on the screen, so using the ADB command line tools, I took a screencap and started looking at how I could load this image into Node. This led me to pngjs (https://github.com/lukeapage/pngjs). I am sure the library has many great features but for the time being it did what I was looking for, took a PNG and gave me back the pixel data.

To my test script I added a quick loop to look for the “red target”, to my surprise it worked pretty much right away.

Here, I am passing the PNG image into this findTarget function. You can see the possible range for the targets was between 566 (at the top) and 1700 (at the bottom) for some reason I also scanned an area 11px wide although it was probably unnecessary in the end.

The comparison of the img.data to the RBG values (255,79,77) would reveal if I had found the target and would return the Y coordinate.

Now I had to wire up the taking of a screen capture using ADB and this search function.

I start by taking the screencap, which returns a promise. To that I chained a function which would pipe the PNG returned (probably in binary format) to the pngjs2 library would would give me the data in a nice usable format in JS. I could then feed that to my findTarget and aimShot functions. The only real tricky part was getting all the promises to work correctly. Normally I don’t think I would have even needed them except I wanted to put this in a loop, so I did not want to start the next iteration until I was sure the work was done.

In actuality this took me a few minutes to get to, I am sure I had to take many smaller steps until I got this working.

Through trial and error I ended up with this equation to get the right spot to aim. Basically the lower the target you need to throw harder and more towards the center. Higher you can throw a bit lighter but it must be above the target. Since it was close enough I rounded the starting x coordinate to 1337 for extra points.

With the aiming done I was ready to do the swipe. ADB does this with the shell command which lets you specify the type of input and parameters.

Lastly, I needed to add another promise around that code and then wire up the loop and I was done.

The trials were great and I started to accumulate some coins, but it was kind of slow. Also it was crushing my battery and causing my phone to heat up, so it was not something I was going to run long-term.

Also, it was not getting me coins faster than I could get myself. I went back to playing the game normally but then I decided I need to give this one more try. I opened up the console and tried doing a direct swipe up in the middle of the screen on the “knife flip” game. The knife went up and landed almost perfectly. With a bit more tweaking I got it as close as I could. Basically ending up with the knife angled slightly to the left. Through the normal course of playing I knew if I then swiped the other direction the knife would land and angle the other direction. So with a bit more trial and error I found two swipes which would perfectly balance each other.

I figured I didnt really even need JS at this point since I could do everything I needed from the command line and I switched to a batch file and let it run.

Success, the knife was flipping and because I had got it to do the shortest throw possible it was crazy fast. After a while some randomness does creep in and the knife will fly off in either direction, but not before racking up some major points.

Even though I ended up using a very trivial solution, I think it was definitely worth doing to learn some of these techniques with PNG files, piping, promises and of course ADB. I ran the cheat for a bit to see how far it would go but it really was not fun or even worth it in the end. I much prefer actually playing the game and definitely recommend it.

I wonder if Hearthstone ever finished downloading.

Recovering Redacted Text From a Document

With the impending release of the final cache of documents related to the JFK assasination I thought it would be interesting to take a look into the topic of information leakage, or the unintentional revealing of information in a secure system. In this case the leaking of information through a redaction in a document.

For a properly redacted document the possibility is almost 0 that you will retrieve any information but according to a paper by Daniel Lopresti and A. Lawrence Spitz of Lehigh University, it may be possible to take an educated guess at a redaction not done properly.

This could be a few cases:

* The redaction method is not sufficient to cover the original text. This can mean that even though the color of the ink (usually black) matches, the original text would still be visible through the redaction.

* The redaction only partially obscures the original text. If you were to just draw a redaction through the center of some text, the tops (ascenders) and bottoms (descenders) of the letters may still be visible. These may give clues as to some of the letters being redacted.

* Analysis of the redaction size combined with knowledge of the font, from the unredacted portions of the document. If you can guess at the amount of letters redacted, even if its not a single word you can start to take a guess as to what the word or phrase is.

If any clues are to be found it may be possible to combine such clues with some Natural Language Processing and a dictionary to reveal possible word combinations for a redaction.

Although this sounded like a very interesting project to me, in the end, I decided I would rather leave the redacted documents to spies. I still felt there were some fun things to learn however and so I decided to create a prank library instead to do some fake analysis of a redacted document instead.

I wanted to learn and refresh myself of the HTML Canvas object and functions and so I decided I’d like my algorithm to scan a Canvas object, automatically finding any redaction and then fill in the redaction with song lyrics while matching the font size.

I started with a very simple HTML page with a canvas object.

<!DOCTYPE html>
<html>
<head>
<title>Deredactyl</title>
<style type="text/css">
#canvas{
border: 1px solid silver;
}
</style>
</head>
<body>
<h1>Drag a redacted image into the area below.</h1>
<p>Click a redaction to try and recover the text.</p>
<canvas id="canvas" width="600" height="800"></canvas>
<script type="text/javascript" src="deredactyl.js"></script>
</body>

</html>

I grabbed a redacted document off the internet and then started to create the JavaScript.

var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');
var img = new Image();

img.addEventListener('load', function() {
canvas.width = img.width;
canvas.height = img.height;
ctx.drawImage(img, 0, 0);
}, false);

img.src = "test.png";

This was the first revision, I wanted to make sure I could at the very least get an image loaded onto a Canvas.

From there it was all about scanning the document looking for large areas of redacted text.

I started at the top and worked my way through using ctx.getImageData(x,y,1,1) to get the RGB data for a given pixel.

It was a complete failure, crashing my browser several times. It turns out this method was just too slow. I removed console output, I even started incrementing by 2,3,5,10 at a time and it was still too slow, forcing me to rethink my algorithm.

I ended up opting for a user click even to find a specific area for a redaction. The new algorithm was this:

* A user clicks a coordinate in a canvas (hopefully in a redaction)
* Determine the left and right bounds by searching in either direction starting from the point of the click until the edge of the document is found or a pixel is found that is lighter than our threshold.
* Determine the upper and lower bounds with the same technique
* Calcuate the width and height of the area
* If the width is greater than 10 pixels and the height greater than 3, fill in the redaction. (This was just actually an arbitrary guess, I could have improved this by seeing if the size met the minimum ratio for a monospace font letter – about 1/1.7)

Here is some of the relevant code.

canvas.addEventListener('mousedown', selectRedaction);

function selectRedaction(event) {
var x = event.layerX;
var y = event.layerY;
var bounds = getRedactionBoundaries(x, y);

if (bounds.width > 10 && bounds.height > 3) {
clearTextAnalysisCalled = false;
for (var q = 0; q < (bounds.width / bounds.height); q++) { analyzeText(bounds); } } } function getRedactionBoundaries(x, y) { var bounds = { left: findLeft(x, y), right: findRight(x, y), top: findTop(x, y), bottom: findBottom(x, y), } bounds.height = bounds.bottom - bounds.top; bounds.width = bounds.right - bounds.left; return bounds } function findLeft(x, y) { for (var search = x; search > 0; search--) {
var pixel = ctx.getImageData(search, y, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity;
if (!isDark)
return search
}
}

function findRight(x, y) {
for (var search = x; search < img.width; search++) {
var pixel = ctx.getImageData(search, y, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity; if (!isDark) return search } } function findTop(x, y) { for (var search = y; search > 0; search--) {
var pixel = ctx.getImageData(x, search, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity;
if (!isDark)
return search
}
}

function findBottom(x, y) {
for (var search = y; search < img.height; search++) {
var pixel = ctx.getImageData(x, search, 1, 1);
var isDark = pixel.data[0] + pixel.data[1] + pixel.data[2] <= sensitivity;
if (!isDark)
return search
}
}

After determining the space I decided I wanted to add some visual effects to make it seem like the code was actually doing some analysis. I really wanted to create a ‘snow like’ effect so again I looped over the height and width of the redaction and again it was way too slow.

I tried just randomly putting up dots on the redaction but this was actually too fast. Of course there was no easy way to slow this down, so I ended up adding a setTimeout calling the function again and again until a limit was reached. This again was too slow.

Finally, after much annoyance with what should be a trivial feature, I decided to spawn several instances of the function each of which would continue for a pre-determined amount of time before calling the next function, a wipe effect.

Of course, I created a race condition in the process. To fix this condition I created a global flag that would be set by the first function to finish, thereby collapsing the x amount of threads into a single one.

The wipe function was comparatively easy, just going from left to right, filling the redaction with white.

Finally, with the smoke and mirrors portion done, I could fill in the text.

I needed a couple pieces of information, one was the size of the font, which I determined by the height of the redaction. The second was the length of text that could fit in the space. For this I took the raw width and divided it by the rough width of a font that would fit in the space given the height

var fontSize = bounds.height;
var textLength = Math.floor((bounds.width / fontSize) * 1.7);

Now with this length I could start to populate words. I am nothing if not a sucker for the classics so I chose the lyrics to Rick Astley’s Never Gonna Give You Up.

To determine which words to use I iterated over the lyrics pulling words whose length would fit into the remaining space. I’d repeat this for one loop and finally give up if I couldn’t fill the space exactly.

function scrapeText(length) {
var words = "";
var attempts = 0;

while (words.length < length && attempts < sourceWords.length) {
var randomWord = sourceWords[wordCounter];
wordCounter++;

if (wordCounter == sourceWords.length)
wordCounter = 0;

if ((words.length + randomWord.length) < length) { words = words + " " + randomWord; attempts = 0; } else { attempts++ } } return words; } 

Once I was certain this was all working I added one last feature – which was more of a re-learning exercise – and that was to drag and drop files on to the canvas. It turned out to be much easier than I had remembered.

 canvas.addEventListener("dragover", function(evt) { evt.preventDefault(); }, false); canvas.addEventListener("drop", function(evt) { var files = evt.dataTransfer.files; if (files.length > 0) {
var file = files[0];

if (typeof FileReader !== "undefined" && file.type.indexOf("image") != -1) {
var reader = new FileReader();

reader.onload = function(evt) {
img.src = evt.target.result;
};

reader.readAsDataURL(file);
}
}

evt.preventDefault();
}, false);

Finally, not wanting to ruin the joke I took the plain text of the lyrics and fed that portion through a JavaScript obfuscator, nothing someone couldn’t reverse if they really wanted, but just enough to fool people on the first glance.

All in all it took roughly an hour to get everything working and to my surprise I was even able to Rick Roll a couple friends although I am sure they were shaking their heads.

 

 

All source code from this article available at: https://github.com/AnErrantProgrammer/deredactyl

Sources:
* Daniel Lopresti and A. Lawrence Spitz – Information Leakage Through Document Redaction: Attacks and Countermeasures (www.cse.lehigh.edu/~lopresti/Publications/2005/spie05a.pdf)

Site Rebranding

So this has been a long time coming.  I have been spending less & less time in ServiceNow over the past year.  I still use SNOW quite a bit but professionally and personally I am using other technologies and tools and would like my blog to reflect what I am actually doing.

As I find interesting bits in ServiceNow I will write about them but I will also open the scope of the blog to include other areas as well.

I hope everyone continues read and enjoy.

Debugging ServiceNow Performance Issues

Performance issues are often some of the most difficult to fix. There are dozens of variables from code to networking that must be examined in order to track down the problem and trying to get the problem to manifest while you are looking can be next to impossible…unless you know where to look.

In this article I will outline several scenarios in which performance is degraded. I’ll use Helsinki for this but most of these techniques should work for Geneva and Fuji as well.

Transaction Logs

The main tool we will use is ServiceNow itself.  The transaction log will contain most of the evidence we need to investigate performance issues.

  1. Under System Logs navigate to one of the Transaction Logs.
  2. On the List view add all the columns referencing “Time” there are about a dozen total.
  3. On the List view add Session, System ID and User Agent
  4. On the form view add every single field.
    I like to create a separate section for the time fields.

Scenarios

I will break these scenarios into one of three categories:

  • Client is the users browser or machine
  • Server is the ServiceNow server
  • Network is everything in between

For each scenario I will list symptoms and potential root causes.

Scenario 1: Poor Network Performance

A fairly common scenario, users may experience poor network performance especially when there are multiple locations, in an on-premise installation, where the users must VPN in or where additional devices such as firewalls or proxies come into play.

Symptoms

  • One or more users will report intermittent degraded performance
  • If multiple users report issue it may only occur at specific times or locations
  • If a single user reports issue it may only happen at specific locations

Verifying The Issue Users should try the following.

  • Logout, completely clear cache, delete all cookies and try logging in again
  • Log in from a different browser
  • Log in from a different machine on the same network

If the issue still persists it may be an issue with the network. To verify:

  1. Navigate to Transaction Logs (All User)
  2. Find all transactions for a particular users session
  3. Find all transaction where the Client response time or Network time is excessive (>5000ms)
  4. If there are excessively high Network times then those should be flagged and you should look for additional logs around the same time
  5. If additional log entries are found this trend will likely point to a network issue
  6. If there are excessively high Client response times you should compare this time to the other “time” fields (excluding Browser Time)
  7. If the sum of the other time fields does not closely match the Client response time then it is likely a network issue.

Once you have determined you have a network issue it is necessary to look for trends to isolate the problem.

  • Do affected users have the same location or network?
  • Is the issue isolated to a specific time of day?
  • Do any users access ServiceNow through a proxy?
  • Is the issue intermittent or consistent?

Potential Root Causes

  • Overloaded or malfunctioning networking equipment – This would almost certainly affect other applications as well.
  • Overloaded proxy – If users access ServiceNow through a proxy, is the machine correctly configured and will it support the current load?
  • Insufficient bandwidth.
  • Poor wireless signal.

Fixes

  • Unfortunately for networking issues there is not much that can be done from ServiceNow. Infrastructure may need to be upgraded or reconfigured to support the load.

Scenario 2: Mis-configured ACL

Usually occurring for non-admin users or those with specific roles, mis-configured ACLs can impact forms but especially lists where ACLs run multiple times.

Symptoms

  • One or more users will report excessively long load times for lists or forms.

Verifying The Issue

  1. Navigate to Transaction Logs (All User).
  2. Find all transactions for a particular users session.
  3. Find all transaction where the ACL time is excessive (>100ms).
  4. For ACLs even a fraction of a second is a very long time.
  5. Search for any other transactions for the same URL.
  6. If the same URL has a trend of long ACL times there may be a mis-configured ACL.

Validate the ACL is mis-configured

  1. Based on the URL you should be able to find the affected table. Usually it will be in the name however if it is not a standard List or Form page (for instance a UI Page) you may need to look through the contents of the page to see the source table(s).
  2. View the ACLs for the table.
  3. Look for any ACLs with Scripted Conditions.
  4. Verify the Script does not have nested for loops or any other violations of best practice.

Potential Root Causes

  • Poorly written Scripted ACLs
  • Excessive number of records being accessed

Fixes

  • Scripts should be re-written to be as efficient as possible.
  • Lists and queries should be properly limited where necessary.
  • Additional filters can be added to lists to cut down on the number of records returned.
  • Potentially re-structuring tables or columns to simplify the script.

Scenario 3: Mis-configured Business Rule

A very common scenario is when Business Rules are not written correctly or several rules are written poorly and stack up to create a poor user experience. Most commonly when a user creates or updates but this can occur any time a business rule runs with the exception of asynchronous types.

Symptoms

  • One or more users will report excessively long wait times saving or loading a record.

Verifying The Issue

  1. Navigate to Transaction Logs (All User).
  2. Find all transactions for a particular users session.
  3. Find all transaction where the Business rule time is excessive (>2000ms).
  4. Find all transaction where the Business rule count is excessive (>10).
  5. Search for any other transactions for the same URL.
  6. If the same URL has a trend of long Business rule times or counts there may be an issue with the BRs.

Validate incorrect Business Rules

  1. Open the Business Rules for the particular table.
  2. Look for Business Rules which match the conditions reported by user (when saving, when creating).
  3. Look for all After or Before rules (Async rules will not apply since they do not stop the user session).
  4. Verify there are not an excessive number or rules.
  5. While having many Business Rules is not bad or uncommon the efficiency of those rules needs to increase with the number. Otherwise many rules that are somewhat slow can add up to a long wait time for end users.
  6. Verify the content of the Business Rules.
  7. Like ACLs you should avoid making an excessive amount of queries or nested queries in a BR.

Potential Root Causes

  • Multiple poorly written Business rules
    • Example: 10 or more rules which each take 250ms will end up 2.5 seconds for the user, this is only a part of the process though so the over all time may end up being excessive.
  • Single poorly written Business rule
    • A single rule taking more than 1000ms will directly impact the users experience

Fixes

  • Scripts should be re-written to be as efficient as possible.
  • Minimizing the number of GlideRecord queries and number of results returned.
  • Potentially moving some scripts to asynchronous where possible

Scenario 4: Under-performing Node

A very uncommon scenario but also one that is not fixable except by ServiceNow. This will usually involve ServiceNow restarting and possibly decommissioning the node.

Symptoms

  • Multiple users will report excessive wait times performing any action in the system
  • Issue will usually be intermittent
  • Issue will occur more frequently at peak times

Verifying The Issue

  1. Navigate to Transaction Logs (All User).
  2. Find all transactions where the Response Time is excessive (>5000ms).
  3. Add the System ID column
  4. Verify that no other cause (Network, BR, ACL) is affecting the performance.
  5. This may manifest as a longer Session wait time, Semaphore Wait time or SQL time
  6. You may also try to eliminate transactions that are expected to take longer, such as those with larger outputs (Output length)
  7. You may also eliminate those requests which affect multiple System IDs
  8. If there is a node (System ID) which is constantly under-performing and where the same requests execute quickly on other nodes it may be isolated to that node.
  9. You may open a HI server ticket to confirm the node is not functioning correctly

Potential Root Causes

  • If the issue consistently happens even after restarting the node hardware may be defective
  • Many other causes are unknown especially since we do not have access to the back-end

Fixes

  • Ask ServiceNow to restart the node via HI ticket or Support

Scenario 5: Browser Performance

A very common scenario especially for environments with older browser (specifically Internet Explorer). Poor performance is almost exclusively limited to IE but there are scenarios where Chrome and Firefox will not work correctly.

Symptoms

  • One or more subset of users will report a performance issue especially loading forms or custom pages
  • Issue will be consistent and easily reproduced
  • Issue will only occur on a certain browser or machine

Verifying The Issue

Before verifying check that the user is actually using a Browser on their desktop or laptop and it is not related to a mobile device.

  1. Navigate to Transaction Logs (All User).
  2. Add the User Agent column.
  3. Find all the Transactions for a particular users session who is reporting the issue.
  4. OR Find all transactions where the Response Time is excessive (>5000ms).
  5. Eliminate any transactions which can be explained by other issues.
  6. Try to correlate any poor performance to a particular User Agent.
  7. Look up the Users User Agent string to determine which browser they are using. UserAgentString.com

Validate the browser is not running in “Compatibility Mode”

  1. For Internet Explorer ask the User to verify their version by going to “About Internet Explorer”
  2. If the user reports a version which does NOT match the User Agent reported in the transaction log they may have compatibility mode on which should be disabled.

Validate the Browser is supported

  1. Older versions of Internet Explorer (<10) generally do not perform well enough to have a good User Experience in ServiceNow
  2. Versions of Internet Explorer (<9) may not work at all

Validate extensions are not interfering

  1. Ask the user to disable extensions OR run an instance of the browser in private browsing mode (this usually disables all extensions)
  2. If the issue does not persist with extensions disabled review the extensions to see if any may be causing an issue
  3. Disable extensions one a time to verify.

Validate no errors are being reported

  1. Ask the user to verify there are no errors in the JavaScript console. Most browsers have a developer console which will output all JavaScript errors.
  2. Believe it or not some errors are expected but an excessive amount especially those which report missing resources could be affecting the user experience.

Potential Root Causes

  • Unsupported Browser.
  • Browser running in compatibility mode.
  • Browser extension impacting performance.
  • GPO pushing browser setting which impacts performance.
  • GPO pushing system setting which prevents caching.
  • Cache disabled through browser setting.
  • Browser unable to reach resources – this may also manifest as the UI appearing strange.

Fixes

  • Upgrading to the latest browser version.
  • Disabling Compatibility Mode.
  • Removing or resolving extension or browser configuration.
  • Verifying network configuration (firewall or proxy) allows users machine to retrieve correct resources.

Scenario 6: Excessive or Incorrect Client Side Scripts

Another common scenario is for the user to see poor performance due to too much form automation on the client side. Offloading some work to Business rules or re-architecting it for performance will usually resolve these.

Symptoms

  • One or more users will report an issue with forms taking a long time to load.
  • Users may reports jittery or unresponsive form when changing values.

Verifying The Issue

  1. Navigate to Transaction Logs (All User).
  2. Find all transactions for the table or form where the issue is reported which have an excessive (>2000ms) Client script or UI Policy time
  3.  Note that these values will only include those scripts run when loading.
  4. If script time is consistently long there may be an issue with the number or efficiency of the script.

Validate the script is impacting performance

  1. For either/both UI Policies and Client Scripts disable all scripts.
  2. Enable each script testing the form.
  3. Take note of the load time. If a single script increases the load time dramatically there may be an issue with the script.
  4. If each script or policy increases the load slightly resulting in a long load time, there may be an issue with the quantity of scripts.

Validate the form layout

  1. Validate there is not an excessive amount of fields,tabs, related lists or embedded lists on the form.
  2. Validate there are not custom Formatters present.
  3. If there are an excessive amount of controls on a form it will directly impact the load time for the user.

Potential Root Causes

  • Too many Scripts or Policies.
  • Poorly written Scripts.
  • Excessive amount of AJAX transactions in scripts.
    • Some AJAX is OK, but while those transactions are occuring synchronously the user is unable to do anything.
    • Synchronous AJAX should only be used when the result is NECESSARY for the user to proceed.
  • Too many fields present on form.
  • Too many lists or embedded lists.

Fixes

  • Re-writing Scripts to be more efficient.
  • Refactoring or combining UI Policies.
  • Changing synchronous AJAX to asynchronous where possible.
  • Changing some form automation to run as Business rules where possible.
  • Removing fields or creating additional views to only show necessary fields.
  • Removing unnecessary fields or lists.

Scenario 7: Incorrect User Expectations

A perfectly valid expectation is that every page will respond quickly but not every page in ServiceNow is equal. Sometimes 5, 10 or even 30 seconds is reasonable when the system is crunching 1000s or rows to display in a report.

Symptoms

  • User reports performance issue for particular pages.

Verifying The Issue

  1. Navigate to Transaction Logs (All User).
  2. Find all the transactions for the users session.
  3. Find any excessively long transactions (>5000).
  4. Take note of the URL.
  5. If the user is loading home.do or a report it may be valid to expect wait times greater than 5 seconds.

Potential Root Causes

  • User is loading a Report or Dashboard.
  • User is loading too many records on a list.

Fixes

  • Explain to the user the wait times are valid.
  • Reconfigure the report to have additional conditions.
  • Reset the user preference to display less rows.
  • Change the users Home page if possible.

Getting the Current URL

Recently ran into this wall which stumped me for a few.  Had to get the full URL including the path from a UI Page on the server.  Using my book as a reference (shameless plug) I found RP.getReferringURL(), but this only returns the page name I found out and RP is not a usable object everywhere.

The solution was to find the actual Request object through the Transaction.

//
var tr = new GlideTransaction.get();
gs.print(tr.getRequest().getRequestURL());
//

So what is happening here? GlideTransaction.get() returns the current transaction. It has a scriptable method ‘getRequest()’ which returns the underlying request for this transaction which an HTTPServletRequest. Not every method is going to be available since SN has whitelisted/blacklisted many but getRequestURL().

There you have it the full URL with the path! What other methods have you found to get the URL?

ServiceNow Sync Plugin for Sublime Text 3

Update (5/2/16): Got the code on github for whoever is interested.  Pull requests are welcome.   github repository

I think many people had given up hope but I decided last night to finally get to creating my ServiceNow Sync plugin for Sublime Text 3. I also took the opportunity to clean up some bugs and introduce diffing.

Installation

Open the Console (Ctrl+) and paste the following code and run

import urllib.request,os,hashlib; pf = 'ServiceNow Sync.sublime-package'; ipp = sublime.installed_packages_path(); urllib.request.install_opener( urllib.request.build_opener( urllib.request.ProxyHandler()) ); by = urllib.request.urlopen( 'http://snaug.com/snowsync/' + pf.replace(' ', '%20')).read(); open(os.path.join( ipp, pf), 'wb' ).write(by)

You may need to restart Sublime to see the changes.

If everything worked correctly you should get a new "ServiceNow Sync" menu.

Alternate Installation

You can install manually by placing this file in the Installed Packages folder for ST3.

 

Creating a Project

To start using the plugin you need to be in a Folder View, so go to File > Open Folder.

If you try to use the plugin otherwise it will give you an error. It used to just crash, so that's an improvement.

Next you will want to create the connection to a ServiceNow instance.  Go to 'ServiceNow Sync' then 'Connect To ServiceNow'

A prompt will open up at the bottom of the screen asking you to enter first your instance name (exclude the .service-now.com).

For instance example if your instance url is https://snaug.service-now.com` then you would put “snaug”.

Next it will prompt for your username and password.  Be careful entering the password as the text is not hidden.

This will create a service-now.json file in your folder.

After this I will usually just try “Test Connection” to make sure everything is working.

 

Syncing Tables

To sync a table you first start by going to the “Sync Table” option in “ServiceNow Sync”.  This will pull up a list of tables which you can search through.

Once a table is selected you will be prompted (at the very bottom) to provide some details about what you want to sync.

The first prompt will ask you for the folder name, this usually matches the table but can be anything.

After that you will be asked some details about the content.  The first option is the name field, which is almost always “name” this will be used as the local file name as well.

The script field is the field which should be used as the contents of the file.  Usually this is “script” as in Business Rules or Script Includes but it could also be: “html” for UI Pages, “xml” for UI Macros”, “programmatic_content” for Dynamic Content Blocks and so on.

In fact this can be any field on a record.

The last option is the file extension.  This should match the content, such as “.js”, “.html” or  “.css”.  The correct extension will give you the appropriate highlighting.

Once complete a folder will be created matching

One note, it is possible to sync the same table multiple times.  You can create several folder that all map to the same table, they will not interfere with each other.  You can also use the same technique to sync to different fields, for instance UI pages can have several folders one for HTML, one for Client Script and one for Processing Script.

Syncing Files

After syncing a table you can now sync files for that table.  To do this just right click on the folder and select ‘Sync File’.  This will retrieve a list of files, once you select the file from the list it will be created locally.

From time to time it may be necessary to resync the same file to a different record in ServiceNow.  In each table folder there is a service-now.json file which has a record of all the files and their sys ids in ServiceNow.  You can remove a particular entry or change it.

Saving Records

Saving a file to a record is all part of the Save process.  Once you save the file will be automatically uploaded to ServiceNow.

Creating Records

Creating a local file and saving the file will create a record in ServiceNow.  It will use the file name as the name in ServiceNow.

Deleting Records

No.  That’s nuts.  Deleting local copies WILL NOT delete anything on your instance.  Deleting records in ServiceNow will not delete the local files.

This may save you one day, it already has saved me multiple times.

File Differences

If you are working in an environment with multiple developers it is very important not to overwrite others work.  To prevent this the ServiceNow Sync plugin will check for differences between the local and remote copy.  If it finds differences it will alert you and ask what to do.

If you are saving, you will be prompted to Overwrite the Remote copy or view the differences.

You may also check for updates by right-clicking on a file and selecting “Compare to Server”.  If there are differences you can Overwrite your local copy or view the differences.

Opening A ServiceNow Record

One other quick feature, if you are in the body of a file, you can right click and select ‘Open ServiceNow Record’ to quickly open the record in the instance.

Sharing Folders

It is possible to share an entire folder with co-workers.  I actually check the whole folder into Git.  The only thing you want to make sure is to not check in the “service-now.json” in the root folder as this has the authentication information in it.  All the sub-folder JSON files are ok to check in or share.

Known Issues and Enhancements

Unicode Characters break the file diffing. Fixed

At some point I would like to add the ability to set an update set on the system, in fact the menu item is already there but disabled.  Some day…

Any other issues please reach out to me directly until I can get this on Github.

YouTube Like Loader

Just a simple script to add a YouTube like loading bar to the top of the page for AJAX Requests. Many users said they wanted a way to know when the system was “thinking”. There actually is a loading GIF in the top right (I noticed it in Fuji) but it’s a bit hard to see. This adds a very obvious loader to every page, even popups and CMS page.

This should go in a Global UI Script. Tweak as necessary!

if(self == top){
var loadingDiv = document.createElement("div");
var loadingInterval = false;
var loadingIntervalWidth = 0;
var stopLoading = false;

loadingDiv.style.width = "100%";
loadingDiv.style.position = "fixed";
loadingDiv.style.height = "2px"
loadingDiv.style.top = "0";
loadingDiv.style.left = "0";
loadingDiv.style.zIndex = "2000";
loadingDiv.style.background = "#29d";
loadingDiv.style.display = "none";

document.addEventListener("DOMContentLoaded", function(event) {
document.body.appendChild(loadingDiv)
});

function startProgress(){

if(loadingInterval == false){
stopLoading = false;
loadingDiv.style.display = "block";
loadingDiv.style.width = "0%";
loadingIntervalWidth = 0;
loadingInterval = setTimeout(incrementProgress, 50);

}
}

function incrementProgress(){

loadingIntervalWidth++;
loadingDiv.style.width = loadingIntervalWidth.toString() + "%";
if(loadingIntervalWidth < 100 && stopLoading == false){ loadingInterval =setTimeout(incrementProgress, 50); } else { loadingDiv.display = "none"; stopProgress() } } function stopProgress(){ stopLoading = true; loadingDiv.style.width = "100%"; loadingInterval = setTimeout(hideProgress, 100); } function hideProgress(){ loadingInterval = false; loadingIntervalWidth = 0; loadingDiv.display = "none"; loadingDiv.style.width = "0%"; clearTimeout(loadingInterval); } CustomEvent.observe('ajax.loading.start', function() { startProgress() }); CustomEvent.observe('ajax.loading.end', function() { stopProgress(); }); }

Using Jelly In ServiceNow Release Candidate Shipping

I am happy to announce I have started sending out the first Release Candidate for Using Jelly In ServiceNow.  I decided to keep the pricing as is at $25 USD for now until the final version comes out.

To purchase you can use the following form and I will send as soon as PayPal alerts me.  I try to do it immediately but sometimes it takes up to a day.

Update: I have started using Gumroad to distribute the book now instead of manually sending copies out via email.

Get the book now!

Table of Contents

0 Introduction
1 Jelly
1.1 Key Concepts
2 Jelly and ServiceNow
2.1 UI Pages
2.1.1 Page Template
2.2 UI Macros
2.2.1 Formatters
2.3 Other Jelly Scriptable Areas
3 Jelly Scripting
3.1 Jelly Tag Library
3.1.1 Variables
3.1.2 Output
3.1.3 JEXL Expressions
3.1.4 Conditions
3.1.5 Looping
3.1.6 Undocumented Jelly Tags
3.2 Glide Tag Library
3.2.1 Using References
3.2.2 Scripting Tags
3.2.2.1 Running JavaScript
3.2.2.2 Scripting Helpers
3.2.2.3 Using Functions and Macros
3.2.2.4 Miscellaneous Functions
3.2.3 UI Tags
3.2.3.1 HTML Elements
3.2.3.2 Form Elements
3.2.3.3 List Elements
3.2.3.4 Choice Lists
3.2.3.5 Content Management Components
3.2.3.6 Knowledge Base Components
3.2.3.7 Chart Components
3.2.3.8 Extended UI Components
3.2.3.9 Other ServiceNow Components
3.2.4 Undocumented Tags
3.3 Debugging
3.3.1 Debugging Tags
4 Advanced Usage
4.1 Glide and Jelly JavaScript Objects
4.1.1 Executing Jelly
4.1.2 Render Properties
4.1.3 Jelly Contexts
4.1.4 Evaluating Expressions
4.2 Creating UI Extensions
4.3 Decorators
4.4 Overriding Default Templates
4.5 System Properties
5 Jelly Examples
5.1 Core Concepts
5.1.1 Multiple Phases
5.1.2 Fizz Buzz
5.1.3 Fibonacci Numbers
5.1.4 Bubble Sort
5.1.5 Calling Functions
5.1.6 Towers of Hanoi
5.2 Static Page
5.2.1 Hello World
5.2.2 Simple Calculator
5.2.3 Output A List
5.2.4 Simple Form
5.2.5 Form Using Macros
5.3 Dynamic Page
5.3.1 Dynamic Calculator
5.4 Multi-Page Application
5.4.1 Simple Portal
5.4.2 Advanced Form
5.5 AJAX Application
5.5.1 AJAX Form
5.5.3 AJAX Form Widget
5.6 Custom Formatter
5.6.1 Custom Element
5.6.2 Form Widgets
5.7 Mobile Page
5.7.1 Mobile Only Page
5.7.2 Responsive Page
5.8 Frameworks and Libraries
5.8.1 Datatables List
5.8.2 Twitter Bootstrap
6 Index
6.1 Jelly Language Reference
6.2 JavaScript Objects Reference
6.3 LIcenses and Copyrights

Two Big Announcements

One, I am at Knowledge15!  Please come say “Hi” at the Cerna Solutions booth.

Second, and this is the reason I have not been blogging much lately, I am nearly done writing my first – and I am pretty sure the world’s only -and I am certain the world’s LAST – book on Jelly in ServiceNow!

Here is a quick summary of what the book “Using Jelly in ServiceNow” contains:

  • Start to end walkthrough of every function with examples
  • Full page examples with walkthroughs
    • Everything from simple pages to portals and AJAX pages
    • Examples showing how to use Frameworks
  • Documentation on all Glide API Interfaces for Jelly
  • Full language reference
  • Full JavaScript Object reference
  • Over 200 Pages of content

During the conference I am offering pre-orders at a special price of $25.  As soon as the book is ready (~Mid April or earlier) it will be delivered by email.

Any questions feel free to stop by the booth or leave a comment here!  Thanks.


Get the book now!

Update: Some folks were asking for more details so here is the Table of Contents for Using Jelly In ServiceNow.

  • 0. Introduction
  • 1. Jelly
  • 1.1. Key Concepts
  • 1.2. Jelly and ServiceNow
  • 1.2.1. UI Pages
  • 1.2.2. UI Macros
  • 1.2.2.1. Formatters
  • 1.2.3. Other Scriptable Areas
  • 2. Jelly Scripting
  • 2.1. Jelly Tag Library
  • 2.1.1. Variables
  • 2.1.2. Output
  • 2.1.3. JEXL Expressions
  • 2.1.4. Conditions
  • 2.1.5. Looping
  • 2.1.6. Unused Jelly Tags
  • 2.2. Glide Tag Library
  • 2.2.1. Setting Up A Reference
  • 2.2.2. Scripting Tags
  • 2.2.2.1. Running JavaScript
  • 2.2.2.2. Scripting Helpers
  • 2.2.2.3. Using Functions and Macros
  • 2.2.2.4. Debugging Tags
  • 2.2.2.5. Miscellaneous Functions
  • 2.2.3. UI Tags
  • 2.2.3.1. Basic HTML Elements
  • 2.2.3.1.1. HTML Elements
  • 2.2.3.1.2. Form Elements
  • 2.2.3.1.3. Choice Lists
  • 2.2.3.2. ServiceNow Elements
  • 2.2.3.3. Miscellaneous Tags
  • 2.2.3.3.1. Undocumented Tags
  • 2.3. Debugging
  • 3. Advanced Usage
  • 3.1. Glide and Jelly JavaScript Objects
  • 3.1.1. Executing Jelly
  • 3.1.2. Render Properties
  • 3.1.3. Jelly Contexts
  • 3.1.4. Evaluating Expressions
  • 3.2. Overriding Default Templates
  • 3.3. System Properties
  • 3.4. UI Element Tags
  • 3.5. Creating Custom UI Tags
  • 4. Jelly Examples
  • 4.1. Core Concepts
  • 4.1.1. Multiple Phases
  • 4.1.2. Fibonacci Numbers
  • 4.1.3. Bubble Sort
  • 4.2. Static Page
  • 4.2.1. Hello World
  • 4.2.2. Simple Calculator
  • 4.2.3. Output A List
  • 4.2.4. Simple Form
  • 4.2.5. Form Using Macros
  • 4.3. Dynamic Page
  • 4.3.1. Dynamic Calculator
  • 4.4. Multi-Page Application
  • 4.4.1. Simple Portal
  • 4.4.2. Advanced Form
  • 4.5. AJAX Application
  • 4.5.1. AJAX Form
  • 4.5.3. AJAX Form Widget
  • 4.6. Custom Formatter
  • 4.6.1. Custom Element
  • 4.6.2. Form Widgets
  • 4.7. Mobile Page
  • 4.7.1. Mobile Only Page
  • 4.7.2. Responsive Page
  • 4.8. Frameworks and Libraries
  • 4.8.1. Datatables List
  • 4.8.2. Twitter Bootstrap
  • 5. Jelly XML Language Reference
  • 6. JavaScript Objects Reference