Tagged : python

in Tech

Citrix NetScalers and the netscaler-tool

The netscaler-tool is a Python script that leverages the Citrix NetScaler Nitro API. If you need to easily discover when something goes wrong with either the NetScaler or its services, you can use netscaler-tool to integrate NetScaler statistics into your existing open source monitoring and alerting projects.

In most large-scale web companies, load balancers (such as the Citrix NetScaler) allow for server failure and traffic increases without any harm to services or the need to page anyone. Keep in mind that this is only true when capacity planning was done correctly, to allow for absorption in these types of events. Eventually, the dead servers will get fixed and/or more capacity will be added to tiers if they are running hot, but that can all be addressed at leisure.

Before the netscaler-tool, monitoring and managing of NetScalers might have consisted of using SSH with an empty passphrase for the private key, creation of a Linux system role user, and parsing output. Some maybe even utilized SNMP. But these are old methods that have been superseded by APIs. Why live in the past and make things harder than they should be?

The NetScaler offers a few management methods, including several APIs (SOAP and Nitro). The SOAP API seemed like an improvement over SSH, but usability was not trivial. Even if you figured out how to code around the API, you might need to generate new NSConfig.wsdl or NSStat.wsdl files from time to time, in order to add a new feature and/or decrease compilation time and program size. Luckily, the Nitro API is much more user-friendly, where messages are in JSON format and there is no need to tweak anything on the NetScaler.

Using the netscaler-tool, anyone can currently monitor/alert on:

  • Node failovers
  • SSL certificate expirations
  • Differences between running and saved configurations
  • Surge queuing on backend services
  • Critical events (e.g. Packet CPU increase, hardware failure)

Feature releases will also allow fetching any of the NetScaler node statistics, which include:

  • HTTP request/s
  • Inbound/outbound IP packets/s
  • Inbound/outbound throughput
  • Current client connections

Here are some statistics from one of our NetScaler pairs that would benefit from this future release. Note: these statistics are based on both internal and external traffic:

  1. HTTP requests/s: ~76k
  2. Inbound IP packets/s: ~990k
  3. Outbound IP packets/s: ~920k
  4. Outbound throughput: ~4.8Gb/s
  5. Inbound throughput: ~4.8Gb/s

Although the tool is currently geared toward fetching data from NetScalers, it can also change a few NetScaler objects; for example, services and load-balancing virtual servers. These abilities enable operators to gracefully stop and start tiered services without affecting clients. One use case could be new web releases. Assuming orchestration code has already been written for code deployment and bouncing of web servers, one could leverage the tool to:

→ disable a web service with a delay to give clients enough time to complete their current request,

→ stop the web server,

→ start the web server,

→ make another call to the tool to enable the service,

→ and then move onto the next web server.

If, for whatever reason, you decide to write your own tool instead of using the netscaler-tool, and don’t want to deal with the Nitro API, you can use the included netscalerapi module.

For more information and source code, see the <a href=”https://github.com/tagged/netscaler-tool”>Github</a> repository.

Using netscaler-tool, we’re able to submit data to Graphite and manipulate NetScaler performance statistics to build better monitoring and alerting.

The netscaler-tool allows us to keep tabs on the largest traffic generators that impact our backend and frontend services.

Alex Kaplan contributed to this post.

Brian Glogower

Brian Glogower is a senior systems administrator on the site operations team at Tagged

More Posts

Python Interface to the HipChat JSON API

One of the many modern challenges to scaling a business is keeping employee accounts consistent. Traditionally, businesses would use a LDAP or Active Directory server and make all their tools plug in to those services. When someone was hired, they would receive a single user account that worked everywhere. At Tagged, we wrote a tool called SUMP: Simple User Management Portal. SUMP hooks into all the systems that we use, such as Google Apps for Businesses, our LDAP server and Hipchat, which we use for group chat.

There were already a few libraries available to access HipChat’s API but none had the user creation features we needed, and we preferred Python. So, we decided to roll our own library and release it as Open Source. Since many organizations have their own tool like SUMP, often written in Python, we figured this has a high value for other groups. Writing these libraries can be so simple that it’s easy for many people to create their own implementation. By releasing it as Open Source, we benefit because it pushes others to use the same HipChat API. This causes HipChat to maintain it for all their customers, and not just for us, so we don’t need to worry that SUMP might stop working one day. We also get the added benefit of possible outside contributors.

When we wrote this code, we decided to make it as Pythonic as possible. The HipChat API returned JSON objects, which are more like dictionary data types, but it was clear there were distinct data objects. Every API call gets wrapped into a typed class. Rather than building up JSON objects, function parameters are converted automatically so the library has a very object-oriented feel. Finally, there are some sample command line tools in the ‘commands.py’ file to demonstrate some common use cases for the package.

While the library is simple, it’s designed with a couple abstractions to make it extensible. Behind the scenes is a base class HipChatObject and a partially applied call_hipchat function for accessing any arbitrary API function. This makes it possible for other developers to add features to the library very rapidly when HipChat introduces new functionality.

The code is available on both GitHub and Gitorious and there have already been some pull requests over Gitorious. It’s just a small contribution, but along with other projects like the Node.js Memcached Library – we’re paving the way for Tagged to do even more Open Source contributions in the future.

Yaakov Nemoy is a Systems Administrator at Tagged and you can follow him on Twitter and on Google+.