Synopsis
Umit is a nmap frontend started with the sponsoring of Google during the Summer
of Code 2005 and improved during Summer of Code 2006. Like any other young tool, Umit needs
some adjustments and improvements to fulfill the needs of it's users. After the release of
it's last version, the project received loads of feedbacks from the community, pointing out
and suggesting new features.
The purpose of this mentoring proposal is to create new tools to integrate Umit and
make network admins life's easier.
How to make a proposal
Last week I made a speech at Universidade Estadual de Goias about open source and Summer of Code. During the speech I talked about how the proposal should be made, and now, I'm making the presentation available in Brazilian Portuguese and in English to help students on their duties of creating a good proposal to increase their chances of been accepted this year.
Ideas list
- UmitMapper
- UmitWeb
- Nmap wrapper for Python
- NSE Facilitator
- Wizard and Profile interface designer
- Independent features
Plus
Some work that I've been developing or planning to develop soon, independently of the Summer of Code.
After analising fe3d, Cheops and NCartographer, I thought about how I should implement this feature at Umit in a way that it won't be a processor resource eater, like fe3d, or visually limited as Cheops and NCartographer. Another preoccupation was to don't add more dependencies. I used them (fe3d, Cheops and Ncartographer) and analised their behaviors and solutions to take a better decision for the Umit Mapper problem.
The conclusion was that user needs more than a mapper: user needs a useful mapper that they can use with a few clicks to browse the network in the easiest way without the need of painfully saving scan results and openning them in another application just to browser it. The Umit project's duty is to make users lifes easier, helpping them to do their jobs faster, and here is where UmitMapper comes in help.
fe3d
This software use the nmap XML output to generate 3D graphics of the scanned network.
The visual efect is amazing. I confess that my first thought was about doing something
similar on Umit. However, analysing this paradigm, I realised that fly across a 3D model
isn't much convenient, mainly when the model is representing a huge network. Because of it's
low usability, this model won't atract users, even with the cool visual efect provided
by a 3D model that we can show to our friends and say "all your networks are belong
to us".
Thinking about the needs of a network administrator, I perceived that what he really
needs is a way to have a global visualization of the network and it's topology. Another thing
I noticed is that, generally, administrators have the interest on navigate through the topology
to see selected informations and details about each host without getting lost. This feature
unfortunatelly isn't provided by fe3d, making it kind of useless for the real network
administrator that needs to save time navigating the network and hosts informations.
Cheops/Cheops-NG
Despite the not that cool and amazing visual efect owned by fe3d, Cheops provides
more functionality and usability. Many of it's current features is already provided by
nmap (as I could see, it doesn't uses nmap) and lots of it's interface features are at Umit.
However, talking about it's mapper, when it comes to a huge network it becomes harder to
navigate through it. Small networks are well attended by Cheops, but our goal is to attend
well networks of any size.
NCartographer
One of the most recent tool developed for this purpose, made on last Summer of Code,
is also intended for an overview of the network found in the scan result. Yet, as it is not
integrated with nmap, you must make the scan in separate, save the result and them open it
at NCartographer. The result view also doesn't scale well with large scan results, and we
need a solution for networks of any size.
Umit Mapper
The Umit Mapper goal is to provide a network topology diagram generation as well as
an ease way to navigate through it for a quick analysis without getting lost in a huge network.
This way, user can navigate through the informations of hosts from each network without
getting confused. Umit Mapper is going to be developed in PyGTK, and will be integrated to the
Umit interface. At first sight it can feel frustrating the fact that the mapper is going to
be developed in PyGTK, but there are many advantages:
- No more dependencies
- Performance
- Easier integration to Umit interface making user's life much easier
- Easy customization
- Faster development when compared to a 3D graphical model
- Higher manutenability
As usual, I developed a visual prototype for the Umit Mapper. It doesn't have any functionality yet. It was developed just to serve as a basis for it's design. At the window shown by the screenshot below, user will be able to have a global view of the network and it's topology. The blue globe is been used as a network icon, and the blue square with arrows means a router. However, these icons and the window design are temporary. The accepted student will have to improve the interface appearence and make a better design for it.
Higher network topology view (with scanned networks)
By clicking on any network icon, user can see the hosts that belongs to that network. The following screenshot is a simplified visualization of the scanned hosts, showing their respective ips and operating systems icons (if recognized) below the icon and the address of the current network. This network address and icon is intended to let user know where he is all the time, avoiding user to get confused or lost about where he is inside the topology. If any host icon is clicked, is shown in the right side of the window informations about the host, like openned ports and it's vulnerability level. Clicking on the network icon, gets user back to the screenshot above.
Simple visualization of hosts from a selected network
The next screenshot is another visualization option that shows the openned ports of each host below it's icon. In this view, users won't need to click over each icon to realise which ports are openned making it easier to have a global view of the network and the services provided on it.
Another visualization of hosts from a selected network
With the model of the next screenshot, it's possible to easily navigate through the network listing hosts informations individually and with an attractive visual.
Individual visualization of hosts from a selected network
User will be able to navigate through the result using filters, that group scanned hosts by a selected common attribute like openned port, vulnerability level or operating system.
Filtering hosts by open ports, vulnerability level and operating system
Umit Mapper is intended to be more than a mapper and provide features really useful for it's users with higher usability.
Despite the mockups and the ideas I put here, the accepted student will have the freedom to suggest more ideas, discuss and argue about the proposal and some statements I made. This proposal is only the basics to help them creating their own proposals.
Some requirements that students must pay attention while making their proposals:
- Show network routers, when found
- Show the whole network and hosts connected to it
- Make an impressive and eyecandy respresentation of the network, yet with low resources consumption
- Must be developed in PyGTK and embeded in Umit, so users won't have to save the scan, and them open another program and open the saved scan again. The idea, is to make user's life easier and network scanning experience painless.
- Must be developed thinking in software re-use, in such a way that the core can be used by other kinds of interfaces, like a web interface.
- Must be light weight and add no dependencies to the software. Though, if you want to add a dependency, you can suggest it and explain me your reasons. This is not rock written.
- Shouldn't make use of too much animation effects. Few are ok, though.
- Changing from a normal result view and a mapper view must be easy, quick and smooth. Shortcuts could be used here to make user's life even easier.
- Fullscreen mode
- Integrate with the Quick Connect feature that is going to be developed in this SoC :)
The goal of this proposal is to create a web interface able to reproduce the main Umit interface functionalities. This interface is intended to make network admins lifes easier, by providing a tool that they can use to make remote scans from anywhere in the globe, using just a browser, and having the scan result in their mail or even in the browser as a report.
Here is why users need a web interface: sometimes you just want to know what is happening with your network without the need of going inside it using a VPN or an SSH to run a command line scan or call the Umit interface remotelly. Sometimes, you're not at home or with your laptop and you need to keep your eyes in your network using your grandma's computer without the need of installing anything there and yet having the same level of functionalities you would have by using the standard interface.
Some requirements that students must pay attention while making their proposals:
- Must run in stand-alone mode, using a embeded web server. User shouldn't need to install and configure a whole web server to use it. The idea is to make it work out of the box.
- Should not use frameworks like django, turbo gears or zope to avoid portability and dependencies issues. Althought theese are really good frameworks, I don't think they fit here because of the simplicity nature of theese tools. Although, the student may convince me that this is a good idea.
- Must be able to send the scan results by email
- Must have an authentication process
- Show scan results in Real-time (like Umit does)
- The web server must be prepared to run as root, to make nmap able to run any option. Because of the obvious security issues of running a web server as root, the student must think in a good solution for the problem, that could be using sudo to run nmap, or chroot the web server, or something else.
- Role Based Access Control, to limit what options users can run
- The installation procedures must be documented and follow the package
- Must be portable, and run on Windows, Mac OS X and Linux systems
- The web interface must be tested in main web browsers, like IE6, IE7, Firefox, Safari and Opera.
- Must have an automated installation process
- Download scan result as an usr (Umit Scan Result -> A Nmap XML output with some adds) file
- All user input must be sanitized very carefully to avoid command injection
- Support SQLite and PostgreSQL at least
- Scan History, so users can see which scans he made in the past
- Search on results, like in Umit standard interface
- Make use of the Scan Scheduler
The wrapper intention is to provide a module from which you can create an Nmap instance, set the desired options and targets and run it without the need of executing it in another process. The wrapper must allow access of Nmap funcionalities like estimated time to finish the scan and runtime user interaction.
User's almost won't note any change with this wrapper. This is a feature that is going to make Nmap and Umit developers life's easier, while adding new features or even integrating with future Nmap options and functionalities.
The idea of the NSE facilitator is to provide a better integration of Umit, Nmap and NSE in such a way that even a newbie could easilly run and share scripts with a few clicks, instead of finding them in the web, downloading, reading documentation to know how to use it with Nmap and running a long command to be able to use it.
Some requirements that students must pay attention while making their proposals:
- List available scripts in the main menu and in a poup-up menu, so user can select a given host, and run another scan with the chosen script on it.
- List and download scripts available at umit website. This listing must contain scripts descriptions, size if it needs root to run and main functionalities.
Wizard and Profile interface designer
Umit's command wizard and profile editor were designed to be created in runtime, based on an XML definition, which says the interface options and their arrengements in the GUI. Altought this XML file is very easy to understand and modify, it is not the best way of making the user's life easier. These editors were designed to allow user to add or remove any option he feels like, or even add the newest nmap option that is not covered by the current Umit version yet. Editing the XML file, user can even combine options and make them easier to use during command or profile creations. The interface designer is a program that would let user add, remove or edit the profile and wizard interface in a visual way and without the need of hand editing the XML file.
Some requirements that students must pay attention while making their proposals:
- Allow adding, removing or editing and interface option
- Apply the changes, so user will be able to use them after Umit restart
- Undo/Redo feature
Besides projects like UmitMapper and UmitWeb, there are some Umit improvement ideas that students could work for. Although these are feature development ideas, they're not less important that the independent projects listed above. Umit needs to grow in every directions, mainly in usability directions, because it's main purpose it's to ease some every-day tasks performed by network admins.
Students with a good background in Python, GTK and usability are elegible to work on these features during the summer. Here follows a list of some of the features that students can invest their time on:
As known, non-root users has limited options on nmap. Currently, this matter is not well supported on Umit, and when a non-root user tries to use a root option, Umit behave strangely. Sometimes it show a error msg, but sometimes it doesn't. I'm planning on making Umit avoid the use of root options by non-root users. Umit will also have a better support for nmap non-root error messages in case user insert by hand those root options at the command field. This will make Umit more user-resistent.
Umit should be able to receive command line arguments, so user's could be able to call umit with arguments specifying that it should run a scan profile after it is openned, for example, and users that use command line, would feel more motivated to use umit because they won't need to run one command to open the interface, waits until it is completly openned and them type a command to be executed or choose a profile. These seconds are too much for a network admin and he surely wants to save them by doing everything he wants in the command line, and them just go after Umit to see the results in a more confortable way.
This feature is responsible for sending a formated result to user's e-mail, with the normal and XML output attached. As this feature is going to be used by other projects, like the Network inventory, Scan Scheduler and UmitWeb, it must be developed inside umitCore module, so it can be re-used by other projects.
Currently, the profile creating/editing/removing progress is not intuitive. This issue ends with usability problems, and must be fixed quickly. Yet, the profile editor interface itself can be better designed and arranged to make user's life even easier.
Nmap has some runtime interaction shortcuts that users can call to increase/decrease verbosity of the output, turn on/off packet tracing or even get the elapsed and estimed time to finish of the scan. But, currently, these options can only be used while running Nmap from command line. The idea, is to let user call these options from Umit interface, and maybe create some widgets to ease the task of using such options.
A progress bar to show the scan progress to users. This is specially usefull for scans that takes too long to finish. The progress bar can make use of the Nmap's runtime interaction option which shows the elapsed time and the estimated time to finish of the scan.
The Network Inventory is intended to help users to easily keep track of the hosts on his network. This is the way this feature is going to work: User enter with basic informations about a regular scan (targets, nmap scan options, etc) and the network inventory system will take care of execute the scan regularly to check changes on the network, in a frequency defined by the user. If any relevant change is noted since the last network scanning, user is notified about this change (through the Umit interface or by e-mail). Let's consider the hypothesis that a machine was violated and the intruder left an active backdoor running and, after that, he left the machine withou been noted. When the Network Inventory scan the network again, it will perceive the new openned door at this machine and notice the network administrator. With this new Umit feature, the administrator can have more chances to detect this kind of trouble in time to avoid more losses.
Another feature related to the Network Inventory is the history. Each verification
scan made by the inventory is logged at the history. Using the history, the network
administrator will be able to verify previous states of the network.
The maximum amount of logs that is going to be stored can be defined by the user.
Inventory is associated with another Umit feature that is going to be described ahead in this proposal. There is only one dependency related to this new feature, that is the SQLite. However, SQLite now is part of the python's 2.5 default library and won't be a dependency anymore for those who is going to use python 2.5. Anyway, SQLite is a pretty small library and is very easy to install.
Sometimes, the user scan the network searching for hosts serving on a given port. Later, with this information, he usually tries to connect to this host using a client application. Currently, a Umit's user must scan the network through the Umit interface and mannually execute an external client to connect to the host. This new feature will allow user to connect to the host using his preferred client directly from Umit's interface by clicking on a quick-button or a right-click over the host.
Let's suppose that you has just scanned and verified that a given host is serving FTP, and now, you want to establish a FTP connection to this host. With a single click over the quick-connection button, or a right-click over the host, Umit will try to execute the default FTP client application providing the host and port information for the connection. If there's no default FTP client application registered, Umit will ask user to provide information about which application should be executed for this kind of service. This asking phase will happen only once, on the first attempt to use this feature to connect on the selected service. The informations provided by the user are going to be stored for future use, and can be easily edited whenever the user feel like it.
Some examples of common scanned services that user usually tries to connect after the scanning:
- SSH - OpenSSH, putty, etc.
- TELNET - telnet
- HTTP - firefox, opera, lynx, elinks, etc.
- FTP - ftp, gftp, etc.
This feature is very useful because is going to ease the everyday's scan-connect task, and it won't add any dependency.
In the end of a speaking about Umit, I was questioned if Umit had a Scan Scheduler. At first sight, I felt strange about the question, but soon I realised it's importance. The Scan Scheduler will be useful to schedule scans and to support the network inventory. As a prototype, I developed a cron like parsing module for the Scan Scheduler and a daemon to verify if something is scheduled to be executed in the moment. You can see these modules here:
Despite they're only prototypes they already works pretty fine, and minor adjusts will be needed. The Scheduler will run as a daemon and user will be able to start it from Umit interface or at operating system initialization. This daemon responsibility is to take care of scan and network inventory schedules, with the duty of verify relevant changes and notify user by e-mail. This feature won't add any dependency to Umit.
Some requirements that students must pay attention while making their proposals:
- Umit interface for interacting with scan scheduler to add and admin scheduled scans
- Daemon communication interface, so Umit, UmitWeb and other application can communicate with it and administer scheduled scans
- Must pay attention with security here. The daemon should not receive remote requests to avoid security issues. There is probably more security issues involved in this question, and the student must pay atention on this manner.
- The daemon must be portable, and work on Windows, Mac OS X and Linux
Plus
Usability must be matured, and the Umit goal is to be always a step forward on this subject. During the development of the proposed features for this SoC, I'll be preoccupated on a better adequacy of the interface for the user daily needs, increasing keyboard short-cuts and grabbing informations on how users use Umit and how they behave while working with it. These informations are useful to place widgets in better places, define shortcuts, add features, etc. To grab these informations, I developed a module called usabilityTest, which is a suite that essentially does the following: shows a form to grab user experience information, starts umit, video-recording and key logging user's desktop and asking him to accomplish some tasks at Umit. Here follows some screenshots:
User form that collects some user personal and experience informations. These informations are intended to help us understand user behaviors.
Yet the user form that collects some user personal and experience informations. In this particular form, user will say which experience does he have with networks, scanning tools, frontends, etc.
The usability test suite was develop in a way that it can be used by any other project, because the user information form and the tasks at the task manager are defined in a XML file. This way, other projects can change the user information form questions and the tasks on task manager to better fit their needs and easily grabbing important informations from user behavior.
Currently, it uses xvidcap to record the screen and lkl as a keylogger. Although the Task Manager is almost done, it is possible to run and test it without any problem. The usability test suite is been kept inside Umit repository, inside the usability directory, but this suite can turn into a separated open source project in the future.
Usability test phase: Umit interface and the Task Manager window. Everything user does here is video-recorded and the keys are logged.
Here is a sample video that I made testing the usbility test suite:
The video goes pretty fast because the machine on which I ran the test is very slow, and the video recording tool I used (xvidcap) wasn't able to capture the minimum number of frames to make a smooth video. As this video was only a test to see the result of the usability test suite execution, we don't need to worry about the video speed yet.
Some automated tests are also going to be made for Umit GUI using Dogtail. The idea is to make sure about the sanity of the GUI behavior and make sure that it is able to handle every Nmap option and expression without raising any exception.
Umit already works on Mac OS X, but it's very hard to install it from source there. As we don't want to botter user that much, I'm planning on making a installation package (.app) for Mac OS X, and spread Umit among OSX users.
Umit porting and installers for Maemo
From Maemo oficial web site:
"What is maemo?
Maemo is an open source development platform to create applications for Nokia Internet
Tablet products like Nokia N800 and Nokia 770. The platform gives developers a powerful
Linux based development environment and optimized end-user interface for handhelds."
Maemo already have a nmap package for it, and as it is used to access internet and networks wirelessly, Umit can be a great tool to have on it. It already runs on it, as we can see in the pics bellow, but it needs some interface improvements.
A porting of Umit for Maemo would include using Hildon to adapt the GTK interface appearence and behaviors and a package to ease the task of installing it. As it is a portable device, it's really boring to open a terminal, get into the directory of the package, un-tar it and type: "python setup.py install". There is a package system based on apt-get for Maemo, and we shaw use it to make things easier.
Screenshot of Umit splash on Gustavo's Nokia 770
Camshot of Umit splash on Gustavo's Nokia 770
Currently, the Umit web site is not good enough for an open source project. It doesn't have a section for documentation, downloads, about, etc. And the design looks ugly. I'll be working on it while students do their works. It's also a good opportunity to make sections in the website to hold their projects, and respectives documentations as well.