This project has moved and is read-only. For the latest updates, please go here.

News on Version 2.0 - with ongoing updates

Nov 15, 2010 at 2:06 PM

 Knowing you only or mainly developed this app for personal use and have a fulltime job I just had to ask after not speaking with you in over a year if you had made any headway on version 2.0 ?

 Granted I know this is opensource and anyone with any GOOD prgramming knowledge can make adjustments to it at will, However I particularly am NOT a good programmer but do love good software that works well.  Thats why I have been using this for almost 3 years now.

 If you have time. Please let us know what your plans or intentions are on Ticketdesk at the moment or if anyone else is reading this and has made any improvements themselves, I would like to hear about what others have done with it.

Coordinator
Nov 15, 2010 at 7:01 PM

I'm still working on 2.0, though not as fast as I'd like to (what else is new eh?). There are now two branches of the project in source control here. One is for 1.x, and the other is a beta of 2.x. The 2.x branch is not quite fully featured enough for a packaged release though.

The core of the back-end system is in good shape, and complete.

The front end has working versions of the ticket center, new ticket, and ticket view/edit pages. The major missing front-end components are the admin tools and ticket search features. 

I'm working with the email notification features. The queue side of notifications is in place and working already.  I'm putting together the outgoing notification system. That's going very well, but since I'm not using asp.net to generate the HTML body this time I'm having to write a new content generator for it.

I'm unhappy with the UI in general. It works decent enough on desktop browsers, but the code and JQuery is more than a little sloppy. It does work decently enough though. I'll probably finish the remaining feature and create a release package before I come back through and do a whole front-end re-design though. When I do come back through for the re-design of the UI, I'll probably migrate it over to the newer razor view engine. 

Overall, I am trying for a workable release version before Christmas, and it still look promising that I'll be able to meet that timeline; but I also know better than to make any promises on that date too.     

Coordinator
Nov 18, 2010 at 8:00 AM

 

Update:

The email sending features are falling into place now. I did find a decent way to get the MVC view engine to render the email body content (this time it will have both a text and html body for emails). I have to say, the MVC view engine is nice, but still has some tight coupling that makes it hard to use outside of a user request (a problem most other view engines don't have). The current source has the back-end functionality for sending emails, and a good start on the front-end content generation.  I have not yet put it on a timer though (I won't do that until the code is nearly complete).

I expect that I'll have the email system fully implemented within the week. 

 

My next major venture will likely be the AD system. Any of you who've used TD 1.x in AD environments are probably aware that performance is far slower than it is in SQL environments. First, I HATE AD. My HATE also HATES AD. The AD membership system doesn't quite expose enough functionality, so TD has a custom component that reads into AD for additional info (user display names and email addresses mainly). The TD 1.x system has a simple cache mechanism to take some of the pain out of the process, and that is what TD 2.x is currently using (in a modified form). 

The big problem though is that AD still takes forever to query, and it is still one of the worst all-around technologies I've ever encountered. To work around the performance issues, and also to keep the system from becoming unreliable should AD be unavailable, I plan to replace this entire mechanism with a much more aggressive cache mechanism.

The new system will use a cache manager that runs in the background. It will query AD periodically for every value it might needs, and then store the fetched values in new tables in the SQL database. The rest of TicketDesk will use the cached AD data stored the SQL database, never directly requesting anything from AD.

In theory, this should have comparable performance (from the end-user perspective) as a setup with pure SQL membership.  

Coordinator
Nov 21, 2010 at 1:02 PM

Update:

I've finished the majority of the email system. HTML and PlainText content bodies will be sent. The timer automation is in place as well. For the current default dev environment the emails are setup to use the file system and  drop to C:\temp, but you should be able to setup web.config to use SMTP in web.config if you want to test that out too.

Outlook is a bit of a problem. Even outlook 2010 still uses MS Word to render email content, and word really sucks at HTML... and not even a little bit. Unfortunately, many environments (mine included) are stuck with crippled HTML emails due to the reliance on outlook. I've created an outlook specific view template, but haven't fully re-designed around all the limitations of MS Word yet. You can set a flag in web.config if you want to enable the outlook template though.

Right now, there are still some screen views you can use to render an HTML content body and plain text body for example purposes. The URLs for these are: ~/EmailTemplate/DisplayHtml  and ~/EmailTemplate/DisplayText. Note that the plain text template is designed for a 75 character display, and uses word-wrapping and field truncation to contain the data to those widths. It isn't meant to be "pretty" really, just functional for the 5 places that still don't allow any HTML emails :P 

I also added ELMAH logging back. There is no GUI link to it, but the error logs should be at: ~Admin/elmah.axd if you are logged in as an admin user. This is only setup in the base web.config right now (I'll add it to the SQL and AD specific ones in the next few days. 

I plan to provide user-selectable email format options (including outlook specific HTML) once I have the admin tools in place. I will also provide the option for users to opt-out of email notifications.

In a later release (after 2.0) I'll extend notifications to support RSS feeds and possibly SMS, twitter, etc.   

At any rate... email notifications as they are planned for 2.0 are functionally complete now (both the queuing and sending sides). All that remains is some final polishing.

Enjoy! 

Coordinator
Nov 22, 2010 at 1:25 AM

Update:

I managed to get the outlook specific email template to render something similar to the standard HTML template. It isn't as elegant, and I'm sure I left a bunch of CSS stuff in there that outlook just ignores. I've only tested with outlook 2010 so far too... so I can't be sure it displays as well in older versions (but the Word HTML rendering hasn't changed much in 2010).

I've also wired up a temporary on-screen view for the template at: ~/EmailTemplate/DisplayOutlookHtml

Of course, on-screen it will look slightly better than in actual outlook.

This should conclude most of the email work for a while. I'll leave the controller actions for on-screen rendering in for a while though until I'm sure I'm about done with the development.

I'll also have to clean up the database test data soon.

Now I'll be moving on to the AD issues.... might take a while though, but I should have plenty of time to work on it over the holiday weekend.    

 

Coordinator
Nov 24, 2010 at 8:53 AM

Update:

I've fixed a problem with the WMD editor that only appears to affect newer releases of Google's Chrome browser. 

I've also just checked in the first revision of the new AD cache system. It has NOT been fully tested yet, so don't expect too much if you pull and run the latest source.

The cache system does NOT deal with the user passwords or authentication; the cache system ONLY deals with the list of usernames for each of the AD groups that TicketDesk is using (for use in dropdown lists) and the users' display name and email address. TicketDesk relies on the operating system for the true "security" features of AD (authenticating users). But the windows security mechanisms do not provide TicketDesk with access to additional information from AD that it needs, such as the user's "friendly display name" and the user's email address. Additionally, the built-in security systems do not provide a way for the system to get a list of all users in an AD group.

Because of these limits, TicketDesk (including 1.x) has always used a direct connection to AD to fetch these additional details. That's what those AD credentials in web.config are needed for. TicketDesk needs the list of users in each AD group so it can generate certain dropdown lists (the filters in ticketcenter, assigned to and owned by lists, etc.). TicketDesk also needs the email address for use in notificaitons, and it needs the user's display name so it can show a friendly names for each user in place of the ugly AD account names.

Unfortunately, AD is not exactly a high-performance database and ticketdesk often needs to pull quite a bit of AD data for use on screen (especially in ticketcenter where it has a lot of different names to display). TicketDesk's perforamcne suffers severely when it has to talk to AD. To mitigate that, it has always used an in-memory cache. But when the cache is empty, or has been dumped, it has to fall back on reading from AD again... resulting delays for the users.

The new cache mechanism is an aggressive mechanism designed to keep the user experience from being affected by reads from AD. The basic idea of the new cache system is based on the following:

  • TicketDesk will store AD information in the local SQL database. This is a persistent cache. A background process in TD is responsible for reading from AD and keeping the SQL store updated from AD periodically. This periodic update doesn't have to occur very frequently since AD information tends not to change often. This background process is the ONLY part of ticketdesk that ever talks to AD.
  • All other parts of TicketDesk rely on a cache service to get at the user data. When data is needed, the cache system will fetch it from the SQL persistent cache. If the data is NOT in the SQL store, it will just not be available to TicketDesk until the background process runs another refresh cycle. If the data is in the SQL store, the cache service will then add it to an in-memory cache so that future requests can be serviced without talking to SQL.
  • The in-memory cache is cleared by the background process when it updates the SQL store from AD.  

The overall result should be that TD will perform very well from a user perspective. The drawback is that changes in AD can take a while to show up in TicketDesk.

I will be testing the new system over the next few days in a real AD environment to work out any bugs or problems.   

 

Coordinator
Nov 27, 2010 at 10:31 PM

Update:

Today is a big(ish) day for for TicketDesk 2.0. I've just checked in a major update that adds search back to the app. This time, I'm using Lucene.net for full text indexing and search.

RANT:

I'm not particularly experienced with Lucene, nor the .NET port so this has been quite an experience for me. The core of Lucene is quite amazing in terms of capability, performance, etc. It is one of the most powerful indexing and search libraries available. However; the .NET port of lucene is also one of the most frustratingly annoying I've ever encountered too. Fist of all, the .NET version is a minimal and raw port from the Java implementation. Worse, the Lucene API's public facade is changed more often than I change my underwear. So, every version introduces radically different APIs. This means that any code samples you find for how to use it are probably referring not only to methods that don't exist anymore, but possibly even to entire sub-systems that don't exist, were moved, or whose behavior is radically different. The actual documentation itself is also some of the worst I've ever seen. Very little is actually described in the documentation, and none of it gives practical examples. The handful of too-simple "demo" apps that ship with it are horribly broken. And even worse, the release builds are WAY behind the actual current source versions too.

I think the part that bothers me the most is that a lot of the people doing the .NET porting don't appear to actually know C# at a proficient level. They know Java, but when they port to C# they fail to take advantage of the appropriate .NET mechanisms.

Despite that though; Lucene.NET remains one of the best search systems available. This is because the real "hard part" is the deep internal algorithms; and lucene has nailed this stuff well. There are other .NET native search systems, but most are either incomplete or commercial. 

Ok, so in TicketDesk, I'm using Lucene.NET as best as I was able to figure out. I've pulled the newest source from SVN and compiled it against .NET 4.

The system is setup to create a file system stored index. This is controlled (currently) by a web.config appSetting. You can change the value there to "ram" if you want to do all the indexing in-memory. This is viable for small environments, or development, but I recommend using the file system for production deployements.  Asp.net will need read/write access to the folder where you store the indexes.

The back-end framework will re-build the entire index on application startup (runs as a background thread, so it shouldn't prevent users from working in the system while the index is built-up). As tickets are changed, the index is updated on-the-fly; but again on a background thread. Changes in tickets should be reflected in searches within a few seconds of being saved. 

The ticket's title, tags, details, and text from all comments are included in the index. I've done some work to prioritize the fields a little. Open tickets will get a boost, and should bubble up to the top of most searches. Tags and titles get more emphasis than the details text, while comment text is slightly de-emphasized. The actual weights I've used are probably horribly in-effective though so they may need some tweaking.  

On the front-end, the search is implemented as a simple "search" box at the top of the page. There is no longer a deticated "search" section of the application like there was in TicketDesk 1.x. There are also no complex filters and such either. Just type what you are looking for and Lucene will do what it can to find the best matches.

One thing missing from search currently are "people". You can't search for user's names. This is because the system doesn't store the display names for users in a way that search can easily leverage. I'll be looking at enhancing that in a later release, but if you need to view tickets for a particular user currently you'll have to use the ticketcenter's filters and do an eye-ball search.

If you are familiar with lucene search syntax, you can do a lot of really advanced stuff. For example "tags:mytag" will search ONLY for tickets whose tag field is "mytag". You can also do wildcards * and ?, and the system supports operators (and/or). And there are fuzzy searches by appending "~" to the end of a search term. 

Unless there are problems with the search system, I'm probably done coding in this area for this immediate release.

 

My next projects are enabling multi-column sorting in TicketCenter, Admin tools, and some back-end architectural fix-ups.   

 

 

 

Coordinator
Nov 28, 2010 at 9:28 AM

Update:

Ok, now I'm (hopefully) done with search for a while. I found a number of bugs and performance deficiencies during live testing in my company's live instance. Mostly the problems were not apparent until the system ran into larger databases (we have upwards of 10k tickets in there now).

I've also discovered that the advanced query syntax doesn't quite work. You can do the field filters (example: "tags:mytag") but the wildcards and other features don't appear to work with the kind of query object I'm using. I'll take a closer look at that for a later release.  

Coordinator
Nov 29, 2010 at 7:17 PM

Update:

Added Multi-Column Sorting back to TicketCenter.

Multi-column sorts were either ignored, or loved in the previous versions of TicketDesk. Most people just ignored them, but some people lived by them. In adding it back, I wanted an unobtrusive way to handle it in the UI, instead of some complex pop-up sort configurator. So in TicketDesk 2, the new mechanism is based on a simple click vs. shift+click action link in the header.

  • Click an unsorted column's header to sort by that column (will remove other sorts).
  • Click a sorted column's header to change the sort direction (will not remove other sorts).
  • Shift+Click an unsorted column's header to add that column to the sort
  • Shift+Click a sorted column's header to remove that column from the sort (will not remove the last sort column though, at least one sort column is required)

By using this simple pattern, any sort is possible. To assist in creating multi-sorts, column headers now also flag the order of the column in the sort chain with a simple numeric superscript along with the up/down arrow to show sort direction. A title element will pop-up simple instructions about the shift+click alternative. 

For users that don't care about multi-sort, the standard click-to-sort behavior will function as it does in just about web app on the planet, producing the expected single column sorting each time. More advanced users that actually care about multi-sort can hopefully figure out how it works for themselves without too much trouble. 

 

 

Coordinator
Dec 4, 2010 at 9:37 PM

Update:

Today's check-in is mostly about getting the admin tools in place. I've created an admin area that contains the admin home page, link to ELMAH logs, application settings section, and email diagnostics section.

The user management tools are NOT in place yet.

The Email Diagnostics section is basically the new home for the HTML body content generator for the notifications system. This admin page lets you preview how the the body contents will render from each of the templates (html, outlook html, and plain/text). It also contains a link that will manually invoke the email send routine; which is really only useful if you are working in development and have the email notifications disabled in application settings (notifications are still queued up even if the system is disabled, it just doesn't send the notifications out... this behavior may change in the future as I'm not sure that's the best design).

The application setting section is an expanded version of the same section from TD 1x. In TD 1.x this only controlled content for the drop down lists (ticket type, category, and priority). In TD 2, this system now stores most of the configuration that was previously in web.config. These changes required a significant change in the Settings table in the database, and also requires that there be some "seed" data in the table to provide default values and descriptions.

Web.config still has settings that control the overall behavior of the security mechanisms. The SqlRole/AdGroup to TDRole mappings are still required in web.config, as is the security mode (SQL or AD) setting. The system needs these values long before enough of the system is initialized to query the settings table in the DB. In AD environments, the AD domain and user account settings are also required. These will get refactored out of web.config (hopefully) in a future version after TD 2 is released. I plan to replace the simple role mappings with a full blown groups and permissions system down the road a bit, and at that time I'll pull these settings into the database where possible.     

The new admin tools aren't quite finished, and are rough around the edges a bit. The Application Settings in particularly is a tad "minimalist" in design. I plan to redesign this entire section in a later version when I switch the front-end over to a more formal MVVM design pattern. For the initial TD 2 roll-out though, this mechanism should be sufficient.

I also fixed a couple moderate bugs that crept in recently. One was that adding attachments with new tickets broke the lucene search indexing. Another was a problem with the "edit details" action in ticket editior not correctly recording the "before" and "after" values for the changed properties.

There were a few UI tweaks, nothing special aside from the TD 2 logo (my graphics skills are minimal, so don't be too disappointed) 

Coordinator
Dec 6, 2010 at 12:06 AM

Update:

I've taken the time to clean up the databases a bit:

  • Combined security and ticketdesk SQLExpress databases: The development project and default file-attached SQLExpress databases will use only a single SQL database for both security and ticketdesk objects now. This should simplify the default configuration of TicketDesk. There will still be separate connection strings in the web.config, but they will all point to the same database in the default distribution. Users will still be able to split their security and ticketdesk among two databases manually if they choose, and I'll document this before I package up the application for the final 2.0 release.
  • Re-created the 3 stock user accounts; toastman, admin, and otherstaffer. These demonstrate the three typical kinds of user in TicketDesk. I recommend, for production environments, that administrators reconfigure the membership provider to use encrypted passwords and setup a  machine key in web.config. But for simplicity in the default distribution version, passwords are clear text.
  • Cleaned out all the tickets and related data from the stock SQLExpress database in app_data.
  • Created SQL scripts for users that want to do a fresh database setup themselves. Included are scripts for the TD objects, the asp.net security objects, and a script to generate the three stock user accounts.       
Coordinator
Dec 9, 2010 at 6:17 AM

Update:

Today's check-in adds the Security Management Admin section, and completes the User Management tools. In addition, the user registration page has finally been updated and brought in-sync with the admin tools. 

TicketDesk 2.0 now includes the ability for admins to create and delete users, as well as change user details, role assignments, and approval status. However, TD 2 will not support password retrieval out-of-the-box, nor can admins reset/change passwords via the UI (maybe in a later version). 

I also fixed/updated:

  • A rendering problem with HTML and Outlook templates and the height of the details area.
  • Code blocks in details and comments of tickets
  • Wrong controller was rendering the admin home section's views
  • Hide security section when in AD security mode 
  • Minor refactoring of Security Service and Security Repositories
  • General tweaks
  • New stock images for admin section
  • Fixed up the admin mini-nav side-bar 
Coordinator
Dec 13, 2010 at 3:40 PM

Update:

This check-in is mostly about trying to clean-up the UIs a little for workable cross-browser rendering. This is very much still a work in progress.

The biggest change is that I've moved the ticket editor activities back to the middle of the screen (similar to where it was in TD 1.x). I liked having the side-bar activity area, but getting it to render and animate correctly cross browser just wasn't working out. So it's back where it used to be. 

I also switched back to the MarkItUp! editor for now. First of all, I needed to make sure that rendering still looked decent with the other editor; but also, I'm still having major problems with the WMD editor. I'm using the openlibrary version of WMD, but even still it has a lot of behavior annoyances that need to be tracked down and killed before it will be usable as a default editor. I have not yet put in the FCKEditor, PlainText editor, nor the configuration support for runtime editor switching... that might slip to version 2.1 though.

While I was tweaking screen UIs, I also got more time to test the email rendering. To my disgust, outlook rendering continued to fail miserably. So I've done research on the "rendering" engine that Office 2007/2010 use (OMGWTF?!?! seriously?!?!?). I had to completely rebuild the UI for the outlook email template. The result is actually quite an excellent example of table layouts... excellent for 1997! I remember writing HTML exactly like that back in the day, yo!

The other area of major work has been the attachment uploader. TD 2 is using a flash uploader, which works great for SQL environments... but wouldn't upload at all in my AD environment. Turns out that the major problem seems to be a flash issue with windows authentication. Failing to find a workable solution to authenticate the uploads in AD environments, I had to hack the controller so that it doesn't authenticate AT ALL for AD setups.

I plan to add a non-flash uploader, and make the flash uploader a optional setting instead; but that's another feature that might slip to the 2.1 packaged release.

Anyway, this check-in may still have some issues. I'm still ironing out and testing the UIs in multiple browsers; plus I  made a lot of changes in areas where I don't have reliable unit testing (the views and javascript behaviors in particular).... so there could be some regressions this check-in.   

Coordinator
Dec 14, 2010 at 3:15 AM

Update:

After continued problems with flash multi-uploader and AD security, I've replaced attachment uploading with a simplistic JQuery uploader. Keeps the same overall pattern of use, but works without flash, can authenticate correctly in AD, and is just plain simple.

I also cleaned up some of the unused scripts from source.  

Dec 14, 2010 at 8:00 PM

 Well after just coming back from vacation, I wanted to give you a shout and let you know I have been monitoring the emails coming in from all your updates.  Sounds really nice so far stephen.  And I have to agree on the one last note you made.  SIMPLISTIC.. Thats what is sooooo attractive about this whole application is the simplicity of it.

 Although as noted before I am by far not a programmer and couldnt manipulate it the way you do if I had to, and hearing you change code in and out looking for what works best based on trial and analysis of real points of interst in each piece is quite intriguing alone.  I can't wait till it's finished and we all start debugging it. 

 I did not see however an upgrade path?  I presume a clean install at this point will be best as the data structure has changed unless you have any insights into keeping the old data ?

Coordinator
Dec 15, 2010 at 1:27 PM

An upgrade path is included. It should be rather simple too; just run one upgrade script against any TD 1.x database and you should be good to go on that end. For the web application, all you have to do is replace the old one with the new one. 

One thing I was very careful about with 2.x was keeping the number of database changes to a minimum. I had several features I'd planned for 2.0 that I cut from the release just because it would complicate the database more than I wanted and make upgrades difficult.

In this at least, I was successful. It is possible to run both TD 1.x and TD 2.x against the same database concurrently... I'm doing that in our offices now for pre-release testing. I've upgraded our database using the upgrade script, and then left the old TD 1.x installation online. Our end-users still use the TD 1.x instance. Our help desk is beta-testing 2.x by using a separate instance of the TD 2.x web app. Both instances are using the same database. The only catch is that you have to disable email notifications in one instance or the other to prevent then both trying to send emails. This is possible largely because the changes to the database schema were very minor, and most of them are new features that only TD 2 needs.

Part of the upgrade script that is very important is creating the default configuration values in the database. These are used by TD 2 instead of using settings in web.config; so after you upgrade the database from the script you need to verify go to the application settings admin tool and make sure everything is setup the way you want. The defaults are pretty sensible though.

For releases after TD 2.0, I plan to have the system perform it's own database upgrades when it starts-up. I've added a version setting that the system can use to decide which upgrade scripts would be needed. So, when 2.1 is ready, all you have to do is deploy the web app and it should take care of the database all by itself.     


Now... if I can just get IE to behave!  

Coordinator
Dec 16, 2010 at 8:01 PM

Update:

I have managed to work out most of the IE rendering problems (I hope).

I also modified the MarkItUp! editor to use a side-by-side layout for the preview windows. That should make better use of screen real-estate. 

I put the finishing touches on the last of the application settings that hadn't been implemented in 2.x yet (flags for submitter allowed to edit priority and submitter allowed to edit tags, and auto-assigning new user registrations to the submitters role). 

I've also removed the role editor admin tool (for now). With the new front-end for editing users, this mechanism didn't seem necessary. I will add it back in a later release though since it can sometimes be convenient; but I was also planning to overhaul the roles/permissions system in one of the 2.x follow-on releases anyway.  

 

Coordinator
Jan 4, 2011 at 6:49 PM

Update:

I've started to put together the 2.0 documentation here on CodePlex. This is rough-draft right now, and far from complete, but I've already put up installation, configuration, and upgrade instructions.

I'll be filling in the docs over the next several days.  

Coordinator
Jan 5, 2011 at 12:26 AM

I've release TicketDesk 2.0 in beta form, and completed most of the draft documentation.

I'll continue work over the next several days until the product reaches a point where I'm comfortable creating a "stable" release. I will also be producing more documentation, screen-shots, etc. 

Thanks for all your support :)

I'll create a new thread in the discussion board for future updates.